How to implement the ReLU function in Numpy

Question

I want to make a simple neural network which uses the ReLU function  Can someone give me a clue of how can I implement the function using numpy

User · Answer

numpy didn t have the function of relu  but you define it by yourself as follow   def relu x       return np maximum 0  x    for example   arr   np array   -1 2 3   1 2 3     ret   relu arr  print ret    print   0 2 3   1 2 3

User · Answer

You can do it in much easier way   def ReLU x       return x    x  gt  0   def dReLU x       return 1     x  gt  0

User · Answer

Richard M  hn s comparison  is not fair  As Andrea Di Biagio s comment  the in-place method np maximum x  0  x  will modify x at the first loop  So here is my benchmark     import numpy as np  def baseline        x   np random random  5000  5000   - 0 5     return x  def relu mul        x   np random random  5000  5000   - 0 5     out   x    x  gt  0      return out  def relu max        x   np random random  5000  5000   - 0 5     out   np maximum x  0      return out  def relu max inplace        x   np random random  5000  5000   - 0 5     np maximum x  0  x      return x    Timing it     print  baseline     timeit -n10 baseline   print  multiplication method     timeit -n10 relu mul   print  max method     timeit -n10 relu max   print  max inplace method     timeit -n10 relu max inplace     Get the results     baseline  10 loops  best of 3  425 ms per loop multiplication method  10 loops  best of 3  596 ms per loop max method  10 loops  best of 3  682 ms per loop max inplace method  10 loops  best of 3  602 ms per loop   In-place maximum method is only a bit faster than the maximum method  and it may because it omits the variable assignment for  out   And it s still slower than the multiplication method  And since you re implementing the ReLU func  You may have to save the  x  for backprop through relu  E g      def relu backward dout  cache       x   cache     dx   np where x  gt  0  dout  0      return dx   So i recommend you to use multiplication method

User · Answer

ReLU x  also is equal to  x abs x   2

User · Answer

If we have 3 parameters  t0  a0  a1  for Relu  that is we want to implement  if x  gt  t0      x   x   a1 else      x   x   a0   We can use the following code   X   X    X  gt  t0    a1    X    X  lt  t0    a0   X there is a matrix

User · Answer

There are a couple of ways    gt  gt  gt  x   np random random  3  2   - 0 5  gt  gt  gt  x array   -0 00590765   0 18932873           -0 32396051   0 25586596            0 22358098   0 02217555     gt  gt  gt  np maximum x  0  array    0            0 18932873            0            0 25586596            0 22358098   0 02217555     gt  gt  gt  x    x  gt  0  array   -0            0 18932873           -0            0 25586596            0 22358098   0 02217555     gt  gt  gt   abs x    x    2 array    0            0 18932873            0            0 25586596            0 22358098   0 02217555      If timing the results with the following code   import numpy as np  x   np random random  5000  5000   - 0 5 print  max method     timeit -n10 np maximum x  0   print  multiplication method     timeit -n10 x    x  gt  0   print  abs method     timeit -n10  abs x    x    2   We get   max method  10 loops  best of 3  239 ms per loop multiplication method  10 loops  best of 3  145 ms per loop abs method  10 loops  best of 3  288 ms per loop   So the multiplication seems to be the fastest

User · Answer

I m completely revising my original answer because of points raised in the other questions and comments  Here is the new benchmark script  import time import numpy as np   def fancy index relu m       m m  lt  0    0   relus          quot max quot   lambda x  np maximum x  0        quot in-place max quot   lambda x  np maximum x  0  x        quot mul quot   lambda x  x    x  gt  0        quot abs quot   lambda x   abs x    x    2       quot fancy index quot   fancy index relu     for name  relu in relus items        n iter   20     x   np random random  n iter  5000  5000   - 0 5      t1   time time       for i in range n iter           relu x i       t2   time time        print  quot    gt 12s     3 0f  ms quot  format name   t2 - t1    n iter   1000    It takes care to use a different ndarray for each implementation and iteration  Here are the results           max  126 ms in-place max  107 ms          mul  136 ms          abs   86 ms  fancy index  132 ms

User · Answer

EDIT As  jirassimok has mentioned below my function will change the data in place  after that it runs a lot faster in timeit  This causes the good results  It s some kind of cheating  Sorry for your inconvenience  I found a faster method for ReLU with numpy  You can use the fancy index feature of numpy as well  fancy index  20 3 ms    272   s per loop  mean    std  dev  of 7 runs  10 loops each   gt  gt  gt  x   np random random  5 5   - 0 5   gt  gt  gt  x array   -0 21444316  -0 05676216   0 43956365  -0 30788116  -0 19952038           -0 43062223   0 12144647  -0 05698369  -0 32187085   0 24901568            0 06785385  -0 43476031  -0 0735933    0 3736868    0 24832288            0 47085262  -0 06379623   0 46904916  -0 29421609  -0 15091168            0 08381359  -0 25068492  -0 25733763  -0 1852205   -0 42816953     gt  gt  gt  x x lt 0  0  gt  gt  gt  x array    0            0            0 43956365   0            0                     0            0 12144647   0            0            0 24901568            0 06785385   0            0            0 3736868    0 24832288            0 47085262   0            0 46904916   0            0                     0 08381359   0            0            0            0              Here is my benchmark  import numpy as np x   np random random  5000  5000   - 0 5 print  quot max method  quot    timeit -n10 np maximum x  0  print  quot max inplace method  quot    timeit -n10 np maximum x  0 x  print  quot multiplication method  quot    timeit -n10 x    x  gt  0  print  quot abs method  quot    timeit -n10  abs x    x    2 print  quot fancy index  quot    timeit -n10 x x lt 0   0  max method  241 ms    3 53 ms per loop  mean    std  dev  of 7 runs  10 loops each  max inplace method  38 5 ms    4 ms per loop  mean    std  dev  of 7 runs  10 loops each  multiplication method  162 ms    3 1 ms per loop  mean    std  dev  of 7 runs  10 loops each  abs method  181 ms    4 18 ms per loop  mean    std  dev  of 7 runs  10 loops each  fancy index  20 3 ms    272   s per loop  mean    std  dev  of 7 runs  10 loops each

User · Answer

This is more precise implementation   def ReLU x       return abs x     x  gt  0

[python] How to implement the ReLU function in Numpy

Examples related to python

Examples related to numpy

Examples related to machine-learning

Examples related to neural-network