Shift elements in a numpy array

Question

Following-up from this question years ago  is there a canonical  shift  function in numpy  I don t see anything from the documentation   Here s a simple version of what I m looking for   def shift xs  n       if n  gt   0          return np r  np full n  np nan   xs  -n       else          return np r  xs -n    np full -n  np nan     Using this is like   In  76   xs Out 76   array   0    1    2    3    4    5    6    7    8    9     In  77   shift xs  3  Out 77   array   nan   nan   nan    0     1     2     3     4     5     6     In  78   shift xs  -3  Out 78   array    3     4     5     6     7     8     9    nan   nan   nan     This question came from my attempt to write a fast rolling product yesterday  I needed a way to  shift  a cumulative product and all I could think of was to replicate the logic in np roll       So np concatenate   is much faster than np r     This version of the function performs a lot better   def shift xs  n       if n  gt   0          return np concatenate  np full n  np nan   xs  -n        else          return np concatenate  xs -n    np full -n  np nan        An even faster version simply pre-allocates the array   def shift xs  n       e   np empty like xs      if n  gt   0          e  n    np nan         e n     xs  -n      else          e n     np nan         e  n    xs -n       return e

User · Answer

If you want a one-liner from numpy and aren't too concerned about performance, try:

np.sum(np.diag(the_array,1),0)[:-1]

Explanation: np.diag(the_array,1) creates a matrix with your array one-off the diagonal, np.sum(...,0) sums the matrix column-wise, and ...[:-1] takes the elements that would correspond to the size of the original array. Playing around with the 1 and :-1 as parameters can give you shifts in different directions.

User · Answer

Benchmarks  amp  introducing Numba 1  Summary  The accepted answer  scipy ndimage interpolation shift  is the slowest solution listed in this page  Numba   numba njit  gives some performance boost when array size smaller than  25 000  quot Any method quot  equally good when array size large   gt 250 000   The fastest option really depends on           1     Length of your arrays          2     Amount of shift you need to do  Below is the picture of the timings of all different methods listed on this page  2020-07-11   using constant shift   10  As one can see  with small array sizes some methods are use more than  2000  time than the best method    2  Detailed benchmarks with the best options  Choose shift4 numba  defined below  if you want good all-arounder   3  Code 3 1 shift4 numba  Good all-arounder  max 20  wrt  to the best method with any array size Best method with medium array sizes    500  lt  N  lt  20 000  Caveat  Numba jit  just in time compiler  will give performance boost only if you are calling the decorated function more than once  The first call takes usually 3-4 times longer than the subsequent calls   import numba   numba njit def shift4 numba arr  num  fill value np nan       if num  gt   0          return np concatenate  np full num  fill value   arr  -num        else          return np concatenate  arr -num    np full -num  fill value     3 2  shift5 numba  Best option with small  N  lt   300   1500  array sizes  Treshold depends on needed amount of shift  Good performance on any array size  max   50  compared to the fastest solution  Caveat  Numba jit  just in time compiler  will give performance boost only if you are calling the decorated function more than once  The first call takes usually 3-4 times longer than the subsequent calls   import numba   numba njit def shift5 numba arr  num  fill value np nan       result   np empty like arr      if num  gt  0          result  num    fill value         result num     arr  -num      elif num  lt  0          result num     fill value         result  num    arr -num       else          result      arr     return result  3 3  shift5  Best method with array sizes   20 000  lt  N  lt  250 000 Same as shift5 numba  just remove the  numba njit decorator   4 Appendix 4 1 Details about used methods  shift scipy  scipy ndimage interpolation shift  scipy 1 4 1  - The option from accepted answer  which is clearly the slowest alternative  shift1  np roll and out  num  xnp nan by IronManMark20  amp  gzc shift2  np roll and np put by IronManMark20 shift3  np pad and slice by  gzc shift4  np concatenate and np full by  chrisaycock shift5  using two times result slice    x by  chrisaycock shift  numba   numba njit decorated versions of the previous   The shift2 and shift3 contained functions that were not supported by the current numba  0 50 1   4 2 Other test results 4 2 1 Relative timings  all methods  Relative timings  10  shift  all methods Relative timings  constant shift  10   all methods  4 2 2 Raw timings  all methods  Raw timings  constant shift  10   all methods Raw timings  10  shift  all methods  4 2 3 Raw timings  few best methods  Raw timings with small arrays  constant shift  10   few best methods Raw timings with small arrays  10  shift  few best methods Raw timings with large arrays  constant shift  10   few best methods Raw timings with large arrays  10  shift  few best methods

User · Answer

Not numpy but scipy provides exactly the shift functionality you want   import numpy as np from scipy ndimage interpolation import shift  xs   np array   0    1    2    3    4    5    6    7    8    9     shift xs  3  cval np NaN    where default is to bring in a constant value from outside the array with value cval  set here to nan  This gives the desired output   array   nan  nan  nan  0   1   2   3   4   5   6      and the negative shift works similarly   shift xs  -3  cval np NaN    Provides output   array    3     4     5     6     7     8     9    nan   nan   nan

User · Answer

You can convert ndarray to Series or DataFrame with pandas first  then you can use shift method as you want   Example   In  1   from pandas import Series  In  2   data   np arange 10   In  3   data Out 3   array  0  1  2  3  4  5  6  7  8  9    In  4   data   Series data   In  5   data Out 5    0    0 1    1 2    2 3    3 4    4 5    5 6    6 7    7 8    8 9    9 dtype  int64  In  6   data   data shift 3   In  7   data Out 7    0    NaN 1    NaN 2    NaN 3    0 0 4    1 0 5    2 0 6    3 0 7    4 0 8    5 0 9    6 0 dtype  float64  In  8   data   data values  In  9   data Out 9   array   nan   nan   nan    0     1     2     3     4     5     6

User · Answer

For those who want to just copy and paste the fastest implementation of shift  there is a benchmark and conclusion see the end   In addition  I introduce fill value parameter and fix some bugs   Benchmark  import numpy as np import timeit    enhanced from IronManMark20 version def shift1 arr  num  fill value np nan       arr   np roll arr num      if num  lt  0          arr num     fill value     elif num  gt  0          arr  num    fill value     return arr    use np roll and np put by IronManMark20 def shift2 arr num       arr np roll arr num      if num lt 0           np put arr range len arr  num len arr   np nan      elif num  gt  0           np put arr range num  np nan      return arr    use np pad and slice by me  def shift3 arr  num  fill value np nan       l   len arr      if num  lt  0          arr   np pad arr   0  abs num    mode  constant   constant values  fill value     -num      elif num  gt  0          arr   np pad arr   num  0   mode  constant   constant values  fill value     -num       return arr    use np concatenate and np full by chrisaycock def shift4 arr  num  fill value np nan       if num  gt   0          return np concatenate  np full num  fill value   arr  -num        else          return np concatenate  arr -num    np full -num  fill value       preallocate empty array and assign slice by chrisaycock def shift5 arr  num  fill value np nan       result   np empty like arr      if num  gt  0          result  num    fill value         result num     arr  -num      elif num  lt  0          result num     fill value         result  num    arr -num       else          result      arr     return result  arr   np arange 2000  astype float   def benchmark shift1        shift1 arr  3   def benchmark shift2        shift2 arr  3   def benchmark shift3        shift3 arr  3   def benchmark shift4        shift4 arr  3   def benchmark shift5        shift5 arr  3   benchmark set     benchmark shift1    benchmark shift2    benchmark shift3    benchmark shift4    benchmark shift5    for x in benchmark set      number   10000     t   timeit timeit   s      x   from   main   import  s    x  number number      print   s time   f     x  t    benchmark result   benchmark shift1 time  0 265238 benchmark shift2 time  0 285175 benchmark shift3 time  0 473890 benchmark shift4 time  0 099049 benchmark shift5 time  0 052836   Conclusion  shift5 is winner  It s OP s third solution

User · Answer

One way to do it without spilt the code into cases  with array   def shift arr  dx  default value       result   np empty like arr      get neg or none   lambda s  s if s  lt  0 else None     get pos or none   lambda s  s if s  gt  0 else None     result get neg or none dx   get pos or none dx     default value     result get pos or none dx   get neg or none dx     arr get pos or none -dx   get neg or none -dx            return result   with matrix it can be done like this   def shift image  dx  dy  default value       res   np full like image  default value       get neg or none   lambda s  s if s  lt  0 else None     get pos or none   lambda s   s if s  gt  0 else None      res get pos or none -dy   get neg or none -dy   get pos or none -dx   get neg or none -dx               image get pos or none dy   get neg or none dy   get pos or none dx   get neg or none dx       return res

User · Answer

You can also do this with Pandas   Using a 2356-long array   import numpy as np  xs   np array          Using scipy   from scipy ndimage interpolation import shift   timeit shift xs  1  cval np nan    956   s    77 9   s per loop  mean    std  dev  of 7 runs  1000 loops each    Using Pandas   import pandas as pd   timeit pd Series xs  shift 1  values   377   s    9 42   s per loop  mean    std  dev  of 7 runs  1000 loops each    In this example  using Pandas was about  8 times faster than Scipy

User · Answer

There is no single function that does what you want  Your definition of shift is slightly different than what most people are doing  The ways to shift an array are more commonly looped    gt  gt  gt xs np array  1 2 3 4 5    gt  gt  gt shift xs 3  array  3 4 5 1 2     However  you can do what you want with two functions  Consider a np array   0    1    2    3    4    5    6    7    8    9      def shift2 arr num       arr np roll arr num      if num lt 0           np put arr range len arr  num len arr   np nan      elif num  gt  0           np put arr range num  np nan      return arr  gt  gt  gt shift2 a 3    nan  nan  nan   0    1    2    3    4    5    6    gt  gt  gt shift2 a -3     3    4    5    6    7    8    9   nan  nan  nan    After running cProfile on your given function and the above code you provided  I found that the code you provided makes 42 function calls while shift2 made 14 calls when arr is positive and 16 when it is negative  I will be experimenting with timing to see how each performs with real data

[python] Shift elements in a numpy array

Examples related to python

Examples related to numpy