Python memory usage of numpy arrays

Question

I m using python to analyse some large files and I m running into memory issues  so I ve been using sys getsizeof   to try and keep track of the usage  but it s behaviour with numpy arrays is bizarre  Here s an example involving a map of albedos that I m having to open    gt  gt  gt  import numpy as np  gt  gt  gt  import struct  gt  gt  gt  from sys import getsizeof  gt  gt  gt  f   open  Albedo map assoc    rb    gt  gt  gt  getsizeof f  144  gt  gt  gt  albedo   struct unpack   df     7200 3600   f read 7200 3600 4    gt  gt  gt  getsizeof albedo  207360056  gt  gt  gt  albedo   np array albedo  reshape 3600 7200   gt  gt  gt  getsizeof albedo  80   Well the data s still there  but the size of the object  a 3600x7200 pixel map  has gone from  200 Mb to 80 bytes  I d like to hope that my memory issues are over and just convert everything to numpy arrays  but I feel that this behaviour  if true  would in some way violate some law of information theory or thermodynamics  or something  so I m inclined to believe that getsizeof   doesn t work with numpy arrays  Any ideas

User · Answer

The field nbytes will give you the size in bytes of all the elements of the array in a numpy.array:

size_in_bytes = my_numpy_array.nbytes

Notice that this does not measures "non-element attributes of the array object" so the actual size in bytes can be a few bytes larger than this.

User · Answer

You can use array nbytes for numpy arrays  for example    gt  gt  gt  import numpy as np  gt  gt  gt  from sys import getsizeof  gt  gt  gt  a    0    1024  gt  gt  gt  b   np array a   gt  gt  gt  getsizeof a  8264  gt  gt  gt  b nbytes 8192

User · Answer

In python notebooks I often want to filter out  dangling  numpy ndarray s  in particular the ones that are stored in  1   2  etc that were never really meant to stay alive   I use this code to get a listing of all of them and their size   Not sure if locals   or globals   is better here     import sys import numpy from humanize import naturalsize  for size  name in sorted       value nbytes  name      for name  value in locals   items       if isinstance value  numpy ndarray      print     gt 30      gt 8   format name  naturalsize size

[python] Python memory usage of numpy arrays

Examples related to python

Examples related to numpy

Examples related to sys