Python memory usage of numpy arrays

Question

I m using python to analyse some large files and I m running into memory issues  so I ve been using sys getsizeof   to try and keep track of the usage  but it s behaviour with numpy arrays is bizarre  Here s an example involving a map of albedos that I m having to open    gt  gt  gt  import numpy as np  gt  gt  gt  import struct  gt  gt  gt  from sys import getsizeof  gt  gt  gt  f   open  Albedo map assoc    rb    gt  gt  gt  getsizeof f  144  gt  gt  gt  albedo   struct unpack   df     7200 3600   f read 7200 3600 4    gt  gt  gt  getsizeof albedo  207360056  gt  gt  gt  albedo   np array albedo  reshape 3600 7200   gt  gt  gt  getsizeof albedo  80   Well the data s still there  but the size of the object  a 3600x7200 pixel map  has gone from  200 Mb to 80 bytes  I d like to hope that my memory issues are over and just convert everything to numpy arrays  but I feel that this behaviour  if true  would in some way violate some law of information theory or thermodynamics  or something  so I m inclined to believe that getsizeof   doesn t work with numpy arrays  Any ideas

User · Answer

In python notebooks I often want to filter out 'dangling' numpy.ndarray's, in particular the ones that are stored in _1, _2, etc that were never really meant to stay alive.

I use this code to get a listing of all of them and their size.

Not sure if locals() or globals() is better here.

import sys
import numpy
from humanize import naturalsize

for size, name in sorted(
    (value.nbytes, name)
    for name, value in locals().items()
    if isinstance(value, numpy.ndarray)):
  print("{:>30}: {:>8}".format(name, naturalsize(size)))

User · Answer

You can use array nbytes for numpy arrays  for example    gt  gt  gt  import numpy as np  gt  gt  gt  from sys import getsizeof  gt  gt  gt  a    0    1024  gt  gt  gt  b   np array a   gt  gt  gt  getsizeof a  8264  gt  gt  gt  b nbytes 8192

User · Answer

The field nbytes will give you the size in bytes of all the elements of the array in a numpy array   size in bytes   my numpy array nbytes   Notice that this does not measures  non-element attributes of the array object  so the actual size in bytes can be a few bytes larger than this

[python] Python memory usage of numpy arrays

Examples related to python

Examples related to numpy

Examples related to sys