How to delete multiple pandas python dataframes from memory to save RAM

Question

I have lot of dataframes created as part of preprocessing  Since I have limited 6GB ram  I want to delete all the unnecessary dataframes from RAM to avoid running out of memory when running GRIDSEARCHCV in scikit-learn   1  Is there a function to list only  all the dataframes currently loaded in memory   I tried dir   but it gives lot of other object other than dataframes   2  I created a list of dataframes to delete   del df  Gender dummies   capsule trans   col   concat df list   coup CAPSULE dummies     amp  ran   for i in del df      del  i    But its not deleting the dataframes  But deleting dataframes individially like below is deleting dataframe from memory   del Gender dummies del col

User · Answer

In python automatic garbage collection deallocates the variable (pandas DataFrame are also just another object in terms of python). There are different garbage collection strategies that can be tweaked (requires significant learning).

You can manually trigger the garbage collection using

import gc
gc.collect()

But frequent calls to garbage collection is discouraged as it is a costly operation and may affect performance.

Reference

User · Answer

del statement does not delete an instance  it merely deletes a name  When you do del i  you are deleting just the name i - but the instance is still bound to some other name  so it won t be Garbage-Collected  If you want to release memory  your dataframes has to be Garbage-Collected  i e  delete all references to them  If you created your dateframes dynamically to list  then removing that list will trigger Garbage Collection   gt  gt  gt  lst    pd DataFrame    pd DataFrame    pd DataFrame     gt  gt  gt  del lst       memory is released  If you created some variables  you have to delete them all   gt  gt  gt  a  b  c   pd DataFrame    pd DataFrame    pd DataFrame    gt  gt  gt  lst    a  b  c   gt  gt  gt  del a  b  c   dfs still in list  gt  gt  gt  del lst       memory release now

User · Answer

This will delete the dataframe and will release the RAM memory del   df 1 df 2   gc collect   df 1 pd DataFrame   df 2 pd DataFrame    the data-frame will be explicitly set to null in the above statements Firstly  the self reference of the dataframe is deleted meaning the dataframe is no longer available to python there after all the references of the dataframe is collected by garbage collector  gc collect    and then explicitly set all the references to empty dataframe  more on the working of garbage collector is well explained in https   stackify com python-garbage-collection

[python] How to delete multiple pandas (python) dataframes from memory to save RAM?

Examples related to python

Examples related to memory-management

Examples related to pandas

Examples related to dataframe

Examples related to ram