As noted in the comments, there are some things to try: gc.collect
(@EdChum) may clear stuff, for example. At least from my experience, these things sometimes work and often don't.
There is one thing that always works, however, because it is done at the OS, not language, level.
Suppose you have a function that creates an intermediate huge DataFrame, and returns a smaller result (which might also be a DataFrame):
def huge_intermediate_calc(something):
...
huge_df = pd.DataFrame(...)
...
return some_aggregate
Then if you do something like
import multiprocessing
result = multiprocessing.Pool(1).map(huge_intermediate_calc, [something_])[0]
Then the function is executed at a different process. When that process completes, the OS retakes all the resources it used. There's really nothing Python, pandas, the garbage collector, could do to stop that.