Hi there, I am trying to run my data analysis codes on RSP for my downloaded images, but got some troubles with the memory issue.
My codes run thorugh a list of objects in a loop, something like:
for obj in obj_list:
analysis(obj)
However, the RAM usage keeps accumulating as it loops though the objects. I have put these lines to clean the memory at the end of each loop:
plt.close()
gc.collect()
clear_output(wait=True)
And also deleted the referrers:
def deep_delete(obj):
for ref in gc.get_referrers(obj):
if isinstance(ref, dict):
for k, v in list(ref.items()):
if v is obj:
del ref[k]
elif isinstance(ref, list):
while obj in ref:
ref.remove(obj)
del obj
But it keeps accumulating, as I check psutil.virtual_memory() before and after I run on one object, it increases from
svmem(total=33651441664, available=23263485952, percent=30.9, used=9868546048, free=18948648960, active=2638352384, inactive=10409881600, buffers=1091153920, cached=3743092736, shared=29171712, slab=1425485824)
Total RAM: 31.34 GB
Available RAM: 21.67 GB
Used RAM: 9.19 GB
Memory Usage: 30.9%
Current Python process is using 6.62 GB of RAM
to:
svmem(total=33651441664, available=22799486976, percent=32.2, used=10332549120, free=18456993792, active=2639040512, inactive=10881220608, buffers=1091153920, cached=3770744832, shared=29171712, slab=1425707008)
Total RAM: 31.34 GB
Available RAM: 21.23 GB
Used RAM: 9.62 GB
Memory Usage: 32.2%
Current Python process is using 7.10 GB of RAM
I am not accessing the calexp images, bulters, and data catalogs of LSST in this code, but just doing a loop of analysis. I thought as all the class objects and the variables will be rewritten in the loop, the RAM usage should stay stable when it goes through? I am not sure what causes the memory leakage here. I’d appreciate your thoughts!