'Python memory consumption when using pickle.load vs process memory usage
I'm not very familiar with Python memory management ou profiling but I'm in a weird situation and can't find any clear explication.
I'm trying to load a large ML Model (1.4 GB) using pickle in my Python code (3.7). When I call pickle.load, my process ram usage increase of around 8GB (according my task manager). I can easily understand that the pickle version on disk should be smaller than the object fully loaded in Python but the difference seems me excessive so I used Pympler to check the size of my model in my Python code and I got around 5GB.
I would like to understand why I get this 3GB difference:
- There is a problem with Pickle?
- Pympler doesn't count all the objects accurately?
- My task manager doesn't report the ram consumption the same way Pympler count it?
- ...
If you have another ideas to profile my code, feel free to answer.
Thanks
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
