'Python memory consumption when using pickle.load vs process memory usage

I'm not very familiar with Python memory management ou profiling but I'm in a weird situation and can't find any clear explication.

I'm trying to load a large ML Model (1.4 GB) using pickle in my Python code (3.7). When I call pickle.load, my process ram usage increase of around 8GB (according my task manager). I can easily understand that the pickle version on disk should be smaller than the object fully loaded in Python but the difference seems me excessive so I used Pympler to check the size of my model in my Python code and I got around 5GB.

I would like to understand why I get this 3GB difference:

  • There is a problem with Pickle?
  • Pympler doesn't count all the objects accurately?
  • My task manager doesn't report the ram consumption the same way Pympler count it?
  • ...

If you have another ideas to profile my code, feel free to answer.

Thanks



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source