'Python's memory-profiler reveals np.random taking up too much memory
I am using Python's memory-profiler to better understand how much memory my code is using and when setting up this simple example, I found a much higher memory usage than expected:
import numpy as np
from memory_profiler import profile
@profile
def func(n):
x1 = np.random.random(n).astype(np.float32)
x2 = np.random.random(n).astype(np.float32)
x_sum = x1 + x2
return x_sum
@profile
def main():
"""
Main script calling func
"""
x = func(1024**2)
if __name__ == "__main__":
main()
Output:
Filename: mem-test-5.py
Line # Mem usage Increment Occurences Line Contents
============================================================
4 52.7 MiB 52.7 MiB 1 @profile
5 def func(n):
6 56.8 MiB 4.1 MiB 1 x1 = np.random.random(n).astype(np.float32)
7 68.7 MiB 11.9 MiB 1 x2 = np.random.random(n).astype(np.float32)
8 68.7 MiB 0.0 MiB 1 x_sum = x1 + x2
9 68.7 MiB 0.0 MiB 1 return x_sum
Filename: mem-test-5.py
Line # Mem usage Increment Occurences Line Contents
============================================================
12 52.7 MiB 52.7 MiB 1 @profile
13 def main():
14 """
15 Main script calling func
16 """
17 64.8 MiB 12.1 MiB 1 x = func(1024**2)
x1 has the size expected (1024 x 1024 x 4 = 4194304 bytes = 4 MiB) but why does creating x2 right after take up 12 MiB when it should also be only 4 MiB? Interestingly, this problem does not occur if using np.ones instead of np.random or if not casting x1 and x2 to float32 (that is, leaving them at default float64):
Filename: mem-test-5.py
Line # Mem usage Increment Occurences Line Contents
============================================================
4 52.5 MiB 52.5 MiB 1 @profile
5 def func(n):
6 60.4 MiB 7.9 MiB 1 x1 = np.random.random(n)
7 68.6 MiB 8.1 MiB 1 x2 = np.random.random(n)
8 76.6 MiB 8.0 MiB 1 x_sum = x1 + x2
9 76.6 MiB 0.0 MiB 1 return x_sum
Filename: mem-test-5.py
Line # Mem usage Increment Occurences Line Contents
============================================================
12 52.5 MiB 52.5 MiB 1 @profile
13 def main():
14 """
15 Main script calling func
16 """
17 60.6 MiB 8.1 MiB 1 x = func(1024**2)
I don't know if this is an issue with np.random, np.float32, the memory profiler or something else, but it's weird! Any insight much appreciated. Also, can anyone can reproduce this behavior?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
