'Python's memory-profiler reveals np.random taking up too much memory

I am using Python's memory-profiler to better understand how much memory my code is using and when setting up this simple example, I found a much higher memory usage than expected:

import numpy as np
from memory_profiler import profile

@profile
def func(n):
    x1 = np.random.random(n).astype(np.float32)
    x2 = np.random.random(n).astype(np.float32)
    x_sum = x1 + x2
    return x_sum


@profile
def main():
    """
    Main script calling func
    """
    x = func(1024**2)


if __name__ == "__main__":
    main()

Output:

Filename: mem-test-5.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     4     52.7 MiB     52.7 MiB           1   @profile
     5                                         def func(n):
     6     56.8 MiB      4.1 MiB           1       x1 = np.random.random(n).astype(np.float32)
     7     68.7 MiB     11.9 MiB           1       x2 = np.random.random(n).astype(np.float32)
     8     68.7 MiB      0.0 MiB           1       x_sum = x1 + x2
     9     68.7 MiB      0.0 MiB           1       return x_sum


Filename: mem-test-5.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    12     52.7 MiB     52.7 MiB           1   @profile
    13                                         def main():
    14                                             """
    15                                             Main script calling func
    16                                             """
    17     64.8 MiB     12.1 MiB           1       x = func(1024**2)

x1 has the size expected (1024 x 1024 x 4 = 4194304 bytes = 4 MiB) but why does creating x2 right after take up 12 MiB when it should also be only 4 MiB? Interestingly, this problem does not occur if using np.ones instead of np.random or if not casting x1 and x2 to float32 (that is, leaving them at default float64):

Filename: mem-test-5.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
     4     52.5 MiB     52.5 MiB           1   @profile
     5                                         def func(n):
     6     60.4 MiB      7.9 MiB           1       x1 = np.random.random(n)
     7     68.6 MiB      8.1 MiB           1       x2 = np.random.random(n)
     8     76.6 MiB      8.0 MiB           1       x_sum = x1 + x2
     9     76.6 MiB      0.0 MiB           1       return x_sum


Filename: mem-test-5.py

Line #    Mem usage    Increment  Occurences   Line Contents
============================================================
    12     52.5 MiB     52.5 MiB           1   @profile
    13                                         def main():
    14                                             """
    15                                             Main script calling func
    16                                             """
    17     60.6 MiB      8.1 MiB           1       x = func(1024**2)

I don't know if this is an issue with np.random, np.float32, the memory profiler or something else, but it's weird! Any insight much appreciated. Also, can anyone can reproduce this behavior?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source