Memory error when running scipy with multiprocess under Windows 10

Question

I write a program that will use concurrent futures ProcessPoolExecutor. When I set the worker number to 2, it runs smooth. But after I increase the worker number, different type of exception starting to throw.

It would help if I increase the virtual memory. Let say, when I just boot the PC, I can run at 6 workers without error. But later, even 4 workers start to raise exception.

My PC is new, Gen 12 i5 with 16GB memory. When the error raise, the memory usage is less than 60%.

The error happen so far

• scipy OSError: [WinError 1455]
• ImportError: DLL load failed while importing _flow. DLL load failed while importing _flow. Page size too small
• ImportError: DLL load failed while importing cython_lapack. Page size too small

Basically, I just use the sample code from python official document here, and I set the workers number to 6. The execution is smooth, but when I add two import line to the program import scipy.optimize and import scipy.stats, it starts to fail.

Both Python 3.9 and Python 3.10 fail in my PC.

import scipy.optimize
import scipy.stats

import concurrent.futures
import math

PRIMES = [
             112272535095293,
             112582705942171,
             112272535095293,
             115280095190773,
             115797848077099,
             1099726899285419] * 10


def is_prime(n):
    if n < 2:
        return False
    if n == 2:
        return True
    if n % 2 == 0:
        return False

    sqrt_n = int(math.floor(math.sqrt(n)))
    for i in range(3, sqrt_n + 1, 2):
        if n % i == 0:
            return False
    return True


def main():
    with concurrent.futures.ProcessPoolExecutor(6) as executor:
        for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
            print('%d is prime: %s' % (number, prime))


if __name__ == '__main__':
    main()