'Python: Using multiprocessing is much slower than loop for optimisation problem. What am I doing wrong?

An obligatory assurance that I have read the many posts on the topic before posting this. I'm aware that multiprocessing entails a fixed cost, but to the best of my knowledge this doesn't seem to be the problem here.

I basically have a number of separate optimisation problems, and want to solve them in parallel. The following code is a simple example:

import psutil
import multiprocessing as mp
import time
from scipy.optimize import minimize
import numpy as np

pset = np.random.uniform(-10,10,500)

def func(x,p):
    out= (x-p)**2
    return out

def object(p):
    def func2(x):
        return func(x,p)    
    output = minimize(func2,
                  x0,
                  method = 'trust-constr')
    xstar = output.x
    return xstar

# 1. Loop
tic = time.perf_counter()
out_list = []
x0 = 0

for p in pset:
    xstar = object(p)
    out_list.append(xstar)

#print(np.vstack(out_list))

toc = time.perf_counter()
print(f'Loop done in {toc - tic:0.4f} seconds')

# 2. Pool
n_cpu = psutil.cpu_count(logical = False)
if __name__ == '__main__':
    pool = mp.Pool(n_cpu)
    #results = pool.map_async(object, pset).get()
    results = pool.map(object,pset)
    pool.close()
    pool.join()
    #print(np.vstack(results))

toc2 = time.perf_counter()
print(f'Pool done in {toc2 - toc:0.4f} seconds')

As you can see, the 'pool' method takes longer and increasingly so the more problems there are to solve (hence my conjecture that this isn't a fixed cost issue). My optimisation problem is actually a lot more complicated than this, and while loop will take a few minutes to solve say 3 problems, 'pool' will keep on running for a long long time, at least 15 minutes before I decide to force terminate.

What could be the reason for this inferior performance? Is there a problem with using parallel computing for optimisation problems? What other tricks could I try to speed things up?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source