'Python: Using multiprocessing is much slower than loop for optimisation problem. What am I doing wrong?
An obligatory assurance that I have read the many posts on the topic before posting this. I'm aware that multiprocessing entails a fixed cost, but to the best of my knowledge this doesn't seem to be the problem here.
I basically have a number of separate optimisation problems, and want to solve them in parallel. The following code is a simple example:
import psutil
import multiprocessing as mp
import time
from scipy.optimize import minimize
import numpy as np
pset = np.random.uniform(-10,10,500)
def func(x,p):
out= (x-p)**2
return out
def object(p):
def func2(x):
return func(x,p)
output = minimize(func2,
x0,
method = 'trust-constr')
xstar = output.x
return xstar
# 1. Loop
tic = time.perf_counter()
out_list = []
x0 = 0
for p in pset:
xstar = object(p)
out_list.append(xstar)
#print(np.vstack(out_list))
toc = time.perf_counter()
print(f'Loop done in {toc - tic:0.4f} seconds')
# 2. Pool
n_cpu = psutil.cpu_count(logical = False)
if __name__ == '__main__':
pool = mp.Pool(n_cpu)
#results = pool.map_async(object, pset).get()
results = pool.map(object,pset)
pool.close()
pool.join()
#print(np.vstack(results))
toc2 = time.perf_counter()
print(f'Pool done in {toc2 - toc:0.4f} seconds')
As you can see, the 'pool' method takes longer and increasingly so the more problems there are to solve (hence my conjecture that this isn't a fixed cost issue). My optimisation problem is actually a lot more complicated than this, and while loop will take a few minutes to solve say 3 problems, 'pool' will keep on running for a long long time, at least 15 minutes before I decide to force terminate.
What could be the reason for this inferior performance? Is there a problem with using parallel computing for optimisation problems? What other tricks could I try to speed things up?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
