Upon finishing a multiprocessing pool in Python, is there a way to combine global variables across all sub-processes?

Question

I figure one solution could be to use a shared memory resource to access the same variables across all sub-processes, but I really only care about the final sum once all processes are finished. Is there a straightforward way to iterate across the finished processes and access specific variables in each one?

Here's a modified snippet of the multiprocessing code:

pool = Pool(cpu_count())
results = list(tqdm.tqdm(pool.imap_unordered(Article.analyze_pool, articles), total=len(articles)))
pool.close()
pool.join()

Answer 1

You can return the values you want to access from the Article.analyze_pool function and they will be available in results. Since you are using imap_unordered() they will be in a different order than articles, watch out for that. You can either also return an identifier of Article from Article.analyze_pool or use the ordered equivalent imap().