How do I run multiple subprocesses in parallel and wait for them to finish in Python

Question

I am trying to migrate a bash script to Python.

The bash script runs multiple OS commands in parallel then waits for them to finish before resuming, ie:

command1 &

command2 &

.

commandn &

wait

command

I want to achieve the same using Python subprocess. Is this possible? How can I wait for a subprocess.call command to finish before resuming?

Answer 1

You can still use Popen which takes the same input parameters as subprocess.call but is more flexible.

subprocess.call: The full function signature is the same as that of the Popen constructor - this functions passes all supplied arguments directly through to that interface.

One difference is that subprocess.call blocks and waits for the subprocess to complete (it is built on top of Popen), whereas Popen doesn't block and consequently allows you to launch other processes in parallel.

Try the following:

from subprocess import Popen
commands = ['command1', 'command2']
procs = [ Popen(i) for i in commands ]
for p in procs:
   p.wait()

Answer 2

Expanding on Aaron and Martin's answer, here is a solution that runs uses subprocess and Popen to run n processes in parallel:

import subprocess
commands = ['cmd1', 'cmd2', 'cmd3', 'cmd4', 'cmd5'] 
n = 2 #the number of parallel processes you want
for j in range(max(int(len(commands)/n), 1)):
    procs = [subprocess.Popen(i, shell=True) for i in commands[j*n: min((j+1)*n, len(commands))] ]
    for p in procs:
        p.wait()

I find this to be useful when using a tool like multiprocessing could cause undesired behavior.