Using string with literal string interpolation as an argument to python subprocess

Question

I'm trying to run shell execution (xcodebuild) with python subprocess.

The desired command is something like below:

xcodebuild -workspace SomeWorkspace.xcworkspace -scheme SomeScheme ...

It takes several arguments with values, and I tried to create them using literal string interpolation.

args=[
    "xcodebuild",
    f"-workspace {workspace}",
    f"-scheme {scheme}",
    ...
]

But when I pass them to subprocess, xcodebuild fails with following message :

xcodebuild: error: invalid option '-workspace SomeWorkspace.xcworkspace'

If I split those keyword-value pairs and appending them to the list, everything works as expected.

So I guess that python subprocess treats each argument elements as a 'single' argument even if it contains space. But I'm wondering how python interpret those arguments and supply them to shell. Is f-literal string treated differently? It's just a string, eventually...

I'm passing these arguments to subprocess as below:

process = subprocess.Popen(
    args=args,
    shell=False,
    ...

---

I found on the python doc which refers exactly this case :

Note in particular that options (such as -input) and arguments (such as eggs.txt) that are separated by whitespace in the shell go in separate list elements, while arguments that need quoting or backslash escaping when used in the shell (such as filenames containing spaces or the echo command shown above) are single list elements.

So this is an intended behavior and I'd have to read shlex.split() for more detail.

! Python doc is somewhat difficult to read(especially for non-English person like myself). Thanks for all your help.

Answer 1

args is very particular that each list element ends up a single argument to the program you're calling. In your example it you end up with just 2 arguments that would be like '-workspace a_workspace' and '-scheme a_scheme'.

What might help you in debugging this is to test against a testing python program that can show you what it is seeing as arguments: We'll call it arg_test.py

import sys
for i, arg in enumerate(sys.argv):
    print(f"{i}: '{arg}'")

Then, you do in your program:

workspace = 'a_workspace'
scheme = 'a_scheme'
args=[
    f"-workspace {workspace}",
    f"-scheme {scheme}",
]
subprocess.Popen(['python', 'arg_test.py'] + args, shell=False)

You'll see:

0: 'arg_test.py'
1: '-workspace a_workspace'
2: '-scheme a_scheme'

Your fix:

args=[
    "-workspace",
    workspace,
    "-scheme",
    scheme,
    ...
]

once you make the fix, the test program should output:

0: 'arg_test.py'
1: '-workspace'
2: 'a_workspace'
3: '-scheme'
4: 'a_scheme'