Forcing cURL to get a password from the environment

Question

This question about using cURL with a username and password has suboptimal answers for me:

1. curl -u "user:pw" https://example.com puts the pw in the process list
2. curl "https://user:[email protected]" puts the pw in the process list
3. curl -u "user:$(cat ~/.passwd)" https://example.com puts the pw in the process list
4. curl -u user https://example.com prompts for the pw
5. curl --netrc-file ~/.netrc https://example.com requires a file

#4 is secure, but I might run this command hundreds of times a day, so it's tedious. #5 is close to secure, but that file could be read by somebody with root access.

The cURL man page says (note the bold text):

-u/--user <user:password>

Specify the user name and password to use for server authentication. Overrides -n/--netrc and --netrc-optional.

If you just give the user name (without entering a colon) curl will prompt for a password.

If you use an SSPI-enabled curl binary and do NTLM authentication, you can force curl to pick up the user name and password from your environment by simply specifying a single colon with this option: -u :.

I've tried setting $USER and $PASSWORD (and $CURLOPT_PASSWORD and others) in the environment, but cURL doesn't pick up either of them when invoked as curl -u : https://example.com (nor does it work without the -u :).

I'm not doing NTLM, so this doesn't work. Unless I'm missing something.

 

Is there a way to pass credentials to curl solely through the environment?

 

_{(Workaround moved to an answer)}

Answer 1

Is there a way to pass credentials to curl solely through the environment?

No, I don't think there is.

The CURLOPT_USERPWD documentation I think describes what you need, but this is an option that would be available using the curl library in some other language. PHP, Perl, C, etc.

The curl binary you run from your shell is just another front end on that library, but the way things like CURLOPT_USERPWD get passed to the library through the curl binary is by use of command line options on the binary.

You could theoretically write your own binary as a front end to the curl library, and write in support for environment variables.

You could alternately hack environment support as you're hoping to see it into the existing curl binary, and compile your own with local functions.

Beware, though, that even environment variables may be leaked by your shell into the process table. (What do you see when you run ps ewwp $$?)

Perhaps a .netrc file with restricted permissions will be the safest way to go. Perhaps you will need to generate a temporary .netrc file to be used by the --netrc-file curl option.

I think you either have to pick the least risky solution for your environment, or write something in a real language that does security properly.

Answer 2

User "Tom, Bom" provides a decent solution for this here: https://coderwall.com/p/dsfmwa/securely-use-basic-auth-with-curl

curl --config - https://example.com <<< 'user = "username:password"'

This prevents passwords from showing up in the process list, though this does not specifically address the OP's original question:

Is there a way to pass credentials to curl solely through the environment?

I still give points to @ghoti for giving a more comprehensive and informative answer.

Answer 3

Previous answers are correct, the best option is using -n for curl(assuming you on linux):

1. create a file (use your own favorite editor)

vi ~YOUR_USER_NAME/.netrc

2. add the followings

machine example.com login YOUR_USER_NAME password THE_ACTUAL_PASSWORD

3. run

curl -n https://example.com/some_end_point

Answer 4

Curl uses SPACE and TAB as delimiters when it parses the tokens in the netrc file:

https://github.com/curl/curl/blob/bc5a0b3e9f16a431523ae54822adc38c3a396a26/lib/netrc.c#L122

The --netrc-file approach therefore can't handle a SPACE or TAB in the password.

Test of SPACE in password

SRV="httpbin.org"
URL="https://$SRV/basic-auth/username/pass%20word"
USERNAME=username
PASSWORD='pass word'
curl -v --netrc-file <(echo "machine $SRV login $USERNAME password $PASSWORD") "$URL"

Result: ? FAIL

WARNING: If your shell's echo command is not a builtin, the above curl invocation will leak $PASSWORD into the process table momentarily. In bash, whether echo is a builtin or not can be tested with type -t echo. WORKAROUND: Use cat and a here-string: Replace <(echo "string") with <(cat <<< "string"). This warning applies to all the examples in this answer.

Test of TAB in password

SRV="httpbin.org"
URL="https://$SRV/basic-auth/username/pass%09word"
USERNAME=username
PASSWORD=$'pass\tword'
curl -v --netrc-file <(echo "machine $SRV login $USERNAME password $PASSWORD") "$URL"

Result: ? FAIL

Test of double quotation mark " in password

SRV="httpbin.org"
URL="https://$SRV/basic-auth/username/pass%22word"
USERNAME=username
PASSWORD='pass\"word'
curl -v --netrc-file <(echo "machine $SRV login $USERNAME password $PASSWORD") "$URL"

Result: ? FAIL

A more robust way

@sfgeorge correctly points out that the -K, ?--config <file> option, with <file> set to -, could be used to supply the password on STDIN. Using STDIN for this purpose, however, would preclude using STDIN for other purposes, like to POST data using --data @-.

Fortunately, we can use process substitution instead of STDIN. A process substitution expands to a filename and can be used where a filename is expected.

Test of SPACE in password

USERNAME=username
PASSWORD='pass word'
curl -v \
  -K <(echo "user: \"$USERNAME:$PASSWORD\"") \
  "https://httpbin.org/basic-auth/username/pass%20word"

Result: ? SUCCESS

Test of TAB in password

USERNAME=username
PASSWORD=$'pass\tword'
curl -v \
  -K <(echo "user: \"$USERNAME:$PASSWORD\"") \
  "https://httpbin.org/basic-auth/username/pass%09word"

Result: ? SUCCESS

And, for extra robustness, let's make it handle double quotation marks in the password as well.

Test of double quotation mark " in password

USERNAME=username
PASSWORD=$'p?a s\ts"wo$rd'

# Build 'user' option
USER_OPT="$USERNAME:$PASSWORD"
USER_OPT=${USER_OPT//\\/\\\\} # Escape `\`
USER_OPT=${USER_OPT//\"/\\\"} # Escape `"`
USER_OPT="user: \"${USER_OPT}\""

curl -v \
  -K <(echo "$USER_OPT") \
  "https://httpbin.org/basic-auth/username/p?a%20s%09s%22wo%24rd"

Result: ? SUCCESS

I threw in an emoji for good measure.

Answer 5

I'd like to point out that even bash here-string syntax (<<<hello) creates a file in /tmp that, while it's only there for an instant, is still susceptible to timing attacks. Running strace bash -c '/bin/cat <<<hello' reveals how it works:

19376 open("/tmp/sh-thd-1651575757", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0600) = 3
19376 write(3, "hello", 5)              = 5
19376 write(3, "\n", 1)                 = 1
19376 open("/tmp/sh-thd-1651575757", O_RDONLY) = 4
19376 close(3)                          = 0
19376 unlink("/tmp/sh-thd-1651575757")  = 0
19376 dup2(4, 0)                        = 0
19376 close(4)                          = 0