'bash script to curl all urls in array

i have an array of urls like (coming out of a jq call):

["https://example.com/text.txt", "https://example.com/text2.txt", "https://example.com/text3.txt"]

how to loop through this and curl each of these urls into the current folder?

edit:

for link in $(jq -c 'xxxx' file.txt); do
    echo "$link"
  done
bashparsingcurljq


Solution 1:[1]

if you are calling a command, to get the list of links, you can simply iterate over the result of the command, something like:

for link in "$(jq <args...>)"; do
    # do something here
done

or:

while read -r link; do
    # do something here
done <<< $(jq <args...>)

Solution 2:[2]

As the JSON array is "coming out of a jq call", just have jq output it as a list of raw-text urls using .[] and the -r option, then pipe that list into xargs using curl.

jq -r '… | .[]' | xargs -I{} curl -O {}

If you cannot modify the original jq filter, insert another call to jq to your pipe:

… | jq -r '.[]' | xargs -I{} curl -O {}

Solution 3:[3]

Add the following to the end of your jq code, and pipe to sh to run it:

jq -r '[ "curl", .[] ] | @sh' | sh

  • The @sh filter correctly escapes output for a shell

  • A single shell command is an array/list, delimited by whitespace: cmd opts ... args ..., and curl can take multiple URL arguments.

  • So in jq, you can prepend the curl command (and any options) as extra elements in the array, and filter through @sh, to generate a correctly quoted shell command.

  • Note that the order of elements in a JSON array is preserved, which is what we want - curl first.

  • You can run jq without the pipe, to inspect what will be executed:

For example:

echo '["https://example.com/text.txt", "https://example.com/text2.txt", "https://example.com/text3.txt"]' |
jq -r '[ "curl", .[] ] | @sh'

Should produce:

'curl' 'https://example.com/text.txt' 'https://example.com/text2.txt' 'https://example.com/text3.txt'

The quotes around curl are fine. This output can be piped (in the shell) to sh, to execute.

Solution 4:[4]

This will create 3 files in the current directory:

echo '["https://example.com/text.txt", "https://example.com/text2.txt", "https://example.com/text3.txt"]' | jq '.[]' | xargs -n 1 curl -O

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1
Solution 2 pmf
Solution 3
Solution 4 Delta George

Related Questions