Using yq in for loop bash

Question

I have a yaml array like below,

identitymappings:
- arn: "arn:aws:iam::12345567:role/AdmRole"
  group: "system:masters"
  user: "user1"
- arn: "arn:aws:iam::12345567:role/TestRole"
  group: "system:masters"
  user: "user2"

I am trying to parse this yaml in a bash script using for loop and yq.

 for identityMapping in $(yq read test.yaml "identitymappings[*]"); do
      roleArn=$identityMapping["arn"]
      group=$identityMapping.group
      user=$identityMapping.user
done

But I am not getting the expected results like not able to fetch the values of roleArn,group,user. Please let me know how to fix this.

Answer 1

The way I would do it is:

# load array into a bash array
# need to output each entry as a single line
readarray identityMappings < <(yq e -o=j -I=0 '.identitymappings[]' test.yml )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

output:

roleArn: arn:aws:iam::12345567:role/AdmRole
roleArn: arn:aws:iam::12345567:role/TestRole

Disclaimer: I wrote yq

Answer 2

The easiest way to read from jq or yq into bash is to use a BashFAQ #1 while read loop to handle line-oriented data; in the below, we use @tsv to generate line-oriented output:

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq -j read test.yaml \
         | jq -r '.identitymappings[] | [.arn, .group, .user] | @tsv')

Note that if you were using the Python yq rather than the Go one, you could remove the yq -j read and just use yq -r '...' in place of jq -r '...'.

Answer 3

There is an improvement of @Rad4's answer that worked for me.

You can neatly loop through using latest yq and jq via:

for im in $(yq eval -o=j test.yaml | jq -cr '.identitymappings[]'); do
      arn=$(echo $im | jq -r '.arn' -)
      group=$(echo $im | jq -r '.group' -)
      user=$(echo $im | jq -r '.user' -)
      echo $arn $group $user
done

This loops through valid online jsons, which makes jq still work inside the loop.

Answer 4

I wasn't able to comment on Charles Duffy's proper answer, but this works for yq v4 without the use of jq...

while IFS=$'\t' read -r roleArn group user _; do
  echo "Role:  $roleArn"
  echo "Group: $group"
  echo "User:  $user"
done < <(yq e '.identitymappings[] | [.arn, .group, .user] | @tsv' test.yaml)

Answer 5

The answer by @mike.f is a good one. However, it does not work on OSX machines, because readarray is not an available command. You can read more about this here.

Here is the equivalent that would work on a mac:

# load array into a bash array
# need to output each entry as a single line
identitymappings=( $(yq e -o=j -I=0 '.identitymappings[]' test.yml ) )

for identityMapping in "${identityMappings[@]}"; do
    # identity mapping is a yaml snippet representing a single entry
    roleArn=$(echo "$identityMapping" | yq e '.arn' -)
    echo "roleArn: $roleArn"
done

Answer 6

I figured out..

for identityMapping in $(yq read test.yaml -j "identitymappings[*]"); do
      echo $identityMapping
      roleArn= echo $identityMapping | jq -r '.arn'
      echo $roleArn
      group= echo $identityMapping | jq -r '.group'
      echo $group
      user= echo $identityMapping | jq -r '.user'
      echo $user