'How to convert a json response into yaml in bash

I read data from a json file with jq. I wanna append the results into a yaml file, but I dont get it working. I am quite new to shell programming. My goal is to append that "users" to an existing "users"-Array in a yaml file.

This is my json file:

#$DEFAULTS_FILE

{"users":
  [
    {"name":"pi",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "groups":"users,docker,video",
      "plain_text_passwd":"pi",
      "lock_passwd":"false",
      "ssh_pwauth":"true",
      "chpasswd": {"expire": false}
    },
    {"name":"admin",
      "gecos": "Hypriot Pirate",
      "sudo":"ALL=(ALL) NOPASSWD:ALL",
      "shell": "/bin/bash",
      "primary-group": "users",
      "groups":"users,docker,adm,dialout,audio,plugdev,netdev,video",
      "ssh-import-id":"None",
      "plain_text_passwd":"pi",
      "lock_passwd":"true",
      "ssh_pwauth":"true",
      "chpasswd": "{expire: false}",
      "ssh-authorized-keys": ["ssh-rsa abcdefg1234567890 [email protected]"]
    }
  ]
  }

I filter it with that:

cat $DEFAULTS_FILE | jq .users

I have no clue how to convert that json into a yaml.

My expected result should be:

users:
  - name:                pi
    gecos:               "Hypriot Pirate"
    sudo:                ALL=(ALL) NOPASSWD:ALL
    shell:               /bin/bash
    groups:              users,docker,video
    plain_text_passwd:   pi
    lock_passwd:         false
    ssh_pwauth:          true
    chpasswd: { expire:  false }
  - name:                admin
    primary-group:       users
    shell:               /bin/bash
    sudo:                ALL=(ALL) NOPASSWD:ALL
    groups:              users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:       None

I tried to use a second tool called yq which is similar to jq and can write yaml files. But I have no positive progress.

EDIT

I know that I can add content to the yaml with that:

yq w -i "my.yml" "users[+]" "some content"

But I dont know how to merge my json into that.

Any help or hint would be nice, thank you in advance...

jsonbashyamljq


Solution 1:[1]

yq a yaml wrapper for jq

With yq 4.18.1+
cat "$DEFAULTS_FILE" | yq -P    # or yq --prettyPrint

See: https://mikefarah.gitbook.io/yq/#notice-for-v4.x-versions-prior-to-4.18.1

With yq version 4.8.0:

cat $DEFAULTS_FILE | yq e -P -

  • e or eval handles file separately. ea or eval-all will merge files first.
  • -P or --prettyPrint YAML output
  • - from STDIN

Note: you can go the other way (yaml to json) too yq e -j file.yaml

With yq version 3.3.2:

cat $DEFAULTS_FILE | yq r -P -

  • r read
  • -P --prettyPrint
  • - from STDIN

Solution 2:[2]

function yaml_validate {
  python -c 'import sys, yaml, json; yaml.safe_load(sys.stdin.read())'
}

function yaml2json {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read())))'
}

function yaml2json_pretty {
  python -c 'import sys, yaml, json; print(json.dumps(yaml.safe_load(sys.stdin.read()), indent=2, sort_keys=False))'
}

function json_validate {
  python -c 'import sys, yaml, json; json.loads(sys.stdin.read())'
}

function json2yaml {
  python -c 'import sys, yaml, json; print(yaml.dump(json.loads(sys.stdin.read())))'
}

More Bash tricks at http://github.com/frgomes/bash-scripts

Solution 3:[3]

I'm not sure what rules you're using to get to your expected result. It seems like you're randomly applying different rules to how the values are being converted.

As I understand it, scalar values are just output as is (with potential encoding), objects are output as key/value pairs, and array objects are output with a - for every item. The indentation associates what's part of what.

So based on those rules if you're going to use jq:

def yamlify:
    (objects | to_entries[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify)"
        else
            "\(.key):\t\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

Then to use it, add it to your .jq file and use it:

$ jq -r yamlify input.json
users:
  - name:       pi
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    groups:     users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth: true
    chpasswd:
        expire: false
  - name:       admin
    gecos:      Hypriot Pirate
    sudo:       ALL=(ALL) NOPASSWD:ALL
    shell:      /bin/bash
    primary-group:      users
    groups:     users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:      None
    plain_text_passwd:  pi
    lock_passwd:        true
    ssh_pwauth: true
    chpasswd:   {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

Here's another variation that aligns the values

def yamlify2:
    (objects | to_entries | (map(.key | length) | max + 2) as $w |
        .[] | (.value | type) as $type |
        if $type == "array" then
            "\(.key):", (.value | yamlify2)
        elif $type == "object" then
            "\(.key):", "    \(.value | yamlify2)"
        else
            "\(.key):\(" " * (.key | $w - length))\(.value)"
        end
    )
    // (arrays | select(length > 0)[] | [yamlify2] |
        "  - \(.[0])", "    \(.[1:][])"
    )
    // .
    ;

$ jq -r yamlify2 input.json
users:
  - name:               pi
    gecos:              Hypriot Pirate
    sudo:               ALL=(ALL) NOPASSWD:ALL
    shell:              /bin/bash
    groups:             users,docker,video
    plain_text_passwd:  pi
    lock_passwd:        false
    ssh_pwauth:         true
    chpasswd:
        expire:  false
  - name:                 admin
    gecos:                Hypriot Pirate
    sudo:                 ALL=(ALL) NOPASSWD:ALL
    shell:                /bin/bash
    primary-group:        users
    groups:               users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id:        None
    plain_text_passwd:    pi
    lock_passwd:          true
    ssh_pwauth:           true
    chpasswd:             {expire: false}
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

Solution 4:[4]

yq eval -P

with mikefarah/yq version 4.0 (released December 2020), installable via most Unix-like OS package managers: via Homebrew for macOS (brew install yq), Debian with apt (apt install yq), Alpine with apk (apk add yq), etc.

See Working with JSON.

To read in json, just pass in a json file instead of yaml, it will just work - as json is a subset of yaml. However, you will probably want to use the Style Operator or --prettyPrint/-P flag to make look more like an idiomatic yaml document.

Solution 5:[5]

I've used ruby to write my json content into yaml.

As for your example, it can be achieved like this:

cat $DEFAULTS_FILE | jq .users | ruby -ryaml -rjson -e 'puts YAML.dump(JSON.parse(STDIN.read))' > my.yml

Solution 6:[6]

I suggest using yq with -y option

$ pip3 install yq # requires jq

$ cat in.json | yq -y
users:
  - name: pi
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    groups: users,docker,video
    plain_text_passwd: pi
    lock_passwd: 'false'
    ssh_pwauth: 'true'
    chpasswd:
      expire: false
  - name: admin
    gecos: Hypriot Pirate
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    primary-group: users
    groups: users,docker,adm,dialout,audio,plugdev,netdev,video
    ssh-import-id: None
    plain_text_passwd: pi
    lock_passwd: 'true'
    ssh_pwauth: 'true'
    chpasswd: '{expire: false}'
    ssh-authorized-keys:
      - ssh-rsa abcdefg1234567890 [email protected]

Solution 7:[7]

Another oneliner:

python -c 'import yaml, sys; print(yaml.dump(yaml.load(open(sys.argv[1])), default_flow_style=False))' input.json

(exploiting the fact that valid json is also valid yaml)

And yaml to json:

python -c 'import yaml, json, sys; print(json.dumps(yaml.load(open(sys.argv[1])), indent=2))' input.yaml

Solution 8:[8]

Solution in jq (without other tools)

Based on the code of @Jeff Mercado in this post I added support for multiline strings and escaping for single quotes.

# purpose: converts Json to Yaml
# remarks:
#   You can use 'yq -y' to convert json to yaml, but ...
#     * this function can be used several times within a single jq program
#     * this function may be faster than using yq
#     * maybe yq is not available in your environment
#
# input: any Json
# output: json converted to yaml
def toYaml:
   def handleMultilineString($level):
      reduce ([match("\n+"; "g")]                       # find groups of '\n'
              | sort_by(-.offset))[] as $match
             (.; .[0:$match.offset + $match.length] +
                 "\n\("    " * $level)" +               # add one extra '\n' for every group of '\n's. Add indention for each new line
                 .[$match.offset + $match.length:]);

   def toYamlString($level):
      if type == "string"
      then handleMultilineString($level)
           | sub("'"; "''"; "g")           # escape single quotes
           | "'\(.)'"                      # wrap in single quotes
      else .
      end;

   def _toYaml($level):
      (objects | to_entries[] |
          if (.value | type) == "array" then
              "\(.key):", (.value | _toYaml($level))
          elif (.value | type) == "object" then
              "\(.key):", "\("    ")\(.value | _toYaml($level))"
          else
              "\(.key): \(.value | toYamlString($level))"
          end
      )
      // (arrays | select(length > 0)[] | [_toYaml($level)] |
          "  - \(.[0])", "\("    ")\(.[1:][])"
      )
      // .;

   _toYaml(1);

Example usage

File 'containsMultilineStrings.json'

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

jq -r 'toYaml' < containsMultilineStrings.json

response:
    code: 200
    message: 'greeting

    that''s all folks



    '

jq -r 'toYaml' containsMultilineStrings.json | yq (roundtrip)

{
  "response": {
    "code": 200,
    "message": "greeting\nthat's all folks\n\n\n"
  }
}

Test

You can test the correctness of the function toYaml by converting json to yaml and than back to json using yq.

FILE='containsMultilineStrings.json'; diff <(cat "$FILE") <(jq -r 'toYaml' $FILE | yq)

Performance

A quick benchmark shows a reduced runtime of the function toYaml compared to the use of yq. On my computer, I measured:

time for i in {1..100}; do yq -y > /dev/null < containsMultilineStrings.json; done

8.4 sec

time for i in {1..100}; do jq -r 'toYaml' > /dev/null containsMultilineStrings.json; done

3.4 sec

Solution 9:[9]

Another option is to use gojq. It is a port of jq with support for reading and writing yaml. It can be installed via GitHub releases, homebrew, and zero install. The command for your question would be:

cat test.json | gojq --yaml-output > test.yaml

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 ryenus
Solution 2
Solution 3
Solution 4
Solution 5 Pavel
Solution 6 botchniaque
Solution 7
Solution 8
Solution 9 Moritz

Related Questions