Getting individual key values from Perl JSON one at a time

Question

I have JSON code that I'm pulling with key names that are the same and I'm trying to pull the values from the keys one at a time and pass them to variables (in a loop) in a perl script but it pulls all of the values at one time instead of iterating through them. I'd like to pull a value from a key and pass it to a variable then iterate through the loop again for the next value. The amount of data changes in JSON so the amount of identical keys will grow.

Perl Script Snippet

#!/usr/bin/perl

use warnings;
use strict;
use JSON::XS;

my $res = "test.json";
my $txt = do {                             
    local $/;                              
    open my $fh, "<", $res or die $!;
    <$fh>;                                 
};

my $json = decode_json($txt);

for my $mdata (@{ $json->{results} }) {
    my $sitedomain = "$mdata->{custom_fields}->{Domain}";
    my $routerip   = "$mdata->{custom_fields}->{RouterIP}";

    #vars
    my $domain  = $sitedomain;
    my $host    = $routerip;

   print $domain;
   print $host;
}

Print $host variable

print $host;

192.168.201.1192.168.202.1192.168.203.1

Print $domain variable

print $domain;

site1.global.localsite2.global.localsite3.global.local

JSON (test.json)

{
"results": [
    {
            "id": 37,
            "url": "http://global.local/api/dcim/sites/37/",
            "display": "Site 1",
            "name": "Site 1",
            "slug": "site1",
            "custom_fields": {
                "Domain": "site1.global.local",
                "RouterIP": "192.168.201.1"
            }
     },
     {
            "id": 38,
            "url": "http://global.local/api/dcim/sites/38/",
            "display": "Site 2",
            "name": "Site 2",
            "slug": "site2",
            "custom_fields": {
                "Domain": "site2.global.local",
                "RouterIP": "192.168.202.1"
            }
      },
      {
            "id": 39,
            "url": "http://global.local/api/dcim/sites/39/",
            "display": "Site 3",
            "name": "Site 3",
            "slug": "site3",
            "custom_fields": {
                "Domain": "site3.global.local",
                "RouterIP": "192.168.203.1"
            }
      }
   ]
}

Answer 1

Your code produces expected result if you add \n to print statement. You can utilize say instead of print if there is no format required.

use warnings;
use strict;
use feature 'say';

use JSON::XS;

my $res = "test.json";
my $txt = do {                             
    local $/;                              
    open my $fh, "<", $res or die $!;
    <$fh>;                                 
};

my $json = decode_json($txt);

for my $mdata (@{ $json->{results} }) {
    my $sitedomain = "$mdata->{custom_fields}->{Domain}";
    my $routerip   = "$mdata->{custom_fields}->{RouterIP}";

    #vars
    my $domain  = $sitedomain;
    my $host    = $routerip;

    say "$domain $host";
}

The code can be re-written in shorter form as following

use strict;
use warnings;
use feature 'say';

use JSON;

my $fname = 'router_test.json';
my $txt   = do {                             
    local $/;                              
    open my $fh, "<", $fname or die $!;
    <$fh>;                                 
};

my $json = from_json($txt);

say "$_->{custom_fields}{Domain} $_->{custom_fields}{RouterIP}" for @{$json->{results}};

Answer 2

It sounds like you want to "slice" the data. You could buffer in code, or collect unique values later. Let's modify what you started with, and make some tweaks:

n.b. No need to quote my $sitedomain = "$mdata->{custom_fields}->{Domain}";. The content of the JSON is already a string, and forcing Perl to make another string by interpolating it is unnecessary.

n.b.2 JSON::XS works automatically if it's installed.

my %domains;
my %ips;

for my $mdata (@{ $json->{results} }) {
    my $sitedomain = $mdata->{custom_fields}->{Domain};
    my $routerip   = $mdata->{custom_fields}->{RouterIP};

    # Collect and count all the unique domains and IPs by storing them as hash keys
    $domains{$sitedomain} += 1;
    $ips{$routerip} += 1;
}

for my $key (keys %domains) {
    printf "%s %s\n", $key, $domains{$key};
    # and so on
}

If we don't know the custom fields, we can play with nested hashes to collect it all:

my %fields;

for my $mdata (@{ $json->{results} }) {
    for my $custom_field (keys %{ $mdata->{custom_fields} }) {
        $fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
    }
}

for my $custom_field (keys %fields) {
    print "$custom_field:\n";
    for my $unique_value (keys %{ $fields{$custom_field} }){
        printf "%s - %s\n", $unique_value, $fields{$custom_field}{$unique_value};
    }
}

Example output:

RouterIP:
192.168.201.1 - 1
192.168.203.1 - 1
192.168.202.1 - 1
Domain:
site2.global.local - 1
site1.global.local - 1
site3.global.local - 1

... or something like that. Nested structures lead very quickly to messy code. You can mitigate it by dereferencing the substructures. It could also be more predictable if we work with a known list of keys e.g.

my @known_keys = qw/RouterIP Domain/;

for my $mdata (@{ $json->{results} }) {
    for my $custom_field (@known_keys) {
        if (exists $fields{$custom_field}) {
            $fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
        }
    }
}

If the JSON file is massive you may run out of memory. For this you would need to look into a package like JSON::SL or JSON::Streaming::Reader. They're more involved to use but prevent you from needing to load the whole file into memory. There are also unix tools like jq that provide the same powers.