Filtering Live json Stream on linux

Question

I have live raw json stream data from the virtual radar server I'm using.

I use Netcat to fetch the data and jq to save it on my kali linux. using the following command.

nc 127.0.0.1 30006 | jq > A7.json

But i want to filter specific content from the data stream.

i use the following command to extract the data.

cat A7.json | jq '.acList[] | select(.Call | contains("QTR"))' - To fetch Selected airline

But i realized later that the above command only works once. in other words, it does not refresh. as the data is updating every second. i have to execute the command over and over again to extract the filter data which is causing to generate duplicate data.

Can someone help me how to filter the live data without executing the command over and over .

Answer 1

Break Down

Why I did what I did?

---

• I have live raw json stream data from the virtual radar server, which is running on my laptop along with Kali Linux WSL on the background.

• For those who don't know virtual radar server is a Modes-s transmission decoder that is used to decode different ADS-B Formats. Also, it rebroadcast the data in a variety of formats.. one of them is the Json stream. And I want to save select aircraft data in json format on Kali Linux.

• I used the following commands to save the data before

$ nc 127.0.0.1 30001 | jq > A7.json - To save the stream.

$ cat A7.json | jq '.acList[] | select(.Call | contains("QTR"))' - To fetch Selected airline

• But I realized two things after using the above. One I'm storing unwanted data which is consuming my storage. Two, when I used the second command it just goes through the json file once and produces the data which is saved at that moment and that moment alone. which caused me problems as I have to execute the command over and over again to extract the filter data which is causing me to generate duplicate data.

Command worked for me

---

The following command worked flawlessly for my problem.

$ nc localhost 30001 | sudo jq --unbuffered '.acList[] | select (.Icao | contains("800CB8"))' > A7.json

The following also caused me some trouble which i explain clearly down below.

Errors X Explanation

---

This error was resulted in the missing the Field name & key in the json object.

$ nc localhost 30001 | sudo jq --unbuffered '.acList[] | select (.Call | contains("IAD"))' > A7.json

#OUTPUT

jq: error (at <stdin>:0): null (null) and string ("IAD") cannot have their containment checked

If you see the below JSON data you'll see the missing Field name & key which caused the error message above.

{
  "Icao": "800CB8",
  "Alt": 3950,
  "GAlt": 3794,
  "InHg": 29.7637787,
  "AltT": 0,
  "Call": "IAD766",
  "Lat": 17.608658,
  "Long": 83.239166,
  "Mlat": false,
  "Tisb": false,
  "Spd": 209,
  "Trak": 88.9,
  "TrkH": false,
  "Sqk": "",
  "Vsi": -1280,
  "VsiT": 0,
  "SpdTyp": 0,
  "CallSus": false,
  "Trt": 2
}
{
  "Icao": "800CB8",
  "Alt": 3950,
  "GAlt": 3794,
  "AltT": 0,
  "Lat": 17.608658,
  "Long": 83.239166,
  "Mlat": false,
  "Spd": 209,
  "Trak": 88.9,
  "Vsi": -1280
}
{
  "Icao": "800CB8",
  "Alt": 3800,
  "GAlt": 3644,
  "AltT": 0,
  "Lat": 17.608795,
  "Long": 83.246155,
  "Mlat": false,
  "Spd": 209,
  "Trak": 89.2,
  "Vsi": -1216
}

Commands that didn't work for me.

---

Command #1

When i used jq with --stream with the filter it produced the below output. --Stream with output filter worked without any errors.

$ nc localhost 30001 | sudo jq --stream '.acList[] | select (.Icao | contains("800"))' > A7.json

#OUTPUT

jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"
jq: error (at <stdin>:0): Cannot index array with string "acList"

Command #2

For some reason -k -l didn't work to listen to the data. but the other command worked perfectly. i think it didn't work because of the existing port outside of the wsl.

$ nc -k -l localhost 30001

$ nc localhost 30001

Thank you to everyone who helped me to solve my issue. I'm very grateful to you guys

Answer 2

As you don't use the --stream option, I suppose your document is a regular JSON document.

To execute your command every second, you can implement a loop that sleeps for 1 second:

while true; do sleep 1; nc 127.0.0.1 30006 | jq '.acList[] | select(…)'; done

To have the output on the screen and also save to a file (like you did with A7.json), you can add a call to tee:

# saves the document as returned by `nc` but outputs result of `jq`
while true; do sleep 1; nc 127.0.0.1 30006 | tee A7.json | jq '.acList[] | …'; done

# saves the result of `jq` and outputs it
while true; do sleep 1; nc 127.0.0.1 30006 | jq '.acList[] | …' | tee A7.json; done

Answer 3

Can you try this ?

nc localhost 30006 | tee -a A7.json |
   while true; do
      stdbuf -o 0 jq 'try (.acList[] | select(.Call | contains("QTR")))' 2>/dev/null
   done

Answer 4

Assuming that no other process is competing for the port, I'd suggest trying:

nc -k -l localhost 30006 | jq --unbuffered ....

Or if you want to keep a copy of the output of the netcat command:

nc -k -l localhost 30006 | tee A7.json | jq --unbuffered ....

You might want to use tee -a A7.json instead.