'PostgreSQL JsonB Update, Delete arbitrary attributes across multiple records

I have a weather-data table that stores hourly records of various sensor readings where the JSONB column holds these readings as a JSON object. There's about 5 million of these records and building and has the following example form:

id date_time met_data
482506 "2003-08-16 01:00:00+00" "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}"
482507 "2003-08-16 02:00:00+00" "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}"
482508 "2003-08-16 03:00:00+00" "{""msl"": 1013.6, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1}"
482509 "2003-08-16 04:00:00+00" "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}"

I need to be able to arbitrarily iterate over the JSON data and remove the attributes set to -1.

Using this SQL:

 SELECT * FROM met_records JOIN jsonb_each(met_records.met_data) e ON true WHERE e.value = '-1';

I can find all the IDs and attributes set to -1 with returns like this:

id date_time met_data key value
482506 "2003-08-16 01:00:00+00" "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}" rain -1
482506 "2003-08-16 01:00:00+00" "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}" iwetb -1
482507 "2003-08-16 02:00:00+00" "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" rain -1
482507 "2003-08-16 02:00:00+00" "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" irain -1
482507 "2003-08-16 02:00:00+00" "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" itemp -1
482508 "2003-08-16 03:00:00+00" "{""msl"": 1013.6, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1}" iwetb -1
482509 "2003-08-16 04:00:00+00" "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}" temp -1
482509 "2003-08-16 04:00:00+00" "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}" wetb -1

I'm now trying to use this SQL to update each row to remove the -1 attributes:

UPDATE met_records AS U SET met_data = U.met_data - S.key::text FROM (SELECT * FROM met_records JOIN jsonb_each_text(met_records.met_data) e ON true WHERE e.value = '-1') AS S
WHERE U.id = S.id;

Is there a faster/better way to do this? I'm running this SQL in PGAdmin 4 and its incredibly slow so I'm not sure if thats a slow SQL thing or a PGAdmin SSL connection-drop thing or something else. When it does run and returns a result of X records updated, when I check the table for more -1s there's still lots there. So I run the UPDATE SQL again and another X number of records are updated, etc.... slowly though I do manage to reduce and remove all -1s with that SQL.

When I had the SQL as 'SET met_data = S.met_data - S.key::text' I realised I was removing a -1 attribute on 1 iteration but putting it back in on the next if the id had more than one -1 attribute, I'm not sure though if I'm still doing something like this!!



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source