'PostgreSQL JsonB Update, Delete arbitrary attributes across multiple records
I have a weather-data table that stores hourly records of various sensor readings where the JSONB column holds these readings as a JSON object. There's about 5 million of these records and building and has the following example form:
| id | date_time | met_data |
|---|---|---|
| 482506 | "2003-08-16 01:00:00+00" | "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}" |
| 482507 | "2003-08-16 02:00:00+00" | "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" |
| 482508 | "2003-08-16 03:00:00+00" | "{""msl"": 1013.6, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1}" |
| 482509 | "2003-08-16 04:00:00+00" | "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}" |
I need to be able to arbitrarily iterate over the JSON data and remove the attributes set to -1.
Using this SQL:
SELECT * FROM met_records JOIN jsonb_each(met_records.met_data) e ON true WHERE e.value = '-1';
I can find all the IDs and attributes set to -1 with returns like this:
| id | date_time | met_data | key | value |
|---|---|---|---|---|
| 482506 | "2003-08-16 01:00:00+00" | "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}" | rain | -1 |
| 482506 | "2003-08-16 01:00:00+00" | "{""msl"": 1021.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1, ""vappr"": 11.1}" | iwetb | -1 |
| 482507 | "2003-08-16 02:00:00+00" | "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" | rain | -1 |
| 482507 | "2003-08-16 02:00:00+00" | "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" | irain | -1 |
| 482507 | "2003-08-16 02:00:00+00" | "{""msl"": 1020.9, ""rain"": -1, ""rhum"": 95, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": -1, ""itemp"": -1, ""iwetb"": 0, ""vappr"": 11.1}" | itemp | -1 |
| 482508 | "2003-08-16 03:00:00+00" | "{""msl"": 1013.6, ""temp"": 9.2, ""wetb"": 8.9, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": -1}" | iwetb | -1 |
| 482509 | "2003-08-16 04:00:00+00" | "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}" | temp | -1 |
| 482509 | "2003-08-16 04:00:00+00" | "{""msl"": 1020.2, ""rain"": 0, ""rhum"": 95, ""temp"": -1, ""wetb"": -1, ""dewpt"": 8.5, ""irain"": 0, ""itemp"": 0, ""iwetb"": 0, ""vappr"": 11.1}" | wetb | -1 |
I'm now trying to use this SQL to update each row to remove the -1 attributes:
UPDATE met_records AS U SET met_data = U.met_data - S.key::text FROM (SELECT * FROM met_records JOIN jsonb_each_text(met_records.met_data) e ON true WHERE e.value = '-1') AS S
WHERE U.id = S.id;
Is there a faster/better way to do this? I'm running this SQL in PGAdmin 4 and its incredibly slow so I'm not sure if thats a slow SQL thing or a PGAdmin SSL connection-drop thing or something else. When it does run and returns a result of X records updated, when I check the table for more -1s there's still lots there. So I run the UPDATE SQL again and another X number of records are updated, etc.... slowly though I do manage to reduce and remove all -1s with that SQL.
When I had the SQL as 'SET met_data = S.met_data - S.key::text' I realised I was removing a -1 attribute on 1 iteration but putting it back in on the next if the id had more than one -1 attribute, I'm not sure though if I'm still doing something like this!!
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
