'How do I iterate through nested pandas dataframe by column?
Problem: I have a 1 row dataframe dfA whose column length can vary, within each cell contains a json object. All fields in each json object are the same.
I'm trying to write code that loops through each cell, parses the JSON object, and loads ALL data into 1 single dataframe dfB, so if I have 5 columns in dfA then the final output of dfB should have 5 rows
I have the code written to where I can manually enter the column number of dfA and dfB outputs correctly:
dataA = dfA[0].values.tolist()
dfB = pd.json_normalize(dataA)
dfB
In the above example, the code pulls the 1st JSON object of dfA and correctly outputs a 1 row dataframe
I've Tried:
for column in dfA:
dataA = dfA[column].values.tolist()
dfB = pd.json_normalize(dataA)
This outputs just the last cell in dfA
I've tried different variations of this and either get the 1st cell, last cell, or the original dfA
Please help, thank you!
EDIT:
dfA:
| 0 | 1 | 2 | 3 | 4 | |
|---|---|---|---|---|---|
| 0 | {'id':'testID434556'... | {'id':'testID568912'... | {'id':'testID652154'... | {'id':'testID756298'... | {'id':'testID812362'... |
dataA = dfA[0].values.tolist()
dfB = pd.json_normalize(dataA)
dfB
| id | field2 | field3 | field4 | field5 | |
|---|---|---|---|---|---|
| 0 | testID434556 | field2data | field3data | field4data | field5data |
Desired Output:
| id | field2 | field3 | field4 | field5 | |
|---|---|---|---|---|---|
| 0 | testID434556 | field2data | field3data | field4data | field5data |
| 1 | testID568912 | field2data | field3data | field4data | field5data |
| 2 | testID652154 | field2data | field3data | field4data | field5data |
| 3 | testID756298 | field2data | field3data | field4data | field5data |
| 4 | testID812362 | field2data | field3data | field4data | field5data |
Solution 1:[1]
dfA[0] only gets the first column. You should convert you dataframe first row to list then feed the result to pd.json_normalize()
pd.json_normalize((dfA.iloc[0, :].values))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Ynjxsjmh |
