'How to find min of each column attribute in pyspark sql rows type data in python given as dictionary to a column of another dataframe

I've the data extracted from a pickle file and it has column which have data of type dictionary. The data is of the following format: Orders

ID Data
1006 {'0': Rows(amount=100, quantity=10), '1': Rows(amount=95, quantity = 2)}
1007 {'0': Rows(amount=100, quantity=10), '1': Rows(amount=95, quantity = 2), '2': Rows(amount=1000, quantity = 3) }
1008 {'0': Rows(amount=100, quantity=10)

The required output is to get the minimum of amount and quantity corresponding to each id:

ID Data Amount Quantity
1006 {'0': Rows(amount=100, quantity=10), '1': Rows(amount=95, quantity = 20)} 95 10
1007 {'0': Rows(amount=322, quantity=17), '1': Rows(amount=196, quantity = 12), '2': Rows(amount=1000, quantity = 3) } 196 3
1008 {'0': Rows(amount=100, quantity=10) 100 10

The noted point is the type of value of key value pair present in Data column is pyspark.sql.types.Row and the key values can be variable for each row of dataframe Orders.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source