'awswrangler flatten function: No such struct field element

I have a highly nested data frame and am getting the following error:

An error was encountered:
No such struct field element in eqp_item_type_txt, eqp_msrp_amt, eqp_make_nm, eqp_ser_num, eqp_mfg_yr, camp_unit_cmp_ftrm_prmamt, camp_unit_col_ftrm_prmamt; line 1 pos 0

using the following flatten function. Any reason why I am getting this? The schema of the dataframe is like this:

|-- a: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- b: string (nullable = true)
 |    |    |-- c: string (nullable = true)
 |    |    |-- d: array (nullable = true)
 |    |    |    |-- element: struct (containsNull = true)
 |    |    |    |    |-- d1: string (nullable = true)
 |    |    |    |    |-- d2: string (nullable = true)

Would I have to explode out the arrays first before using the flatten function?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source