'How to drop all columns with type NullType in Spark Scala?

I want to drop all columns with data type NullType including columns that exist in nested columns(ArrayType or StructType). I found a solution (link) that drop columns based on their data type but it does not work on nested columns!

How can I somehow change below script that works on nested columns as well?

df.schema.fields
    .collect({case x if x.dataType.typeName == "null" => x.name})
    .foldLeft(df)({case(dframe,field) => dframe.drop(field)})

Thank you in advance.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'How to drop all columns with type NullType in Spark Scala?

Sources

Related Questions