'drop rows based on array size (data feteched from mongo) in a dataframe pysaprk

I have data fetched from mongo and one columns has array of values with different dataTypes, The data is as shown

+---+------+-----------+
|_id|acitve| Params    |
+---+------+-----------+
|  2|  true|    {['a']}|
|  3|  true|    {[]}   |
|  4|  true|    {[]}   |
|  5|  true|    {['b']}|
|  6|  true|    {[]}   |
|  7|  true|    {[]}   |

I have applied filter

df = mobile_app.filter(size(mobile_app.Params) >= 0).show()

error as shown

cannot resolve 'size(`Params`)' due to data type mismatch: argument 1 requires (array or map) type, however, '`ruleParams`' is of struct<condition:array<struct<_class:string,fields:array<struct<cField:boolean,parator:string,fieldName:string,macroValue:string,mode:string,op:string,service:string,threshold:int,value:string>>,op:string>>> type.;

I would like to filter out all the rows where "Param" array is empty "{[]}". Please let me know

I have also tried

df = mobile_app.filter(mobile_app.Params != {[]}).show()

It failed saying unhashable type: 'list'



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source