'drop rows based on array size (data feteched from mongo) in a dataframe pysaprk
I have data fetched from mongo and one columns has array of values with different dataTypes, The data is as shown
+---+------+-----------+
|_id|acitve| Params |
+---+------+-----------+
| 2| true| {['a']}|
| 3| true| {[]} |
| 4| true| {[]} |
| 5| true| {['b']}|
| 6| true| {[]} |
| 7| true| {[]} |
I have applied filter
df = mobile_app.filter(size(mobile_app.Params) >= 0).show()
error as shown
cannot resolve 'size(`Params`)' due to data type mismatch: argument 1 requires (array or map) type, however, '`ruleParams`' is of struct<condition:array<struct<_class:string,fields:array<struct<cField:boolean,parator:string,fieldName:string,macroValue:string,mode:string,op:string,service:string,threshold:int,value:string>>,op:string>>> type.;
I would like to filter out all the rows where "Param" array is empty "{[]}". Please let me know
I have also tried
df = mobile_app.filter(mobile_app.Params != {[]}).show()
It failed saying unhashable type: 'list'
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
