'Column attributes of type STRUCT cannot be used in SELECT DISTINCT

I have this sql statement

Select distinct _data.attributes from rd-bigdata-lake-prd.lake_gcs.xyz

and I get this error

400 Column attributes of type STRUCT cannot be used in SELECT DISTINCT

The value of _data.attributes is

{"url":"/services/data/v52.0/sobjects/Message/123456789","type":"Message"}

How to change the sql statement, that the field _data.attributes is readable with select distinct? I tried to cast _data.attributes to string. But it doesn't work.

apache-sparkpysparkstructapache-spark-sql


Solution 1:[1]

Convert the struct to string using to_json

select distinct 
       to_json(_data.attributes)

from   rd-bigdata-lake-prd.lake_gcs.xyz

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 David דודו Markovitz

Related Questions