'Scala from_json function is throwing error when i use options
Below is my code:
val opts = Map("allowUnquotedFieldNames" -> "true")
val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))
Below is the error:
(<console>:198: error: overloaded method value from_json with alternatives:
(e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.Column,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: String,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: String,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
(e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column
cannot be applied to (org.apache.spark.sql.Column, org.apache.spark.sql.types.StructType, scala.collection.Map[String,String])
val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))
^
,0,1388)
Can someone please help me understand why this code is throwing error and how to fix this?
Background:
Data element is a Nested JSON stored as string. So i have defined a schema for Data element and converting it as JSON using from_JSON. Also, the fields names inside Data are under quotes so i am trying to use options here(Map("allowUnquotedFieldNames" -> "true"))
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
