'Scala from_json function is throwing error when i use options

Below is my code:

val opts = Map("allowUnquotedFieldNames" -> "true")
    
val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))

Below is the error:

(<console>:198: error: overloaded method value from_json with alternatives:
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.Column,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: String,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: String,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column
 cannot be applied to (org.apache.spark.sql.Column, org.apache.spark.sql.types.StructType, scala.collection.Map[String,String])
           val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))
                                                     ^
,0,1388)

Can someone please help me understand why this code is throwing error and how to fix this?

Background:

Data element is a Nested JSON stored as string. So i have defined a schema for Data element and converting it as JSON using from_JSON. Also, the fields names inside Data are under quotes so i am trying to use options here(Map("allowUnquotedFieldNames" -> "true"))



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source