'Scala Databricks: read a JSON file and return two DataFrame
I have the following code :
def reader (path:String):DataFrame={
val path =sc.textFile("/FileStore/tables/Data_exo2.json").reduce((a,b)=>s"$a$b")
val df = Seq((path)).toDF()
val schema_tran = new StructType()
.add("Devis", StringType, false)
.add("IdTransaction",LongType, false)
.add("Pays", StringType, false)
.add("Prix",DoubleType, false)
.add("TypeProduit", StringType, false)
val schema = new StructType().add("Transaction", ArrayType(schema_tran),true)
val df_66 = df.select(from_json($"value",schema)as "struct")
.select($"struct.*")
.withColumn("Transaction", explode(col("Transaction")))
.select($"Transaction.*")
val schema_devis = new StructType()
.add("Devis", StringType, false)
.add("Taux",DoubleType, false)
val schema_1 = new StructType()
.add("Devis", ArrayType(schema_devis),true)
val df_67 = df.select(from_json($"value",schema_1)as "struct")
.select($"struct.*")
.withColumn("Devis", explode(col("Devis")))
.select($"Devis.*")
*****}
So I have two dataframe df_66 and df_67. Basicaly, I want to define a fonction to return those dataframe, one by one like this reader(path)(0)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
