'how to create an encoder for a row type of map[(String,String),List[Row]] for creating a Dataset[ map[(String,String),List[Row]] ] in spark?

This is my piece of code . There is a good lot of business logic happening here. I have tried to explain it in understandable manner as much as possible. I have not given definition for some methods which I used.


import org.apache.spark.sql.Row
import org.apache.spark.sql.{Encoder, Encoders}

val changeField="col3"
spark.read.load("")
.groupByKey(x=> (x.getAs[String]("col1"),x.getAs[String]("col2")))(Encoders.product[(String,String)]).mapGroups((key,rowList) => 

// Some business computation 
val rowsList=rowList.toList
val validRow=rowsList.filter(x=> x.getAs[T]("filterCol") == "FilterVal") //Some business logic
val targetRow = validRow.filterNot(x=> x.getAs[T]("filterCol") == "FilterVal")

val targetSchema=targetRow.head.schema

val targetRows= targetRow.map( x => Row(targetSchema.foldLeft(Seq.empty[Any])(SeqVal,curVal) 
 => currVal.name ==changeField match {case true => SeqVal:+computeSomeVal(x.getAs[String](curVal))  case false => x.getAs[String](curVal) }) :_* )

Map(key,targetRows ++ validRow)
)
(WHAT ROW ENCODER MUST COME HERE ???)

kindly help me with this folks.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source