'VariableSubstitution class in spark 3.1.x issue
My below scala/spark code used to pass sqlconf variables through VariableSubstitution class constructor in spark 2.x. It was further used to replace the variables in a sql query whose content is stored in text files in my filesystem. Below is just relevant code snippet, not the exact:
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.VariableSubstitution
val sqlConf = new SQLConf()
sqlConf.setConfString("startDate", "2020-09-08")
sqlConf.setConfString("endDate", "2020-09-08")
sqlConf.setConfString("geoCode", "A")
val rawQuery = scala.io.Source.fromFile(ClassX.locateFile(queryFileName)).mkString
val variableSubstitution = new VariableSubstitution(sqlConf)
val finalQuery = variableSubstitution.substitute(rawQuery)
In spark 3, the code is giving error with message as
error: no arguments allowed for nullary constructor VariableSubstitution: ()org.apache.spark.sql.internal.VariableSubstitution [ERROR] val variableSubstitution = new VariableSubstitution(sqlConf)
Is there any alternative solution to my above problem . Is there any other way I can pass the sqlconf variables in the query content. ?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
