'VariableSubstitution class in spark 3.1.x issue

My below scala/spark code used to pass sqlconf variables through VariableSubstitution class constructor in spark 2.x. It was further used to replace the variables in a sql query whose content is stored in text files in my filesystem. Below is just relevant code snippet, not the exact:

import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.VariableSubstitution

val sqlConf = new SQLConf()
sqlConf.setConfString("startDate", "2020-09-08")
sqlConf.setConfString("endDate", "2020-09-08")
sqlConf.setConfString("geoCode", "A")

val rawQuery =  scala.io.Source.fromFile(ClassX.locateFile(queryFileName)).mkString  
val variableSubstitution = new VariableSubstitution(sqlConf)  
val finalQuery = variableSubstitution.substitute(rawQuery)

In spark 3, the code is giving error with message as

error: no arguments allowed for nullary constructor VariableSubstitution: ()org.apache.spark.sql.internal.VariableSubstitution [ERROR] val variableSubstitution = new VariableSubstitution(sqlConf)

Is there any alternative solution to my above problem . Is there any other way I can pass the sqlconf variables in the query content. ?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source