Databricks Connect: Unable to run scala program in databricks cluster from IntelliJ

Question

Followed the steps mentioned in this docs. databricks-connect test command works fine. However, when I launch the test scala program from Intellij, I'm seeing following error:

Exception in thread "main" java.lang.NoSuchMethodError: com.databricks.spark.util.MetricDefinitions$.EVENT_INITIAL_CONFIG_LOG()Lcom/databricks/spark/util/MetricDefinition;
    at com.databricks.spark.util.InitialConfigLogging$.recordInitialConfigs(InitialConfigLogging.scala:47)
    at org.apache.spark.sql.internal.SQLConf.recordSessionInitialConfs(SQLConf.scala:4166)
    at org.apache.spark.sql.SparkSession.<init>(SparkSession.scala:150)
    at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:1033)
    at com.srikanth.Demo$.main(Demo.scala:13)
    at com.srikanth.Demo.main(Demo.scala)

Environment details:

python - 3.8
java - 1.8
databricks-connect - 9.1