How to register a JDBC Spark dialect in Python?

Question

I am trying to read from a databricks table. I have used the url from a cluster in the databricks. I am getting this error:

 java.sql.SQLDataException: [Simba][JDBC](10140) Error converting value to int.

After these statements:

jdbcConnUrl= "jdbc:spark://adb....."
testquery="(select * from db.table limit 3)"
testdf=spark.read.format("jdbc").option("url", jdbcConnUrl).option("dbtable", testquery).option("fetchsize", "10000").load()
testdf.show()

I have come across all Scala solutions for this issue but I am using python. I want a python equivalent of this code:

import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects}
JdbcDialects.registerDialect(new JdbcDialect() {
override def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:spark:")
override
def quoteIdentifier(column: String): String = column
})

Answer 1

You can use %scala to run the same scala code in databricks python notebook.
No need to create separate code for python.

Just add below to the notebook cell and run:

%scala
import org.apache.spark.sql.jdbc.{JdbcDialect, JdbcDialects}

JdbcDialects.registerDialect(new JdbcDialect() {
    override
    def canHandle(url: String): Boolean = url.toLowerCase.startsWith("jdbc:spark:")
    override
    def quoteIdentifier(column: String): String = column
})