Py4JJavaError while loading Jar file in SageMaker jupyter notebook

Question

I am having issue when I try to load jar file in SageMaker Jupyter notebook.

import sagemaker_pyspark
from pyspark.sql import SparkSession

classpath = "/home/ec2-user/SageMaker/someJar.jar"
spark = SparkSession.builder.config("spark.driver.extraClassPath", classpath).getOrCreate()

This is image of the error I am getting

Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.NoClassDefFoundError: Could not initialize class scala.xml.Null$
    at org.apache.spark.ui.jobs.AllJobsPage.<init>(AllJobsPage.scala:43)
    at org.apache.spark.ui.jobs.JobsTab.<init>(JobsTab.scala:45)
    at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:61)
    at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:80)
    at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:175)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:444)
    at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:238)
    at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
    at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)

I was able to load different jar file without any issue.

My assumption is that it could be because of Scala version mismatch between jar and SageMaker spark. I don't know how to resolve this issue. If anyone have any insight into this issue. Please let me know.

Thank you