PySpark: Can only call getServletHandlers on a running MetricsSystem

Question

I am trying to execute some simple pyspark scripts to test our installation of Apache Spark within our Kubernetes cluster.

import pyspark
from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.master("spark://<host>:<port>")
    .appName("SparkSession")
    .getOrCreate()
)

print(f"PySpark Version : {spark.version}")
print(f"PySpark Version : {spark.sparkContext.version}")

However, when running the script I get this error:

22/05/04 13:07:48 ERROR SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

I have been able to reach the cluster using telnet <host> <port>, the Spark WebUI is available on the local network and the workers are listed as available.

The spark-shell also works fine from the master pod. I've checked the version number on my machine and the remote, and they are the same 3.2.1.

Has anyone encountered the something similar? What causes the error?