'Pyspark SparkContext error (windows, python 3.8.3)
Please help me, I get an error just initiating SparkContext. Thanx
TLDR:
File "C:\ProgramFiles\anaconda\lib\site-packages\py4j-0.10.9.3-py3.8.egg\py4j\protocol.py", line 326, in get_return_value raise Py4JJavaError(
Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x69fb6037) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x69fb6037
Full Output: sc = pyspark.SparkContext(appName="intro_to_spark")
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 22/03/18 11:33:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Traceback (most recent call last):
File "", line 1, in sc = pyspark.SparkContext(appName="intro_to_spark")
File "C:\ProgramFiles\anaconda\lib\site-packages\pyspark\context.py", line 146, in init self._do_init(master, appName, sparkHome, pyFiles, environment, batchSize, serializer,
File "C:\ProgramFiles\anaconda\lib\site-packages\pyspark\context.py", line 209, in _do_init self._jsc = jsc or self._initialize_context(self._conf._jconf)
File "C:\ProgramFiles\anaconda\lib\site-packages\pyspark\context.py", line 329, in _initialize_context return self._jvm.JavaSparkContext(jconf)
File "C:\ProgramFiles\anaconda\lib\site-packages\py4j-0.10.9.3-py3.8.egg\py4j\java_gateway.py", line 1585, in call return_value = get_return_value(
File "C:\ProgramFiles\anaconda\lib\site-packages\py4j-0.10.9.3-py3.8.egg\py4j\protocol.py", line 326, in get_return_value raise Py4JJavaError(
Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.lang.IllegalAccessError: class org.apache.spark.storage.StorageUtils$ (in unnamed module @0x69fb6037) cannot access class sun.nio.ch.DirectBuffer (in module java.base) because module java.base does not export sun.nio.ch to unnamed module @0x69fb6037 at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala:213) at org.apache.spark.storage.StorageUtils$.(StorageUtils.scala) at org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:110) at org.apache.spark.SparkEnv$.$anonfun$create$9(SparkEnv.scala:348) at org.apache.spark.SparkEnv$.registerOrLookupEndpoint$1(SparkEnv.scala:287) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:336) at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:191) at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:277) at org.apache.spark.SparkContext.(SparkContext.scala:460) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:58) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:238) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.base/java.lang.Thread.run(Thread.java:833)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
