'How to resolve failed Apache Spark 3.2.1 Maven Test: SparkSubmitCommandBuilderSuite.testPySparkLauncher:154
I am trying to get Apache Spark 3.2.1 up and running, per https://spark.apache.org/docs/latest/building-spark.html
However, I can't pass the test 'Spark Project Launcher' from Maven's ./build/mvn test suite. I am invoking the tests from $SPARK_HOME (/opt/spark). It is failing the 'PySpark' launcher.
Please find details below:
It is for a (highly powered) single server running in standalone mode. OS is Ubuntu 20.04, running on a Docker container. I am SSH into the container when running it.
I have installed the pre-requisites: -Maven 3.6.3 -Java 8 installed via apt-get install openjdk-8-jdk -y -originally did not install Scala 2.12/2.13, as documentation states it's including in the Maven build, however I then did do so (2.12.15) and same issue below.
- I also pre-installed Hadoop 3.3.1, so that I can use HDFS commands in terminal.
- Python 3.7.12
- The build command I run is : ./build/mvn -Dhadoop.version=${HADOOP_VERSION} -Phive -Phive-thriftserver -Psparkr -DskipTests clean package -e
as I would like to have PySpark and Spark SQL running ('building spark' docs recommand including -phive to ensure this)
- My env looks like:
Note: these Yarn variables are populated but it should be running standalone.
JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
HDFS_DATANODE_USER=root
LANG=C.UTF-8
HADOOP_MAPRED_HOME=/opt/hadoop
YARN_HOME=/opt/hadoop
YARN_NODEMANAGER_USER=root
YARN_RESOURCEMANAGER_USER=root
HADOOP_COMMON_HOME=/opt/hadoop
HDFS_SECONDARYNAMENODE_USER=root
PYSPARK_PYTHON=/usr/bin/python3
HADOOP_HDFS_HOME=/opt/hadoop
HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
HADOOP_HOME=/opt/hadoop
PYTHONUTF8=1
SPARK_CONF_DIR=/opt/spark/conf
SPARK_HOME=/opt/spark
JAVA_VERSION=1.8.0_312
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin:/opt/spark/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/hadoop/bin
MAVEN_OPTS=-Xss64m -Xmx2g -XX:ReservedCodeCacheSize=1g
- SPARK CONFIG: spark-env.sh override contains:
export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export SPARK_EXECUTOR_CORES=1
export SPARK_EXECUTOR_MEMORY=1G
export SPARK_DRIVER_MEMORY=1G
export SPARK_WORKER_MEMORY=2G
export SPARK_DAEMON_MEMORY=1G
Note: I will edit these memory amounts later, once the tests are working first.
- HADOOP CONFIG:
core-site.xml properties include:
<configuration>
<property>
<name>fs.default.name</name>
<value>( This is filled in with the server details) </value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>
hdfs-site.xml properties include:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.safemode.extension</name>
<value>0</value>
</property>
<property>
<name>dfs.safemode.min.datanodes</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.safemode.min.datanodes</name>
<value>1</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>( IS SET ) </value>
</property>
<property>
<name>dfs.data.dir</name>
<value>( IS SET ) </value>
</property>
<property>
<name>dfs.namenode.rpc-bind-host</name>
<value>( IS SET ) </value>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>( IS SET ) </value>
</property>
<property>
<name>dfs.namenode.servicerpc-address</name>
<value>( IS SET ) </value>
</property>
<property>
<name>dfs.namenode.servicerpc-bind-host</name>
<value> ( IS SET ) </value>
</property>
</configuration>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>(IS SET)</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>local</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>(IS SET)</value>
</property>
</configuration>
- Dependencies include:
apt-get install ssh pdsh libdigest-sha-perl openjdk-8-jdk software-properties-common dirmngr libkrb5-dev tini libc6 libpam-modules krb5-user libnss3 procps -y
apt-get install python3-pip -y && \
pip3 install -U pip && \
/usr/bin/python3 -m pip install --upgrade pip && \
pip install --upgrade pandas pyarrow wheel setuptools cloudpickle requests flake8 flaky pytest krb5 requests-kerberos
I am running as root user. If this is an issue, can you please advise why/ what the implications are for running tests initially, as the documentation does not elaborate.
Spark-submit works if I run a test myself (not the test suite). E.g. the included simple sums of numbers Spark includes.
The error message stack trace for ./build/mvn -X test . Failure point is SparkSubmitCommandBuilderSuite.testPySparkLauncher:154
[INFO] T E S T S
[INFO] -------------------------------------------------------
[DEBUG] Determined Maven Process ID 1874165
[DEBUG] Fork Channel [1] connection string 'pipe://1' for the implementation class org.apache.maven.plugin.surefire.extensions.LegacyForkChannel
[DEBUG] boot classpath: /root/.m2/repository/org/apache/maven/surefire/surefire-booter/3.0.0-M5/surefire-booter-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/surefire-api/3.0.0-M5/surefire-api-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/surefire-logger-api/3.0.0-M5/surefire-logger-api-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/surefire-shared-utils/3.0.0-M4/surefire-shared-utils-3.0.0-M4.jar /root/.m2/repository/org/apache/maven/surefire/surefire-extensions-spi/3.0.0-M5/surefire-extensions-spi-3.0.0-M5.jar /opt/spark/launcher/target/scala-2.12/test-classes /opt/spark/launcher/target/scala-2.12/classes /root/.m2/repository/org/apache/logging/log4j/log4j-api/2.17.1/log4j-api-2.17.1.jar /root/.m2/repository/org/apache/logging/log4j/log4j-core/2.17.1/log4j-core-2.17.1.jar /root/.m2/repository/org/apache/logging/log4j/log4j-1.2-api/2.17.1/log4j-1.2-api-2.17.1.jar /root/.m2/repository/org/mockito/mockito-core/4.2.0/mockito-core-4.2.0.jar /root/.m2/repository/net/bytebuddy/byte-buddy/1.12.4/byte-buddy-1.12.4.jar /root/.m2/repository/net/bytebuddy/byte-buddy-agent/1.12.4/byte-buddy-agent-1.12.4.jar /root/.m2/repository/org/objenesis/objenesis/3.2/objenesis-3.2.jar /root/.m2/repository/org/slf4j/jul-to-slf4j/1.7.32/jul-to-slf4j-1.7.32.jar /root/.m2/repository/org/slf4j/slf4j-api/1.7.32/slf4j-api-1.7.32.jar /root/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar /root/.m2/repository/org/apache/spark/spark-tags_2.12/3.3.0-SNAPSHOT/spark-tags_2.12-3.3.0-SNAPSHOT.jar /root/.m2/repository/org/scala-lang/scala-library/2.12.15/scala-library-2.12.15.jar /root/.m2/repository/org/apache/spark/spark-tags_2.12/3.3.0-SNAPSHOT/spark-tags_2.12-3.3.0-SNAPSHOT-tests.jar /root/.m2/repository/org/apache/hadoop/hadoop-client-api/3.3.1/hadoop-client-api-3.3.1.jar /root/.m2/repository/org/apache/hadoop/hadoop-client-runtime/3.3.1/hadoop-client-runtime-3.3.1.jar /root/.m2/repository/org/apache/htrace/htrace-core4/4.1.0-incubating/htrace-core4-4.1.0-incubating.jar /root/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar /root/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.jar /root/.m2/repository/org/spark-project/spark/unused/1.0.0/unused-1.0.0.jar /root/.m2/repository/org/scalatest/scalatest_2.12/3.3.0-SNAP3/scalatest_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-core_2.12/3.3.0-SNAP3/scalatest-core_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-compatible/3.3.0-SNAP3/scalatest-compatible-3.3.0-SNAP3.jar /root/.m2/repository/org/scalactic/scalactic_2.12/3.3.0-SNAP3/scalactic_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scala-lang/modules/scala-xml_2.12/1.2.0/scala-xml_2.12-1.2.0.jar /root/.m2/repository/org/scalatest/scalatest-featurespec_2.12/3.3.0-SNAP3/scalatest-featurespec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-flatspec_2.12/3.3.0-SNAP3/scalatest-flatspec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-freespec_2.12/3.3.0-SNAP3/scalatest-freespec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-funsuite_2.12/3.3.0-SNAP3/scalatest-funsuite_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-funspec_2.12/3.3.0-SNAP3/scalatest-funspec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-propspec_2.12/3.3.0-SNAP3/scalatest-propspec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-refspec_2.12/3.3.0-SNAP3/scalatest-refspec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-wordspec_2.12/3.3.0-SNAP3/scalatest-wordspec_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-diagrams_2.12/3.3.0-SNAP3/scalatest-diagrams_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-matchers-core_2.12/3.3.0-SNAP3/scalatest-matchers-core_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-shouldmatchers_2.12/3.3.0-SNAP3/scalatest-shouldmatchers_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scalatest/scalatest-mustmatchers_2.12/3.3.0-SNAP3/scalatest-mustmatchers_2.12-3.3.0-SNAP3.jar /root/.m2/repository/org/scala-lang/scala-reflect/2.12.15/scala-reflect-2.12.15.jar /root/.m2/repository/org/scalatestplus/scalacheck-1-15_2.12/3.3.0.0-SNAP3/scalacheck-1-15_2.12-3.3.0.0-SNAP3.jar /root/.m2/repository/org/scalacheck/scalacheck_2.12/1.15.4/scalacheck_2.12-1.15.4.jar /root/.m2/repository/org/scalatestplus/mockito-4-2_2.12/3.2.11.0/mockito-4-2_2.12-3.2.11.0.jar /root/.m2/repository/org/scalatestplus/selenium-3-141_2.12/3.3.0.0-SNAP3/selenium-3-141_2.12-3.3.0.0-SNAP3.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-java/3.141.59/selenium-java-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-api/3.141.59/selenium-api-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-chrome-driver/3.141.59/selenium-chrome-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-edge-driver/3.141.59/selenium-edge-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-firefox-driver/3.141.59/selenium-firefox-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-ie-driver/3.141.59/selenium-ie-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-opera-driver/3.141.59/selenium-opera-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-remote-driver/3.141.59/selenium-remote-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-safari-driver/3.141.59/selenium-safari-driver-3.141.59.jar /root/.m2/repository/org/seleniumhq/selenium/selenium-support/3.141.59/selenium-support-3.141.59.jar /root/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.jar /root/.m2/repository/com/squareup/okhttp3/okhttp/3.11.0/okhttp-3.11.0.jar /root/.m2/repository/com/squareup/okio/okio/1.14.0/okio-1.14.0.jar /root/.m2/repository/org/seleniumhq/selenium/htmlunit-driver/2.50.0/htmlunit-driver-2.50.0.jar /root/.m2/repository/net/sourceforge/htmlunit/htmlunit/2.50.0/htmlunit-2.50.0.jar /root/.m2/repository/xalan/xalan/2.7.2/xalan-2.7.2.jar /root/.m2/repository/xalan/serializer/2.7.2/serializer-2.7.2.jar /root/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar /root/.m2/repository/org/apache/commons/commons-text/1.9/commons-text-1.9.jar /root/.m2/repository/org/apache/httpcomponents/httpmime/4.5.13/httpmime-4.5.13.jar /root/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar /root/.m2/repository/org/apache/httpcomponents/httpcore/4.4.14/httpcore-4.4.14.jar /root/.m2/repository/commons-codec/commons-codec/1.15/commons-codec-1.15.jar /root/.m2/repository/net/sourceforge/htmlunit/htmlunit-core-js/2.50.0/htmlunit-core-js-2.50.0.jar /root/.m2/repository/net/sourceforge/htmlunit/neko-htmlunit/2.50.0/neko-htmlunit-2.50.0.jar /root/.m2/repository/xerces/xercesImpl/2.12.0/xercesImpl-2.12.0.jar /root/.m2/repository/xml-apis/xml-apis/1.4.01/xml-apis-1.4.01.jar /root/.m2/repository/net/sourceforge/htmlunit/htmlunit-cssparser/1.7.0/htmlunit-cssparser-1.7.0.jar /root/.m2/repository/commons-io/commons-io/2.11.0/commons-io-2.11.0.jar /root/.m2/repository/commons-net/commons-net/3.8.0/commons-net-3.8.0.jar /root/.m2/repository/org/brotli/dec/0.1.2/dec-0.1.2.jar /root/.m2/repository/com/shapesecurity/salvation2/3.0.0/salvation2-3.0.0.jar /root/.m2/repository/org/eclipse/jetty/websocket/websocket-client/9.4.40.v20210413/websocket-client-9.4.40.v20210413.jar /root/.m2/repository/org/eclipse/jetty/jetty-client/9.4.44.v20210927/jetty-client-9.4.44.v20210927.jar /root/.m2/repository/org/eclipse/jetty/jetty-http/9.4.44.v20210927/jetty-http-9.4.44.v20210927.jar /root/.m2/repository/org/eclipse/jetty/jetty-util/9.4.44.v20210927/jetty-util-9.4.44.v20210927.jar /root/.m2/repository/org/eclipse/jetty/jetty-io/9.4.40.v20210413/jetty-io-9.4.40.v20210413.jar /root/.m2/repository/org/eclipse/jetty/websocket/websocket-common/9.4.40.v20210413/websocket-common-9.4.40.v20210413.jar /root/.m2/repository/org/eclipse/jetty/websocket/websocket-api/9.4.40.v20210413/websocket-api-9.4.40.v20210413.jar /root/.m2/repository/junit/junit/4.13.2/junit-4.13.2.jar /root/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar /root/.m2/repository/com/github/sbt/junit-interface/0.13.3/junit-interface-0.13.3.jar /root/.m2/repository/org/scala-sbt/test-interface/1.0/test-interface-1.0.jar /root/.m2/repository/org/apache/maven/surefire/surefire-junit4/3.0.0-M5/surefire-junit4-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/common-java5/3.0.0-M5/common-java5-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/common-junit3/3.0.0-M5/common-junit3-3.0.0-M5.jar /root/.m2/repository/org/apache/maven/surefire/common-junit4/3.0.0-M5/common-junit4-3.0.0-M5.jar
[DEBUG] boot(compact) classpath: surefire-booter-3.0.0-M5.jar surefire-api-3.0.0-M5.jar surefire-logger-api-3.0.0-M5.jar surefire-shared-utils-3.0.0-M4.jar surefire-extensions-spi-3.0.0-M5.jar test-classes classes log4j-api-2.17.1.jar log4j-core-2.17.1.jar log4j-1.2-api-2.17.1.jar mockito-core-4.2.0.jar byte-buddy-1.12.4.jar byte-buddy-agent-1.12.4.jar objenesis-3.2.jar jul-to-slf4j-1.7.32.jar slf4j-api-1.7.32.jar log4j-slf4j-impl-2.17.1.jar spark-tags_2.12-3.3.0-SNAPSHOT.jar scala-library-2.12.15.jar spark-tags_2.12-3.3.0-SNAPSHOT-tests.jar hadoop-client-api-3.3.1.jar hadoop-client-runtime-3.3.1.jar htrace-core4-4.1.0-incubating.jar commons-logging-1.1.3.jar jsr305-3.0.0.jar unused-1.0.0.jar scalatest_2.12-3.3.0-SNAP3.jar scalatest-core_2.12-3.3.0-SNAP3.jar scalatest-compatible-3.3.0-SNAP3.jar scalactic_2.12-3.3.0-SNAP3.jar scala-xml_2.12-1.2.0.jar scalatest-featurespec_2.12-3.3.0-SNAP3.jar scalatest-flatspec_2.12-3.3.0-SNAP3.jar scalatest-freespec_2.12-3.3.0-SNAP3.jar scalatest-funsuite_2.12-3.3.0-SNAP3.jar scalatest-funspec_2.12-3.3.0-SNAP3.jar scalatest-propspec_2.12-3.3.0-SNAP3.jar scalatest-refspec_2.12-3.3.0-SNAP3.jar scalatest-wordspec_2.12-3.3.0-SNAP3.jar scalatest-diagrams_2.12-3.3.0-SNAP3.jar scalatest-matchers-core_2.12-3.3.0-SNAP3.jar scalatest-shouldmatchers_2.12-3.3.0-SNAP3.jar scalatest-mustmatchers_2.12-3.3.0-SNAP3.jar scala-reflect-2.12.15.jar scalacheck-1-15_2.12-3.3.0.0-SNAP3.jar scalacheck_2.12-1.15.4.jar mockito-4-2_2.12-3.2.11.0.jar selenium-3-141_2.12-3.3.0.0-SNAP3.jar selenium-java-3.141.59.jar selenium-api-3.141.59.jar selenium-chrome-driver-3.141.59.jar selenium-edge-driver-3.141.59.jar selenium-firefox-driver-3.141.59.jar selenium-ie-driver-3.141.59.jar selenium-opera-driver-3.141.59.jar selenium-remote-driver-3.141.59.jar selenium-safari-driver-3.141.59.jar selenium-support-3.141.59.jar commons-exec-1.3.jar okhttp-3.11.0.jar okio-1.14.0.jar htmlunit-driver-2.50.0.jar htmlunit-2.50.0.jar xalan-2.7.2.jar serializer-2.7.2.jar commons-lang3-3.12.0.jar commons-text-1.9.jar httpmime-4.5.13.jar httpclient-4.5.13.jar httpcore-4.4.14.jar commons-codec-1.15.jar htmlunit-core-js-2.50.0.jar neko-htmlunit-2.50.0.jar xercesImpl-2.12.0.jar xml-apis-1.4.01.jar htmlunit-cssparser-1.7.0.jar commons-io-2.11.0.jar commons-net-3.8.0.jar dec-0.1.2.jar salvation2-3.0.0.jar websocket-client-9.4.40.v20210413.jar jetty-client-9.4.44.v20210927.jar jetty-http-9.4.44.v20210927.jar jetty-util-9.4.44.v20210927.jar jetty-io-9.4.40.v20210413.jar websocket-common-9.4.40.v20210413.jar websocket-api-9.4.40.v20210413.jar junit-4.13.2.jar hamcrest-core-1.3.jar junit-interface-0.13.3.jar test-interface-1.0.jar surefire-junit4-3.0.0-M5.jar common-java5-3.0.0-M5.jar common-junit3-3.0.0-M5.jar common-junit4-3.0.0-M5.jar
[DEBUG] Forking command line: /bin/sh -c cd /opt/spark/launcher && /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -ea -Xmx4g -Xss4m -XX:MaxMetaspaceSize=2g -XX:ReservedCodeCacheSize=128m -XX:+IgnoreUnrecognizedVMOptions --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED -Dio.netty.tryReflectionSetAccessible=true -jar /opt/spark/launcher/target/surefire/surefirebooter724989174307651539.jar /opt/spark/launcher/target/surefire 2022-03-20T00-46-11_265-jvmRun1 surefire4692929599609625015tmp surefire_0469901132567964224tmp
[DEBUG] Fork Channel [1] connected to the client.
[INFO] Running org.apache.spark.launcher.InProcessLauncherSuite
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.604 s - in org.apache.spark.launcher.InProcessLauncherSuite
[INFO] Running org.apache.spark.launcher.SparkSubmitCommandBuilderSuite
[ERROR] Tests run: 17, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.045 s <<< FAILURE! - in org.apache.spark.launcher.SparkSubmitCommandBuilderSuite
[ERROR] org.apache.spark.launcher.SparkSubmitCommandBuilderSuite.testPySparkLauncher Time elapsed: 0 s <<< FAILURE!
java.lang.AssertionError
at org.apache.spark.launcher.SparkSubmitCommandBuilderSuite.testPySparkLauncher(SparkSubmitCommandBuilderSuite.java:154)
[INFO] Running org.apache.spark.launcher.LauncherServerSuite
[INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 s - in org.apache.spark.launcher.LauncherServerSuite
[INFO] Running org.apache.spark.launcher.SparkSubmitOptionParserSuite
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.39 s - in org.apache.spark.launcher.SparkSubmitOptionParserSuite
[INFO] Running org.apache.spark.launcher.CommandBuilderUtilsSuite
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.apache.spark.launcher.CommandBuilderUtilsSuite
[INFO] Running org.apache.spark.launcher.ChildProcAppHandleSuite
[INFO] Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.179 s - in org.apache.spark.launcher.ChildProcAppHandleSuite
[DEBUG] Closing the fork 1 after saying GoodBye.
[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] SparkSubmitCommandBuilderSuite.testPySparkLauncher:154
[INFO]
[ERROR] Tests run: 46, Failures: 1, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Launcher 3.3.0-SNAPSHOT:
[INFO]
[INFO] Spark Project Launcher ............................. FAILURE [ 7.156 s]
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project ML Local Library ..................... SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 8.110 s
[INFO] Finished at: 2022-03-20T00:46:13Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project spark-launcher_2.12: There are test failures.
[ERROR]
[ERROR] Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test) on project spark-launcher_2.12: There are test failures.
Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.
Please refer to /opt/spark/launcher/target/surefire-reports for the individual test results.
Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
at org.apache.maven.plugin.surefire.SurefireHelper.throwException (SurefireHelper.java:271)
at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution (SurefireHelper.java:159)
at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary (SurefirePlugin.java:453)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked (AbstractSurefireMojo.java:1175)
at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute (AbstractSurefireMojo.java:932)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:972)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:293)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:196)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Can you please kindly confirm what the issue may be, as I suspect it's something really simple but I can't work this out, and have tried everything I can think of, the official build documentation end to end, and all Stack Overflow resources. The 'sbt' tests seem to run a bit better, but still include a few failures towards the end (it am pretty sure it is able to launch spark though).
I have also reviewed the 'sure fire reports' listed and can't see any issues.
I am not hugely familiar with 'JARS' and Classpaths etc. if there is are any implied setup steps of this nature I may have missed.
Thanks very much for your help, it is much appreciated.
Regards, PB
EDIT: I have since run './build/sbt test', with the following results. Therefore wondering if the above failing is an issue ?
[info] Run completed in 2 hours, 24 minutes, 2 seconds.
[info] Total number of tests run: 3812
[info] Suites: completed 175, aborted 0
[info] Tests: succeeded 3812, failed 0, canceled 2, ignored 19, pending 0
[info] All tests passed.
[error] (core / Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] (sql / Test / test) sbt.TestsFailedException: Tests unsuccessful
Noting that at the bottom there are 2 errors listed under 'core' and 'sql' suites (or with the summary are they ok) Thanks?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|