Error while spark-submit , launching jupyter kernel

Question

Command Executed:

sudo -E -u mlp-user /var/lib/pf-spark/bin/spark-submit --master yarn --deploy-mode cluster --py-files hdfs:///user/mlp-user/notebooks/mlsdk/mlsdk.zip --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false --conf spark.hive.mapred.supports.subdirectories=true --conf spark.hadoop.mapreduce.input.fileinputformat.input.dir.recursive=true --conf spark.yarn.tags=${KERNEL_ID:-ERROR__NO__KERNEL_ID},${KERNEL_USER_SYSTEM:-UNSPECIFIED} --conf spark.yarn.archive=hdfs:///user/mlp-user/spark_jars_archive/spark_jars_2.4.tar.gz --conf spark.yarn.executor.nodeLabelExpression= --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./fintech.Linux_debian_9_3.py36.fintech_rsm_v2_2.tar.gz/bin/python ${KERNEL_EXTRA_SPARK_OPTS} --jars=hdfs:///user/mlp-user/hive_jars/json-serde-1.3-SNAPSHOT-jar-with-dependencies.jar,hdfs:///user/mlp-user/hive_jars/dimlookup-hive-udf-1.0-SNAPSHOT-jar-with-dependencies.jar,hdfs:///user/mlp-user/hive_jars/fdp-hive-exec.jar --archives=hdfs:///user/mlp-user/images/fintech.Linux_debian_9_3.py36.fintech_rsm_v2_2.tar.gz '' hdfs:///user/mlp-user/notebook_launcher/launch_ipykernel.py '' /root/.local/share/jupyter/runtime/kernel-0a9127b9-18ae-446f-ab50-48fd38b171d3.json --RemoteProcessProxy.response-address 10.33.98.78:31001 --RemoteProcessProxy.port-range 0..0 --RemoteProcessProxy.spark-context-initialization-mode lazy

Output:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: ./SecurityAuth.audit (Permission denied)
    at java.io.FileOutputStream.open0(Native Method)
    at java.io.FileOutputStream.open(FileOutputStream.java:270)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)
    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)
    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)
    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)
    at org.apache.log4j.DailyRollingFileAppender.activateOptions(DailyRollingFileAppender.java:223)
    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
    at org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
    at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
    at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
    at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
    at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
    at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
    at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
    at org.apache.spark.internal.Logging$class.initializeLogging(Logging.scala:120)
    at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:108)
    at org.apache.spark.deploy.SparkSubmit.initializeLogIfNecessary(SparkSubmit.scala:71)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:79)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
log4j:ERROR Either File or DatePattern options are not set for appender [exceptionAppender].
Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/home/harshil.sokhadia/
    at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:668)
    at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:225)
    at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:117)
    at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(SparkSubmit.scala:909)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.parseArguments(SparkSubmit.scala:909)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:81)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:922)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:931)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Answer 1

You seem to have an error in your $SPARK_HOME/conf/log4j.properties file.

log4j:ERROR Either File or DatePattern options are not set for appender [exceptionAppender]