'spark submit Error: Failed to load class FirstDemo
I create sbt project with Intellij and build Artifacts to jar file. I put jar file to server and submit, but I got this error:
this is my code :-
import org.apache.spark.{SparkConf,SparkContext}
object FirstDemoo {
def main(args: Array[String]): Unit = {
val conf=new SparkConf().setMaster("local[*]").setAppName("FirstDemo")
val sc=new SparkContext(conf)
val rdd=sc.parallelize(Array(5,10,30))
println(rdd.reduce(_+_))
}
}
here is my build.sbt file:-
name := "DemooSpark"
version := "0.1"
scalaVersion := "2.12.6"
// https://mvnrepository.com/artifact/org.apache.spark/spark-core
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.2.0"
// https://mvnrepository.com/artifact/org.apache.spark/spark-sql
libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.0"
when running spark-submit it is showing error saying:
spark-submit --class FirstDemoo /home/hduser/IdeaProjects/DemooSpark/out/artifacts/DemooSpark_jar2/DemooSpark.jar
22/03/11 14:22:24 WARN Utils: Your hostname, kuldeep-VirtualBox resolves to a loopback address: 127.0.0.1; using 10.0.2.15 instead (on interface enp0s3)
22/03/11 14:22:24 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Error: Failed to load class FirstDemoo.
log4j:WARN No appenders could be found for logger (org.apache.spark.util.ShutdownHookManager).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
