'Maven Spark Source Code Build Fails in Ubuntu 20.04
I am trying to build Spark 3.1.1 source code using mvn build as below:
./build/mvn -DskipTests clean package
However build fails without giving any proper error as below
killed 13456 "${MVN_BIN}" -DzincPort=${ZINC_PORT} "$@"
Any help is much appreciated.
My environment is as below: OS: Ubuntu 20.04 Java : Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=1G; support was removed in 8.0 java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode) spark Version : Spark 3.1.1
Solution 1:[1]
I encountered the same problem. After investigation, it was confirmed that the Linux oom-killer killed the process and the reason was insufficient memory.
Maven Log:
[INFO] --- scala-maven-plugin:4.5.6:compile (scala-compile-first) @ spark-hive-thriftserver_2.12 ---
[INFO] Using incremental compilation using Mixed compile order
[INFO] Compiler bridge file: /home/tianshuang/.sbt/1.0/zinc/org.scala-sbt/org.scala-sbt-compiler-bridge_2.12-1.5.8-bin_2.12.15__52.0-1.5.8_20211211T222914.jar
[INFO] compiler plugin: BasicArtifact(com.github.ghik,silencer-plugin_2.12.15,1.7.6,null)
[INFO] compiling 27 Scala sources and 86 Java sources to /home/tianshuang/IdeaProjects/latest/spark/sql/hive-thriftserver/target/scala-2.12/classes ...
./build/mvn: line 185: 8772 Killed "${MVN_BIN}" "$@"
Cmd:
dmesg -T | grep -i kill
Output:
[Wed Apr 13 22:07:44 2022] fsnotifier invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[Wed Apr 13 22:07:44 2022] oom_kill_process.cold+0xb/0x10
[Wed Apr 13 22:07:44 2022] [ 1862] 1000 1862 114275 95 110592 117 0 gsd-rfkill
[Wed Apr 13 22:07:44 2022] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[email protected],task=java,pid=8772,uid=1000
[Wed Apr 13 22:07:44 2022] Out of memory: Killed process 8772 (java) total-vm:15551840kB, anon-rss:4862120kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:10256kB oom_score_adj:0
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Poison |