Scala version issues while using spark runtime configuration on Apache Hop

Question

I have spark cluster running over kubernetes (v3.1.2), which comes with scala jars of v2.12.10

While running any pipeline with Scala provided with HOP results into the following error, using HOP version 1.1.0 (Comes with scala Jars of v2.11.12)

===[Starting HopRun]=========================================================
2022/01/31 11:33:49 - HopRun - Start of Hop Run
2022/01/31 11:33:49 - HopRun - Enabling project 'samples'
2022/01/31 11:33:49 - HopRun - Starting pipeline: .\config\projects\samples\transforms\calculator-basic.hpl
2022/01/31 11:33:55 - General - Created Apache Beam pipeline with name 'calculator-basic'
2022/01/31 11:33:55 - General - Handled generic transform (TRANSFORM) : Input Data, gets data from 0 previous transform(s), targets=0, infos=0
2022/01/31 11:33:55 - General - Handled generic transform (TRANSFORM) : Calculator, gets data from 1 previous transform(s), targets=0, infos=0
2022/01/31 11:33:55 - General - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark'
2022/01/31 11:33:56 - General - ERROR: Error starting the Beam pipeline
2022/01/31 11:33:56 - General - ERROR: org.apache.hop.core.exception.HopException:
2022/01/31 11:33:56 - General - Error executing pipeline with runner Spark
2022/01/31 11:33:56 - General - scala.Product.$init$(Lscala/Product;)V
2022/01/31 11:33:56 - General -
2022/01/31 11:33:56 - General -         at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:244)
2022/01/31 11:33:56 - General -         at org.apache.hop.beam.engines.BeamPipelineEngine.startThreads(BeamPipelineEngine.java:267)
2022/01/31 11:33:56 - General -         at org.apache.hop.run.HopRun.runPipeline(HopRun.java:245)
2022/01/31 11:33:56 - General -         at org.apache.hop.run.HopRun.runPipeline(HopRun.java:201)
2022/01/31 11:33:56 - General -         at org.apache.hop.run.HopRun.run(HopRun.java:141)
2022/01/31 11:33:56 - General -         at org.apache.hop.run.HopRun.main(HopRun.java:737)
2022/01/31 11:33:56 - General - Caused by: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
2022/01/31 11:33:56 - General -         at org.apache.spark.SparkConf$DeprecatedConfig.<init>(SparkConf.scala:799)
2022/01/31 11:33:56 - General -         at org.apache.spark.SparkConf$.<init>(SparkConf.scala:596)
2022/01/31 11:33:56 - General -         at org.apache.spark.SparkConf$.<clinit>(SparkConf.scala)
2022/01/31 11:33:56 - General -         at org.apache.spark.SparkConf.contains(SparkConf.scala:455)
2022/01/31 11:33:56 - General -         at org.apache.beam.runners.spark.translation.SparkContextFactory.createSparkContext(SparkContextFactory.java:89)
2022/01/31 11:33:56 - General -         at org.apache.beam.runners.spark.translation.SparkContextFactory.getSparkContext(SparkContextFactory.java:67)
2022/01/31 11:33:56 - General -         at org.apache.beam.runners.spark.SparkRunner.run(SparkRunner.java:220)
2022/01/31 11:33:56 - General -         at org.apache.hop.beam.engines.BeamPipelineEngine.executePipeline(BeamPipelineEngine.java:238)
2022/01/31 11:33:56 - General -         ... 5 more

Answer 1

I hope you were able to solve this by now? If not it will probably be best to align those jars and try again.

As this is a minor version difference you can probably replace the jar in the Hop installation and re-create the fat jar