'spark-avro dependency not found
How to resolve the spark-avro dependency issue.
I am using spark on kubernetes. In the entrypoint.sh file, spark-submit command is used to run the driver.
CMD=(
"$SPARK_HOME/bin/spark-submit"
--conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
--deploy-mode client
--packages org.apache.spark:spark-avro_2.12:3.2.0
--conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp"
"$@"
)
Sometimes the dependency gets resolved, but many times its throws an error of
:: org.apache.spark#spark-avro_2.12;3.2.0: not found
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
