'spark-avro dependency not found

How to resolve the spark-avro dependency issue.

I am using spark on kubernetes. In the entrypoint.sh file, spark-submit command is used to run the driver.

CMD=(
      "$SPARK_HOME/bin/spark-submit"
      --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS"
      --deploy-mode client
      --packages org.apache.spark:spark-avro_2.12:3.2.0
      --conf spark.driver.extraJavaOptions="-Divy.cache.dir=/tmp -Divy.home=/tmp"
      "$@"
    )

Sometimes the dependency gets resolved, but many times its throws an error of

:: org.apache.spark#spark-avro_2.12;3.2.0: not found



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source