'Wrong FS s3://ss-pprd-v2-dart//tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -expected s3a://ss-pprd-v2-dart

I am using spark 3.2.1, Java8 ->1.8.0_292 (AdoptOpenJDK), Scala 2.12.10 and trying to read and write the data from/to redshift by using below mentioned jars and packages. But i am not able write the data back. While writing the data back to redshift. it was creating avro files with one manifest.json file in the temp directory but in my current versions it is not able to create the manifest.json file but it is creating all the avro files.

Jars and Packages:-

RedshiftJDBC42-no-awssdk-1.2.54.1082.jar,
hadoop-aws-3.3.1.jar,aws-java-sdk-1.12.173.jar ,
org.apache.spark:spark-avro_2.12:3.2.1,
io.github.spark-redshift-community:spark-redshift_2.12:5.0.3,
com.eclipsesource.minimal-json:minimal-json:0.9.5

Code i am trying to run:

from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession

conf=SparkConf().setAppName("Testing")
sc=SparkContext.getOrCreate(conf)
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", AWS_ACCESS_KEY)
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", AWS_SECRET_KEY)

df.write \
        .format("io.github.spark_redshift_community.spark.redshift")\
        .option("url", REDSHIFT_JDBC_URL) \
        .option("dbtable",MASTER_TABLE) \
        .option("forward_spark_s3_credentials", "true") \
        .option("extracopyoptions", EXTRACOPYOPTIONS) \
        .option("tempdir", "s3a://" + str(S3_BUCKET) + "/tempdir") \
        .mode("append") \
        .save()

print("Sucesss")

Stack Trace:

Traceback (most recent call last):
  File "/Users/brajeshmishra/Documents/TEMP/Temp_Py.py", line 65, in <module>
    .mode("append") \
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 738, in save
  File "/opt/homebrew/Cellar/apache-spark/3.2.1/libexec/python/lib/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1322, in __call__
  File "/opt/homebrew/Cellar/apache-

List item

spark/3.2.1/libexec/python/lib/pyspark.zip/pyspark/sql/utils.py", line 117, in deco
pyspark.sql.utils.IllegalArgumentException: Wrong FS s3://ss-pprd-v2-dart//tempdir/962c6007-77c0-4294-b021-b9498e3d66ab/manifest.json -expected s3a://ss-pprd-v2-dart
javascalahadooppyspark


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions