pyspark read file from S3 Compatible Storage(Dell ECS) not working

Question

I have a spark standalone configured with 3 nodes.

I want to read csv data stored in s3-compatible storage (dell ecs) in this pySpark.

Here's the method and configuration I've tried:

1. Put hadoop-aws-3.3.1.jar and aws-java-sdk-bundle-1.11.375.jar in $SPARK_HOME/jars path of all spark nodes.

2. $SPARK_HOME/conf/spark-defauly.conf.

.

spark.hadoop.fs.s3a.impl org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.access.key ACCESS_KEY

spark.hadoop.fs.s3a.secret.key SECRET_KEY

spark.hadoop.fs.s3a.endpoint http://ECS_IP:9020

spark.hadoop.fs.s3a.connection.ssl.enabled false

spark.hadoop.fs.s3a.path.style.access true

3. $SPARK_HOME/bin/pyspark.

   df = spark.read.options(header='True', delimiter=',', inferSchema=True).csv("s3a://my-bucket/file.csv")

Here is the result:

22/04/25 15:45:34 WARN FileSystem: Failed to initialize fileystem s3a://my-bucket/file.csv: java.lang.IllegalArgumentException: bucket
22/04/25 15:45:34 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: s3a://my-bucket/file.csv.
java.lang.IllegalArgumentException: bucket
at org.apache.hadoop.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
at org.apache.hadoop.fs.s3a.S3AUtils.propagateBucketOptions(S3AUtils.java:1152)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:374)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:370)
at org.apache.spark.sql.DataFrameReader.loadV1Source(DataFrameReader.scala:274)
at org.apache.spark.sql.DataFrameReader.$anonfun$load$3(DataFrameReader.scala:245)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:245)
at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:571)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:282)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
at java.lang.Thread.run(Thread.java:748)
22/04/25 15:45:34 WARN FileSystem: Failed to initialize fileystem s3a://my-bucket/file.csv: java.lang.IllegalArgumentException: bucket
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/spark/python/pyspark/sql/readwriter.py", line 410, in csv
    return self._df(self._jreader.csv(self._spark._sc._jvm.PythonUtils.toSeq(path)))
  File "/usr/local/spark/python/lib/py4j-0.10.9.2-src.zip/py4j/java_gateway.py", line 1309, in __call__
  File "/usr/local/spark/python/pyspark/sql/utils.py", line 117, in deco
    raise converted from None
pyspark.sql.utils.IllegalArgumentException: bucket