'exception:java.lang.IllegalArgumentException:Wrong FS: abfs://[email protected]/folder, expected: hdfs://master-node:port
I am trying to execute a scala file using Spark Submit in CDP CDH 7.2.9 hosted on Azure Platform. But I am getting below error
User class threw exception:java.lang.IllegalArgumentException:Wrong FS: abfs://[email protected]/folder, expected: hdfs://master-node:port at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:773)
I have tried below options as suggested in stackoverflow but no luck !
a.conf.spark.hadoop.fs.defaultFS = “abfs://[email protected]”
b.val hdfs = FileSystem.get(new java.net.URI(s"abfs://[email protected]"), spark.sparkContext.hadoopConfiguration) in scala program.
c.val adlsURI = s"abfs://[email protected]" FileSystem.setDefaultUri = (spark.sparkContext.hadoopConfiguration, new java.net.URI(adlsURI)) val hdfs: FileSystem = FileSystem.get(spark.sparkContext.hadoopConfiguration)
in Scala program.
d.conf.spark.yarn.access.hadoopFileSystems=abfs://[email protected]
I am using Spark 2.4 version.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|