'Write Spark dataframe into delta lake
I am trying to convert Spark data frame into delta format using the example code provided by documentation but always getting this strange error. Can you please help or guide?
df_sdf.write.format("delta").save("/mnt/.../delta/")
Error looks like:
org.apache.spark.SparkException: Job aborted.
--------------------------------------------------------------------------- Py4JJavaError Traceback (most recent call last) <command-3011941952225495> in <module> ----> 1 df_sdf.write.format("delta").save("/mnt/.../delta/") /databricks/spark/python/pyspark/sql/readwriter.py in save(self, path, format, mode, partitionBy, **options) 737 self._jwrite.save() 738 else: --> 739 self._jwrite.save(path) 740 741 @since(1.4)
/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py in call(self, *args) 1255 answer = self.gateway_client.send_command(command) 1256 return_value = get_return_value( -> 1257 answer, self.gateway_client, self.target_id, self.name) 1258 1259 for temp_arg in temp_args:
/databricks/spark/python/pyspark/sql/utils.py in deco(a, *kw)
Solution 1:[1]
/mnt/.../delta/
is not a valid path.
Solution 2:[2]
Try this:
df_sdf.write.format("delta").save("/mnt/.../delta/sdf")
Solution 3:[3]
I had the same error and the issue was that I was using Spark 3.0 Preview. I had to change the spark version to 2.4 and the issue was resolved.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Nathan |
| Solution 2 | Ebraim Carvalho |
| Solution 3 | Uncle John |