'How to write to a folder in a Azure Data Lake container using Delta?

How do I write to a folder in an Azure Data Lake container using Delta?

When I run:

    write_mode = 'overwrite'
    write_format = 'delta'
    save_path = '/mnt/container-name/folder-name'
     
    df.write \
        .mode(write_mode) \
        .format(write_format) \
        .save(save_path)

I get an incompatible format error. This does not happen when I write directly to the container ('/mnt/container-name').

I have also tried writing using the SQL commands but I get the same error.

What am I doing wrong?

Solution 1:^[1]

It turns out that you cannot save to a folder within a delta table. The mistake that I made was I saved the delta table directly to the container and then tried to save another delta table to a folder within this container.

When I saved to a folder within a container that only had folders in it, I did not get this problem.

Solution 2:^[2]

Check weather in this path is there any data Available or not /mnt/container-name/folder-name and also check storage connection between the data lake and data bricks .

Otherwise Create mount and Create a storage connection between the data lake and data bricks.Follow below steps.

spark.conf.set("fs.azure.account.key.blobstoragename.dfs.core.windows.net",dbutils.secrets.get(scope="scopename",key="keyvalue"))

Create mount

dbutils.fs.mount(
source = "wasbs://[email protected]",
mount_point = "/mnt/iotd/fgh",
extra_configs = {"fs.azure.account.key. blobstoragename.blob.core.windows.net":" past Access key "})

For example, to overwrite the data in a table you can:

df.write.format("delta").mode("overwrite").save("/mnt/container-name/folder-name")

Reference:

https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-get-started.html

Table batch reads and writes - Azure Databricks | Microsoft Docs

Table batch reads and writes — Delta Lake Documentation

https://www.youtube.com/watch?v=cbobqI3ZGuA

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	KamKam
Solution 2	BhanunagasaiVamsi-MT

'How to write to a folder in a Azure Data Lake container using Delta?

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]