'How to update CSV file placed at Blob Storage in Azure
I have Blob Storage in Azure that holds a CSV file. While incrementally updating it creates a new CSV file instead we need to update the existing CSV file.
Is there any solution in the Azure data factory to update incremental data into the existing CSV file?
Solution 1:[1]
You can work on incremental updates to a Blob Storage file using Azure Databricks and in ADF you can call the databricks notebook activity
In this you will have to mount the Azure storage on databricks cluster first
Then you will have to read the original and incremental file or data as a dataframe
You can perform joins on the files and create an incremental data frame which you can overwrite to your original file which will be your incremental data and that too in a single file
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Pratik Somaiya |