'ADLS move files to a parent directory

Lets say I have around 5TB of files in adls folder(ADLS Gen2). Folder structure is like this

files/country/

Now all the files belong to US region and I want to add one more folder in the hierarchy, kind of like this.

files/country/USA/

How to achieve this in the fastest and cost most cost efficient manner?

azureazure-data-lake-gen2


Solution 1:[1]

You can use Data Factory.

Select DataFlow Activity. In Source Settings Select Key as Set partitioning.

Now select Country column in Unique value per partition.

enter image description here

Refer - https://docs.microsoft.com/en-us/azure/data-factory/tutorial-data-flow-write-to-lake#hierarchical-folder-output

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 AbhishekKhandave-MT

Related Questions