'Is there a way to compare two Get Metadata Activity Child Items in ADF?
Just wondering if someone could help me with a problem I have.
I am trying to set up a Pipeline to download files from an FTP Server into ADLS2, as the Pipeline will run everyday I need it to only download new files from the FTP Server.
As I have no control over the files that are input I will need to check whether the filename is already existent in ADLS2.
I have attempted to use a ForEach Loop and pass through each filename from the FTP Server (from the relative Get Metadata Activity) to check whether it is contained in the ADLS2 Folder (Get Metadata Activity). However, this process is incredibly slow when dealing with a large number of files (10k+).
Does anyone have any ideas of how I could do a bulk comparison between the two outputs?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|