'Azure Data Factory - How to filter out specific files in multiple Zip. files?

I have set up a ADF pipeline that gets a set of .Zip files from Azure Storage, and iterates through each Zip file's folders and files to land them in an output container with preserved hierarchy.

Get Metadata: enter image description here

For Each: enter image description here

Issue: The issue is that there is a specific .PDF file (ASC_NTS.pdf) that is embedded within each .Zip file that has the same name:

enter image description here

enter image description here

It is causing this error when trying to run the pipeline:

Error Operation on target ForEach1 failed: Activity failed because an inner activity failed; Inner activity name: Copy data1, Error: ErrorCode=AdlsGen2OperationFailedConcurrentWrite,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Error occurred when trying to upload a file. It's possible because you have multiple concurrent copy activities runs writing to the same file 'FAERS_output/ascii/ASC_NTS.pdf'. Check your ADF configuration.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'PreconditionFailed'. Account: 'asastgssuaefdbhdg2dbc4'. FileSystem: 'curated'. Path: 'FAERS_output/ascii/ASC_NTS.pdf'. ErrorCode: 'LeaseIdMissing'. Message: 'There is currently a lease on the resource and no lease ID was specified in the request.'. RequestId: 'b21022a6-b01f-0031-641a-453ab6000000'. TimeStamp: 'Thu, 31 Mar 2022 16:15:56 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'PreconditionFailed',Source=Microsoft.DataTransfer.ClientLibrary,'

Is there a workaround for this pipeline setup that allows me to filter within the For Each loop? I just need the .TXT files, the .PDF files can be discarded.

This was the closest reference I could find, but does not address my use case: Filter out file using wildcard path azure data factory

filterazure-data-factoryazure-data-factory-2azure-data-factory-pipeline


Solution 1:[1]

Have you tried using an If Condition activity? You can set the expression to check for the correct file extension.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Noelle

Related Questions

Error "MetaMask - RPC Error: results is not defined" appeared in none changed code

Groupby & Sum - Create new column with added If Condition

Find numbers in Python list which are within a certain distance of each other

jenkins builds filtering based on build description

I'm getting item.map is not a function when I try to delete an item with filter

Adding new data in excel and cannot be sorted or filtered

How to disable DefaultSecurityFilterChain in a Spring Boot app?

Create array of unique objects by property

module 'skimage.filters' has no attribute 'gaussian_filter'

Using jq to stream, filter large json file and save output as csv

Create a filter query for Microsoft Graph that works equivalent to "contains"

I can't put backdrop-filter: blur(); atribute in my div-container

Filter rows in csv file based on another csv file and save the filtered data in a new file

Google Cloud: Pubsub subscription filter expression options

How to filter object with multiple optionnal conditions?