How to execute long running durable Azure Functions with Azure Data Factory?

Question

My problem:

1. I have Azure Data Factory (ADF) calling Azure Function(Python) with HTTP trigger.
2. ADF provides parameters to Function by json.
3. Function processing takes over 230 seconds and timeout occurs
4. Azure Factory get timeout errors:

Solution idea:

• Use durable function and make process aync based on: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=csharp
• Use tutorial as base: https://docs.microsoft.com/en-us/azure/azure-functions/durable/quickstart-python-vscode

Question? A) If ADF calls HTTP Starter max 230 timeout occurs? Or this has no max time limit? B) How to make Durable Function to return value to ADF. (I would next request complete status with next ADF activity)

Answer 1

As per the official Microsoft document, Azure Functions times out after 230 seconds regardless of the functionTimeout setting you've configured in the settings. The solution for this is to make it a background work with Durable Functions.

Durable Functions are running on the same platform as normal Azure Functions, but they run asynchronously. When you trigger a Durable Function, it creates a background process and gives you a few URLs that you can interact with that process; including one to query its status. When you call it, it returns the status of the background job as well as the data if it’s finished.

The idea is to trigger a Durable Function through an HTTP endpoint, wait for it to finish and then get the result. Here’s how it looks like:

And here’s how it looks inside the Until activity:

Please follow the below steps:

1. First, we trigger our Durable Function through an HTTP trigger using Azure Function activity.

2. Then with the Until activity, we check status of that function.

   • The Wait activity waits around 30 seconds (or different, up to you) to let function to be executed

   • The Web activity makes a request to the statusQueryUrl that Azure Function activity returns, by calling @activity('StartUntar').output.statusQueryGetUri

3. Until activity checks the result of CheckStatus Web activity with expression @not(or(equals(activity('CheckStatus').output.runtimeStatus, 'Pending'), equals(activity('CheckStatus').output.runtimeStatus, 'Running'))).

4. It repeats until the function is finished or failed, or until it times out (set on the Timeout property).

Answer 2

I would recommend you use the Webhook activity in ADF and have your durable function perform a callback upon completion (success or failure) to the callback URI. That should be less expensive in ADF than looping for an extended period of time.