Integrating azure data factory and airflow

Question

My end goal is to run Azure data factory (ADF) pipelines using Airflow. My current setup is a docker file which has python packages required for this like azure data providers and a helm chart from apache airflow. I have a custom values.yaml with celery executor which I am upgrading every time to run airflow locally.

So far this part is success.

The python packages give an azure data factory option just like how I wanted but I am unable to validate these. I tested the same credentials (resource group, client id, client secret, tenant id, subscription id) using a python code and it is valid.

This doesn't work on Airflow and I see this error

*** Trying to get logs (last 100 lines) from worker pod azuredatafactoryrunpipeline.3f394072d09e4c6e8d59566776f18b78 ***

*** Unable to fetch logs from worker pod azuredatafactoryrunpipeline.3f394072d09e4c6e8d59566776f18b78 ***
(404)
Reason: Not Found

HTTP response headers: HTTPHeaderDict({'Audit-Id': '4a506d8b-9bac-44d8-8a7c-ee01810e478f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Mon, 07 Feb 2022 22:05:52 GMT', 'Content-Length': '288'})

HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \\"azuredatafactoryrunpipeline.3f394072d09e4c6e8d59566776f18b78\\" not found","reason":"NotFound","details":{"name":"azuredatafactoryrunpipeline.3f394072d09e4c6e8d59566776f18b78","kind":"pods"},"code":404}\n'

How do I debug this?

Is there a way to test ADF connection from Airflow ?