I am trying to run a scala jar file from AIRFLOW using emr and the jar file is designed to read mssql-jdbc and postgresql. From airflow, I'm able to create clus
Airflow 1.10.14 and composer 1.15.2, google Bigtable in GCP I'm getting this issue {taskinstance.py:1152} ERROR - <_MultiThreadedRendezvous of RPC that termi
I'm trying to configure a SSH connection from Airflow UI on google cloud composer environment to an on premise posgresql server Where I should store my private
**mlflow training code:** import mlflow from mlflow.tracking import MlflowClient client = MlflowClient() " training the model and saving the model artificats"
We have a database containing around ~20 Million records with an indexed field order_id. Every day, after each hour, we receive an incremental update, starting
Hi All I am planning to build a system for my team where we can start a AWS batch infra - run a task - once job done destroy the infra. I am thinking of : Make
I want to get conf value from dag area. "{{ dag_run.conf['company'] }}" is recognized as a string. How can I get this value? Values are passed fin
When using pandas pd.read_excel() in an airflow task inside a container I get the openpyxl error below. I tried installing openpyxl using poetry and even using
I am facing an issue where my DAG isn't importing into Airflow due to a "ModuleNotFoundError: No Module Name package" error. The provider throwing the error sho
I would like to ask the question about whether command of Apache Airflow is work or not. Thank you. OS Version: Ubuntu 20.04.4 LTS Apache Airflow Version: 2.2.5
I'm trying to use airflow to orchestrate a workflow where a neo4j docker is run and then a python script to query data from the neo4j database (the code runs on
I started a backfill job in airflow and ran into an issue with the dag definition half way through the backfill. The code changed required a tiny tweak to an ex
I have a function run_step that produces a dynamic number of emr tasks within a task group. I want to keep this function in a separate file named helpers.py so
I am running a python project through DAG in airflow, and I encounter the following exception when the dag runs this line from the project - df = spark.sql(quer
I have been trying to source data from S3 bucket to GCS with provided operator: S3ToGCSOperator. Below is the error I am getting: ______________________________
Does anyone use MWAA in production? We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unkno
Decided to try and really understand the docker-compose.yaml file for airflow. At the beginning of the file there is this piece of code. x-airflow-common: &am
I try to connect to Azure Data Lake using Airflow. I use Airflow connection via the Web UI. When I try to connect using the test button, I get an error Bad Requ
I have a deployment of Airflow running in a kubernetes cluster. I deployed it there using the official helm chart (described here). I manage DAGs using the reco
I try to configure Secret Manager for my Composer (ver 1.16, airflow 1.10) but I have a weird situation like below. In my Composer, I've used a variable.json fi