I am new to airflow automation, i dont now if it is possible to do this with apache airflow(or luigi etc) or should i just make a long bash file to do this. I
Setup: Airflow 2.0.1 with Kubernetes 1.18 and Python 3.8, Kubernetes Client: 18.17.x Pod template file: apiVersion: v1 kind: Pod metadata: name: workerPod sp
I would like to hide the field in Airflow UI: "Task Instance Details" -> Section "Task Attributes" -> Attribute "Env". I have some credentials stored in t
Requirement: To run local Airflow using official docker-compose with Airflow version 2.3.0 Issue: "You are running pip as root Please use user to run pip" Airfl
I have successfully written an API in Python to read Gmail message, URL from the message, call the URL and store CSV file, however, when I am deploying this in
I have an Airflow Http sensor that calls a REST endpoint and checks for a specific value in the JSON structure returned by the API sensor = HttpSensor( soft
I am using AWS's MWAA service (2.2.2) to run a variety of DAGs, most of which are implemented with standard PythonOperator types. I bundle the DAGs into an S3 b
I deployed the default helm chart for airflow 2. The postgres pod is reporting an error: ERROR: relation "log" does not exist at character 13 This appears af
I am new at airflow and when i click run 'ignore all dependence' on Task Instance Context Menu like this: Task Instance Context Menu It leads to 'Only works
I see that one can trigger_dag with parameters/config key-value pairs using the airflow command line: For Apache Airflow, How can I pass the parameters when ma
I would like to use DBT in MWAA Airflow enviroment. To achieve this I need to install DBT in the managed environment and from there run the dbt commands via the
Problem Is there any way in Airflow to create a workflow such that the number of tasks B.* is unknown until completion of Task A? I have looked at subdags but
How we can create the folder using gsutil command. I am using Bashoperator in airflow where I need to use the gsutil Bash command, Bucket is already created I w
Can we integrate ServiceNow with Airflow ? I mean if there is any DAG failure, can we rerun the failed DAG/tasks by submitting a ServiceNow request ? Also can w
I want to integrate Slack alert messages when a task fail in Airflow with SlackWebHookOperator. I have issue with the import of the operator. In this tutorial