Category "airflow-2.x"

An error occurred (403) when calling the HeadObject operation: Forbidden in airflow (2.0.0)+

Error - *** Failed to verify remote log exists s3://airflow_test/airflow-logs/demo/task1/2022-05-13T18:20:45.561269+00:00/1.log. An error occurred (403) when ca

Is it possible to use an XCom from an operator parameter without using Jinja?

Is it possible to pass an XCom to an operator parameter without using a Jinja template? I have a dict stored in an XCom and I need to pass it to an Operator tha

Defining complex workflow dependency in airflow 2.0 taskflow API

Let's say I have the follow dummy DAG defined as below: @dag(default_args=default_args, schedule_interval=None, start_date=days_ago(2)) def airflow_ta

Error in airflow 2 even though my task's have actually completed? ERROR - Could not serialize the XCom value into JSON

Hi All so my dag actully runs fine, all the outputs are working but airflow's UI does not change to succes and fails due to the following issue. Reading online

set priority for the multiple dag runs

I have a dag that I want to run multiple times say 30. But airflow can parallelly execute 16 dag runs at a time. Suppose one dag run takes longer time to execut

Airflow Subdag tasks are stuck in None state while subdag is showing as Running

I have a problem with my dag getting stuck at subdag. The subdag is in RUNNING state but on zooming in all the tasks of the subdag are in None status. Using Air

DAG run as per timezone

I want to run my dag as per new york time zone. As the data comes as per the new york time zone and dag fails for the initial runs and skips last runs data as w

Airflow pool - less priority task is triggering first

I've been using the airflow pool to control my concurrent tasks. so I've created a test_pool with 10 slots and have created 4 tasks, out of which I have assigne

Airflow Scheduler liveness probe crashing (version 2.0)

I have just upgraded my Airflow from 1.10.13 to 2.0. I am running it in Kubernetes (AKS Azure) with Kubernetes Executor. Unfortunately, I see my Scheduler getti

Creating dynamic workflows for Airflow tasks present in a Python list

I have a list of lists in the following way - [['X_API', 'Y_API',....], ['Z_API', 'P_API', ...], [....], [...] .... ] Here, each API name corresponds to a Pytho

Airflow:Docker-Compose:"You are running pip as root Please use user to run pip"

Requirement: To run local Airflow using official docker-compose with Airflow version 2.3.0 Issue: "You are running pip as root Please use user to run pip" Airfl

airflow kubernetes default setup ERROR: relation "log" does not exist at character 13

I deployed the default helm chart for airflow 2. The postgres pod is reporting an error: ERROR: relation "log" does not exist at character 13 This appears af

How to load environment variable from azure key-vault in docker compose

Here I want to load the sql connection string from azure keyvault. version: '3' x-airflow-common: &airflow-common image: ${AIRFLOW_IMAGE_NAM

Other Categories