'Increase priority of execution for an Airflow DAG?
Airflow has a queuing mechanism to execute queued tasks, and it does this across all the dags. I have a specific DAG which is high priority, meaning if there a task from this DAG scheduled and queued to run, I want it to be processes high priority. Is there a way to instruct airflow to do this?
Solution 1:[1]
Airflow consider priority of tasks when scheduling thus you can use priority weight to increase the ranking of specific tasks. Note that this is not a DAG parameter but an Operator parameter. If you wish to apply it to all tasks in a specific DAG use default_args.
Example:
from airflow.operators.bash import BashOperator
from airflow.utils.weight_rule import WeightRule
BashOperator(
task_id="my_task",
bash_command="echo 1",
weight_rule=WeightRule.ABSOLUTE,
priority_weight=10000,
)
You can change the weight_rule & priority_weight according to what you wish.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Elad Kalif |