'How to get conf value from airflow dag?
I want to get conf value from dag area.
"{{ dag_run.conf['company'] }}" is recognized as a string.
How can I get this value?
Values are passed fine when calling other dags.
t_trigger = TriggerDagRunOperator(
task_id="t-trigger",
trigger_dag_id="example_dag",
conf={
"company": "{{ dag_run.conf['company'] }}",
},
)
However, in the dag area the value is recognized as a string.
t_task_a = PythonOperator(
task_id="t-task-a",
python_callable=task-a,
)
employees = Variable.get(
"{{ dag_run.conf['company'] }}", # problem
default_var=['company'],
deserialize_json=True
)
for employee in employees:
t_employee_operator = PythonOperator(
task_id=f"t-test-operator",
python_callable=employee_operator,
op_kwargs={"employee": employee}
)
t_task_a >> t_employee_operator
Solution 1:[1]
As you already noticed Airflow does not render templates outside of Operators scope - this is expected. You can read this answer for more information about it.
Furthermore you are using Variable.get() as top level code which is very abusive. I explained the reasons in detail in this answer.
I would suggest to redesign your DAG structure to fit Airflow DAG writing practices.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Elad Kalif |
