'Airflow - pass data between DAGs

I have a DAG that will has some TriggerDagRunOperator tasks that call child DAGs. I need the child DAGs to operate on a list of client, and the list is stored in a database. I would prefer to have the parent DAG query for the list, and then pass the same list to all of the child DAGs. The child DAGs use the list to dynamically create some of the tasks - one for each client. My first attempt to do this looked like this:

    clients = [ {"client_id": 1}, {"client_id": 54}, ...]
    ...  
    some_child_operations_dag = TriggerDagRunOperator(
                                task_id='some_child_operations_dag',
                                trigger_dag_id="some_child_operations_dag",
                                wait_for_completion=True,
                                execution_date=now(),
                                conf={"clients": clients}
                            )

In the child DAG, I try to retrieve the clients list from the config like this:

def get_clients():
    return "{{ dag_run.conf['clients'] }}"

...
clients = get_clients()

...
    for client in clients:
        client_id = client['client_id']
        client_var = "client_id: " + client_id
        with TaskGroup(group_id=client_id + '_client_tasks') as client_task_group:
            client_setup_work = GKEStartPodOperator(task_id="client_" + client_id,...

This doesn't work. Airflow complains that the DAG is broken if fails on client_id = client['client_id'] with the message "TypeError: string indices must be integers".

Looking more closely at the documentation, I saw this:

The parameters from dag_run.conf can only be used in a template field of an operator.

So I think that my attempt to pass data between DAGs failed because I'm not referencing the dag_run.conf in a template field of an operator.

So what's the next best way to pass data from a parent DAG to a child DAG? Would xcoms work? My understanding is that they are only to be used for communication between tasks of the same DAG.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source