Apache Airflow get execution time difference between two DAG runs

Question

I'm currently trying to figure out how to use two dag runs' execution time to get their time difference to prevent from the DAG processing the same file multiple times. Any idea how could I check this?

Answer 1

You could use croniter, a library packaged with Airflow to do this:

>>> from airflow import DAG
>>> from datetime import timedelta, datetime
>>> from croniter import croniter
>>>
>>> dag_a = DAG(
...     dag_id="A",
...     schedule_interval="00 10 * * *",
... )
>>>
>>> dag_b = DAG(
...     dag_id="B",
...     schedule_interval="00 14 * * *",
... )
>>>
>>> now = datetime.now()
>>> cron_a = croniter(dag_a.schedule_interval, now)
>>> cron_b = croniter(dag_b.schedule_interval, now)
>>> time_difference = cron_b.get_prev(datetime) - cron_a.get_prev(datetime)
>>> time_difference
datetime.timedelta(0, 14400)
>>> time_difference.seconds
14400 # 4 hours