'Airflow - How to have a task run on retry, the exact same code from the first run
Sorry if my title isn't clear. Essentially, I have a DAG that needs to only run when triggered, and runs the last sql file created in the directory (I used max and glob functions).
However, future cases may happen where a user may push a new file up while the task is running or while a task is being fails and retried thus creating a change in the code (a new file becomes the new "max" script) where the script thats being passed (the sql_path) to the operator changes. Is there a solution where the task on retry runs the exact sql script that gets passed on the first run, and not the file that got pushed up?
with dag:
dirname=os.path.dirname(os.path.realpath(__file__))
last_file = max(glob(f"{dirname}/*/*.sql"), key=os.path.getctime)
sql_path = '/' + last_file.split('/')[-2] + '/' + last_file.split('/')[-1]
run_sql = PostgresScriptOperator(
task_id='one_time_sql_runs',
sql=sql_path,
task_concurrency=1,
postgres_conn_id='my_conn_id',
)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
