'Airflow - How to have a task run on retry, the exact same code from the first run

Sorry if my title isn't clear. Essentially, I have a DAG that needs to only run when triggered, and runs the last sql file created in the directory (I used max and glob functions).

However, future cases may happen where a user may push a new file up while the task is running or while a task is being fails and retried thus creating a change in the code (a new file becomes the new "max" script) where the script thats being passed (the sql_path) to the operator changes. Is there a solution where the task on retry runs the exact sql script that gets passed on the first run, and not the file that got pushed up?

with dag:
    dirname=os.path.dirname(os.path.realpath(__file__))
    last_file = max(glob(f"{dirname}/*/*.sql"), key=os.path.getctime)
    sql_path = '/' + last_file.split('/')[-2] + '/' + last_file.split('/')[-1]

    run_sql = PostgresScriptOperator(
        task_id='one_time_sql_runs',
        sql=sql_path,
        task_concurrency=1,
        postgres_conn_id='my_conn_id',
    )


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source