Improvements in Airflow v2.2.2

Question

Let me know if any of these got improvements.

1. Airflow 2.0.2 and lower constantly faces issues with respect to the mini scheduler.

[2022-02-21 08:40:23,027] {{process_utils.py:100}} INFO - Sending Signals.SIGTERM to GPID 4081
[2022-02-21 08:41:23,036] {{process_utils.py:113}} WARNING - process psutil.Process(pid=4081, name='airflow task ru', status='sleeping', started='08:40:17') did not respond to SIGTERM. Trying SIGKILL
[2022-02-21 08:41:23,044] {{process_utils.py:66}} INFO - Process psutil.Process(pid=4081, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='08:40:17') (4081) terminated with exit code Negsignal.SIGKILL
[2022-02-21 08:41:23,044] {{standard_task_runner.py:130}} ERROR - Job 9926 was killed before it finished (likely due to running out of memory)

And the recommendation is to change schedule_after_task_execution to False if True which is the default.

Task supervisor process runs a "mini scheduler". However, after marking it as success, it will call _run_mini_scheduler_on_child_tasks. And while local_task_job.py will detect the task as success and not running, it will terminate the process which might be still executing under _run_mini_scheduler_on_child_tasks.

Sometimes this is undesirable.

2. Airflow scheduler constantly parses the python file for building and running the tasks. Any code in DAG file but that is outside the task runs multiple times even though the DAG is scheduled to run just once a day. Any improvements to this?

Answer 1

1. There is no reason to change this setting unless you experience problems. Example for when you might consider changing it is in https://stackoverflow.com/a/71196200/14624409

2. No. Airflow must scan the DAG folder searching for new DAGs so any top level code that you have in your dag .py files will be parsed every min_file_process_interval which defaults to 30 seconds. There will be future improvements on somehow related area in 2.3.0 when AIP-42 Dynamic Task Mapping will be completed and allow map-reduce like design