Composer Airflow - No module named "exchangelib"

Question

I'm facing a very strange error while deploying a DAG into a Composer Airflow environment.

I have created a custom operator with the following import:

from exchangelib import DELEGATE, Account, Credentials, Message, FileAttachment, HTMLBody

The custom operator is defined in a file named airflow_mail_operator.py and the DAG is defined in a file named test_main.py.

The error I get is from the airflow webserver and states ModuleNotFoundError: No module named 'exchangelib' even if the Composer environment already presents this library.

The error stops showing if I add the import statement even in the DAG file, and this sounds really strange to me!

Following the full stacktrace:

Traceback (most recent call last): File "/usr/local/lib/airflow/airflow/models/dagbag.py", line 243, in process_file m = imp.load_source(mod_name, filepath) File "/opt/python3.6/lib/python3.6/imp.py", line 172, in load_source module = _load(spec) File "<frozen importlib._bootstrap>", line 684, in _load File "<frozen importlib._bootstrap>", line 665, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 678, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/home/airflow/gcs/dags/dg_dags/test_main.py", line 5, in <module> from dg_dags.airflow_mail_operator import CustomEmailOperator File "/home/airflow/gcs/dags/dg_dags/airflow_mail_operator.py", line 5, in <module> from exchangelib import DELEGATE, Account, Credentials, Message, FileAttachment, HTMLBody ModuleNotFoundError: No module named 'exchangelib'

Have someone evere faced (and maybe resolved) a similar error?

My environment is:

composer-1.10.0-airflow-1.10.6

python3

Thanks in advance!

Answer 1

The solution for me was to force an update on Composer cluster by updating the underlying k8s cluster!