'How to configure Apache Airflow with Celery, SQS and Postgres
I am having problem with setting airflow configuration using SQS and Postgres.
I did following configuration in airflow.cfg
Celery configuration:
result_backend = db+postgresql://xxx:yyy@zzz:5432/airflow
broker_url = sqs:// #Configure IAM role for the SQS on EC2
Airflow Configuration:
executor = CeleryExecutor
sql_alchemy_conn = postgresql+psycopg2://xxx:yyy@yyy:5432/airflow_db
When I start airflow webserver, it is working fine without any issue, but when I starts airflow celery worker I getting following error
\[2022-03-25 02:03:49 -0500\] \[16054\] \[ERROR\] Can't connect to ('0.0.0.0', 8793)
\[2022-03-25 02:04:31,329: WARNING/MainProcess\] No hostname was supplied.
Reverting to default 'None'
\[2022-03-25 02:04:31,329: ERROR/MainProcess\] consumer: Cannot connect to **sqs://localhost//**: Could not connect to the endpoint URL: "https://queue.amazonaws.com/".
Trying again in 2.00 seconds... (1/100)
Please help if anyone knows the solution for this.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|