'To trigger a Databricks job in one region by an airflow job running in another region
We plan to have one common AWS MWAA cluster in us west region which triggers Databricks jobs in different regions.
Is there a way to trigger a Databricks job in one region by an airflow job running in another region? I checked the databricks connection document as here but it does not list any region parameters. How to achieve this?
Solution 1:[1]
Airflow operator doesn't know anything about regions - it just needs URL of the workspace, and personal access token. That workspace could be in the same region, or in another - it doesn't matter to a Databricks operator.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Alex Ott |