'EMR Cluster AutoTerminate by Idling
It may sound quite simple: I need to set an autotermination flag in an EMR Cluster by idling. Currently I have an Airflow DAG, and although the DAG can kill the Cluster through an error callback, my SRE team demands some EMR specific configurations and my pipeline will not be approved to production without those. I've tried a lot of options, but the only form I found to enable the autotermination was using a flag in the awscli, which not helps me since the Airflow Operator demands JSON configs. I was thinking about creating a personalized Airflow Operator just for it, but sounded over-engineering.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
