'AWS Managed Apache airflow (MWAA) or AWS Batch for simple batch job flow
I have simple workflow to design where there will be 4 batch job running one after another sequentially and each jobs is running in multi node master/slave architecture.
My question is AWS Batch can manage simple workflow using job queue and can manage multi-node parallel job as well. Now, should I use AWS Batch or Airflow ?
With Airflow , I can use KubernetesPodOperator and job will run in Kubernetes cluster. But Airflow does not inherently support multi node parallel jobs.
Note: The batch job is written in java using Spring batch remote partitioning framework that support master/slave architecture.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
