'Best way to manage Terraform apply infra -AWS batch- run a script (could take 1hr to 1day) - destroy infra
Hi All I am planning to build a system for my team where we can start a AWS batch infra - run a task - once job done destroy the infra.
I am thinking of : Make file steps:- 1. Terraform apply AWS batch infra, 2. Run the task, 3. Check on regular interval if the task is complete 4. If the task is complete destroy the infra.
What is most efficient way for doing this. Given our team would need wide variety of task to run on the AWS batch, we want to automate where we can just make one command do this.
Should we explore Airflow for this? Or is there better way to so this? Your thoughts would be highly appreciated. Thank you
Solution 1:[1]
Many companies have chosen to build their own Terraform Automation & Collaboration Software (TACOS) but it's a lot less work to use an existing service such as the open source atlantis or an enterprise saas platform such as spacelift or terraform cloud.
However, if you were to create your own, you would need to confirm plans safely. The tools above can use rego from open policy agent to do so.
From your workflow, it sounds like you simply need a tool to auto apply your changes. I have seen a cron job running in jenkins that can do the trick. You can also run a cron ecs or ecs fargate task on a scheduled interval. Airflow seems like overkill.
If I were you, I'd strongly look at all the options and list the pros and cons of each before considering rolling your own. I'm interested to know if the above services have shortcomings that warrant your team to build a new service.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | SomeGuyOnAComputer |