Terraform - one centralized state or multiple modular states

Question

I have created a terraform stack for all the required resources that we utilise to build out a virtual data center within AWS. VPC, subnet, security groups etc, etc.

It all works beautifully :). I am having a constant argument with network engineers that want to have a completely separate state for networking etc. As a result of this we have to manage multiple state files and it requires 10 to 15 terraform plan/apply commands to being up the data center. Not alone do we have to run the commands multiple times, we cannot reference the module output variables from when creating ec2 instances etc, so now there are "magic" variables appearing within variable files. I want to put the scripts to create the ec2 instances, els etc within the same directory as the "data center" configuration so that we manage one state file (encrypted in s3 with dynamodb lock) and that our git repo has a one to one relationship with our infrastructure. There is also the added benefit that a single terraform plan/apply will build the whole datacenter in a single command.

Question is really, is it a good idea to manage data center resources (vpc, subnets, security groups) and compute resources in a single state file? Are there any issues that I may come across? Has anybody experience in managing an AWS environment with terraform this way?

Regards, David

Answer 1

During the last years there have been a lot of discussion about layouts for Terraform projects.

Times have also changed with Terraform 1.0 so I think this question deserve some love.

As a result of this we have to manage multiple state files and it requires 10 to 15 terraform plan/apply commands to being up the data center.

Using modules is possible to maintain separated states without requiring executing commands for each state.

Not alone do we have to run the commands multiple times, we cannot reference the module output variables

Terraform support output values. Leveraging Terraform Cloud or Terraform remote states is possible to introduce dependencies between states.

A prerequisite to adventure into multiple Terraform states in my opinion is using state locking (OP refers to using AWS DynamoDB lock mechanism but other Storage backend support this too).

Generally having everything in a single state is not the best solution and may be considered an anti-pattern.

Having multiple state is referred to as state isolation.

Why would you want to isolate states?

Reasons are multiple and the benefits are clear:

• bugs blast radius. If you introduce a bug somewhere in the code an you apply all the code for the entire datacenter in the worst possible scenario everything will be affected. On the other hand if networking was separated in the worst scenario the bug could only affect networking (which in a DC would be a very severe issue but still better than everything).
• state (write) lock. If you use state lock Terraform will lock the state for any operation that may possibly write to the state. This means that with a single state multiple teams working on separate areas are not able to write to the state at the same time, so updating the networking blocks instance provisioning for example.
• secrets. Secrets are written plain-text to the state. A single state means all teams secrets will end up in the same state (that you must encrypt, OP is correctly doing this). As with anything with security having all eggs in the same basket is a risk.

A side benefit of isolating state is that file layout may help with code ownership (across teams or project).

How to isolate state?

There are 3 mainly ways:

1. via file layout (with or without modules)
2. via workspaces (not to be confused with Terraform Cloud Workspaces)
3. mix up the above ways (Here be dragons!)

There is no wide consensus on how to do it but for further reading:

• https://blog.gruntwork.io/how-to-manage-terraform-state-28f5697e68fa 2016 article, I think this is sort of the root of the discussion.
• https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/
• https://www.terraform-best-practices.com/code-structure

A tool worth looking at may be Terragrunt from Gruntwork.