Terraform upgrade from 12.31 to 13 or later

Question

I'm trying to upgrade out terraform as I think there is a bug with dynamic blocks in terraform 12.31 for AWS (it keeps trying to recreate the security groups now in a dynamic block on each apply)

I've tried following the documentation, but I can't get past a provider error. The current terraform config is using the AWS provider version 2.70.

I've tried removing the local .terraform, reinitializing, hardcoding the AWS provider version as 4.4.0, using terraform13.upgrade but no matter what I get back an error at some point similar to:

No provider "aws" plugins meet the constraint "~> 2.70,~> 2.70,~> 2.70,~> 2.70,~> 2.70,~> 4.4.0,~> 4.4.0,~> 4.4.0".

How do I remove the version of 2.70 without damaging the resources? Is terraform state replace-provider also supposed to be used with replacing provider versions?

Any help would be great

Answer 1

Although this new error was coincident with you upgrading Terraform CLI, I believe what's really going on here is that some but not all of the modules you are using have been updated to the latest major version of the AWS provider, which includes various breaking changes that can mean that it's impossible for a single module to support both version 4 and older versions at the same time.

I expect this was working for you before because you were using earlier versions of those modules which had not yet been updated for AWS provider version 4. Indeed, AWS provider version 4 was released only in February 2022, about a month ago at the time I'm writing this, and so that is a practical horizon on how long ago those modules would've been updated to support and require it.

To move forward I expect you will need to pin your module dependencies to older versions that did not yet require AWS provider v4. You can use the terraform providers command to see which providers each of your modules depend on, including which version constraints they call for. For each one that requires Terraform v4.0.0 or later you will need to look at that module's available versions and choose an earlier one which did not yet have that requirement, and pin that version selection in your configuration.

For example, if you learn that major version 1 of one of the modules is the latest one which supports AWS provider version 2, you could write a constraint like this in the relevant module block:

module "example" {
  source  = "example/example/example"
  version = "~> 1.0.0"
}

This will then force Terraform's module installer to select only a version from the v1 series of that module.

Another possible approach is to upgrade to newer versions of the modules which currently seem to require major version 2 of the AWS provider, but I've assumed that won't be possible because those modules are not yet updated to work with the new provider. You can downgrade to an older version that's still available, but of course you cannot upgrade to a newer version that has not yet been released.

In this answer I'm assuming that the module in question is published in a Terraform module registry and therefore it's meaningful to discuss module version numbers. Terraform supports other module installation methods too, which don't support version numbers in this way but typically still offer other ways to specify exactly which source you want, as long as the older version is still available to install from somewhere.