Terraform: How to create API Gateway endpoints and methods from a list of objects?

Question

I want to create a terraform (v0.12+) module that outputs an AWS API Gateway with Lambda integration(s). I cannot quite understand how (or if it is even possible) to iterate over a list of maps to dynamically output resources.

The user should be able to instantiate the module like so:

module "api_gateway" {
  source = "./apig"

  endpoints = [
    {
      path = "example1"
      method = "GET"
      lambda = "some.lambda.reference"
    },
    {
      path = "example1"
      method = "POST"
      lambda = "some.lambda.reference"
    },
    {
      path = "example2"
      method = "GET"
      lambda = "another.lambda.reference"
    }
  ]
}

From the endpoints interface, I want to output three resources:

1. an aws_api_gateway_resource where path_part = endpoint[i].path
2. an aws_api_gateway_method where http_method = endpoint[i].method
3. an aws_api_gateway_integration that takes a reference to endpoint[i].lambda, etc

Terraform's for_each property doesn't seem robust enough to handle this. I know Terraform also supports for loops and for / in loops, but I have not been able to find any examples of using such expressions for resource declaration.

Answer 1

Let's start off by writing out the declaration of that endpoints variable, since the rest of the answer depends on it being defined this way:

variable "endpoints" {
  type = set(object({
    path   = string
    method = string
    lambda = string
  })
}

The above says that endpoints is a set of objects, which means that the ordering of the items is not significant. The ordering is insignificant because we're going to create separate objects in API for each one anyway.

The next step is to figure out how to move from that given data structure into a structure that is a map where each key is unique and where each element maps to one instance of the resources you want to produce. To do that we must define what mapping we're intending, which I think here would be:

• One aws_api_gateway_resource for each distinct path.
• One aws_api_gateway_method for each distinct path and method pair.
• One aws_api_gateway_integration for each distinct path and method pair.
• One aws_api_gateway_integration_response for each distinct path/method/status_code triple.
• One aws_api_gateway_method_response for each distinct path/method /status_code triple.

So it seems that we need three collections here: first is a set of all of the paths, second is a map from a path+method pair to the object that describes that method, and third is every combination of endpoints and status codes we want to model.

locals {
  response_codes = toset({
    status_code         = 200
    response_templates  = {} # TODO: Fill this in
    response_models     = {} # TODO: Fill this in
    response_parameters = {} # TODO: Fill this in
  })

  # endpoints is a set of all of the distinct paths in var.endpoints
  endpoints = toset(var.endpoints.*.path)

  # methods is a map from method+path identifier strings to endpoint definitions
  methods = {
    for e in var.endpoints : "${e.method} ${e.path}" => e
  }

  # responses is a map from method+path+status_code identifier strings
  # to endpoint definitions
  responses = {
    for pair in setproduct(var.endpoints, local.response_codes) :
    "${pair[0].method} ${pair[0].path} ${pair[1].status_code}" => {
      method              = pair[0].method
      path                = pair[0].path
      method_key          = "${pair[0].method} ${pair[0].path}" # key for local.methods
      status_code         = pair[1].status_code
      response_templates  = pair[1].response_templates
      response_models     = pair[1].response_models
      response_parameters = pair[1].response_parameters
    }
  }
}

With these two derived collections defined, we can now write out the resource configurations:

resource "aws_api_gateway_rest_api" "example" {
  name = "example"
}

resource "aws_api_gateway_resource" "example" {
  for_each = local.endpoints

  rest_api_id = aws_api_gateway_rest_api.example.id
  parent_id   = aws_api_gateway_rest_api.example.root_resource_id
  path_part   = each.value
}

resource "aws_api_gateway_method" "example" {
  for_each = local.methods

  rest_api_id = aws_api_gateway_resource.example[each.value.path].rest_api_id
  resource_id = aws_api_gateway_resource.example[each.value.path].resource_id
  http_method = each.value.method
}

resource "aws_api_gateway_integration" "example" {
  for_each = local.methods

  rest_api_id = aws_api_gateway_method.example[each.key].rest_api_id
  resource_id = aws_api_gateway_method.example[each.key].resource_id
  http_method = aws_api_gateway_method.example[each.key].http_method

  type                    = "AWS_PROXY"
  integration_http_method = "POST"
  uri                     = each.value.lambda
}

resource "aws_api_gateway_integration_response" "example" {
  for_each = var.responses

  rest_api_id = aws_api_gateway_integration.example[each.value.method_key].rest_api_id
  resource_id = aws_api_gateway_integration.example[each.value.method_key].resource_id
  http_method = each.value.method
  status_code = each.value.status_code

  response_parameters = each.value.response_parameters
  response_templates  = each.value.response_templates

  # NOTE: There are some other arguments for
  # aws_api_gateway_integration_response that I've left out
  # here. If you need them you'll need to adjust the above
  # local value expressions to include them too.
}

resource "aws_api_gateway_response" "example" {
  for_each = var.responses

  rest_api_id = aws_api_gateway_integration_response.example[each.key].rest_api_id
  resource_id = aws_api_gateway_integration_response.example[each.key].resource_id
  http_method = each.value.method
  status_code = each.value.status_code

  response_models     = each.value.response_models
}

You'll probably also need an aws_api_gateway_deployment. For that, it's important to make sure it depends on all the API gateway resources we've defined above so that Terraform will wait until the API is fully configured before trying to deploy it:

resource "aws_api_gateway_deployment" "example" {
  rest_api_id = aws_api_gateway_rest_api.example.id

  # (whatever other settings are appropriate)

  depends_on = [
    aws_api_gateway_resource.example,
    aws_api_gateway_method.example,
    aws_api_gateway_integration.example,
    aws_api_gateway_integration_response.example,
    aws_api_gateway_method_response.example,
  ]
}

output "execution_arn" {
  value = aws_api_gateway_rest_api.example.execution_arn

  # Execution can't happen until the gateway is deployed, so
  # this extra hint will ensure that the aws_lambda_permission
  # granting access to this API will be created only once
  # the API is fully deployed.
  depends_on = [
    aws_api_gateway_deployment.example,
  ]
}

---

API Gateway details aside, the general procedure for situations like this is:

• Define your input(s).
• Figure out how to get from your inputs to collections that have one element per instance you need for each resource.
• Write local expressions to describe that projection from input to the repetition collection.
• Write resource blocks where for_each refers to the appropriate local value as its repetition value.

for expressions, along with the flatten and setproduct functions, are our primary tool for projecting data from a structure that is convenient for the caller to provide in an input variable to the structure(s) we need for for_each expressions.

API Gateway has a particularly complex data model though, and so expressing all of its possibilities within the Terraform language can require a lot more projection and other transformation than might be required for other services. Because OpenAPI already defines a flexible declarative language for defining REST APIs and API Gateway already natively supports it, it could be more straightforward and flexible to make your endpoints variable take a standard OpenAPI definition and pass it directly to API Gateway, thus getting all the expressiveness of the OpenAPI schema format without having to implement all the details in Terraform yourself:

variable "endpoints" {
  # arbitrary OpenAPI schema object to be validated by API Gateway
  type = any
}

resource "aws_api_gateway_rest_api" "example" {
  name = "example"
  body = jsonencode(var.endpoints)
}

Even if you do still want your endpoints variable to be a higher-level model, you could also consider using the Terraform language to construct an OpenAPI schema by deriving a data structure from var.endpoints and finally passing it to jsonencode.

Answer 2

have a configuration file (json)

#configuration.json
{
  "lambda1": {
    "name": "my-name",
    "path": "my-path",
    "method": "GET"
  },
 "lambda2": {
    "name": "my-name2",
    "path": "my-path2",
    "method": "GET"
  },

}

and the following terraform

locals {
  conf = jsondecode(file("${path.module}/configuration.json"))
  name="name"
}

data "aws_caller_identity" "current" {}


resource "aws_lambda_permission" "apigw_lambda" {
  for_each      = local.conf
  statement_id  = "AllowExecutionFromAPIGateway"
  action        = "lambda:InvokeFunction"
  function_name = each.value.name
  principal     = "apigateway.amazonaws.com"

  # More: http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-control-access-using-iam-policies-to-invoke-api.html
  source_arn = "arn:aws:execute-api:${var.region}:${data.aws_caller_identity.current.account_id}:${aws_api_gateway_rest_api.api.id}/*/${aws_api_gateway_method.methods[each.key].http_method}${aws_api_gateway_resource.worker-path[each.key].path}"
}

resource "aws_api_gateway_rest_api" "api" {
  name        = local.name
  description = "an endpoints...."
  endpoint_configuration {
    types = ["REGIONAL"]
  }
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_resource" "country-endpoint" {
  rest_api_id = aws_api_gateway_rest_api.api.id
  parent_id   = aws_api_gateway_rest_api.api.root_resource_id
  path_part   = local.country-code # https.exmaple.com/stage/uk
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_resource" "worker-path" {
  for_each    = local.conf
  rest_api_id = aws_api_gateway_rest_api.api.id
  parent_id   = aws_api_gateway_resource.country-endpoint.id
  path_part   = each.value.path # https.exmaple.com/stage/uk/path_from_json
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_method" "methods" {
  for_each      = local.conf
  http_method   = each.value.method
  resource_id   = aws_api_gateway_resource.worker-path[each.key].id
  rest_api_id   = aws_api_gateway_rest_api.api.id
  authorization = "NONE"
}


resource "aws_api_gateway_integration" "lambda-api-integration-get-config" {
  for_each      = local.conf
  # The ID of the REST API and the endpoint at which to integrate a Lambda function
  resource_id   = aws_api_gateway_resource.worker-path[each.key].id
  rest_api_id   = aws_api_gateway_rest_api.api.id
  # The HTTP method to integrate with the Lambda function
  http_method = aws_api_gateway_method.methods[each.key].http_method
  # AWS is used for Lambda proxy integration when you want to use a Velocity template
  type = "AWS_PROXY"
  # The URI at which the API is invoked
  uri = data.terraform_remote_state.workers.outputs.lambda_invoke[each.key]
  integration_http_method = "POST"
}