terraform: filter list of maps based on key

Question

I'm implementing a security group modules such that it will create security group rules by taking & filtering cidr & source_security_group_id to create a security group rule.

The current module configuration.

securty_group_module.tf

resource "aws_security_group" "this" {
  name                   = var.name
  description            = var.description
  vpc_id                 = var.vpc_id
  revoke_rules_on_delete = var.revoke_rules_on_delete
}

## CIDR Rule

resource "aws_security_group_rule" "cidr_rule" {
  count = length(var.security_group_rules)

  type              = var.security_group_rules[count.index].type
  from_port         = var.security_group_rules[count.index].from_port
  to_port           = var.security_group_rules[count.index].to_port
  protocol          = var.security_group_rules[count.index].protocol
  cidr_blocks       = var.security_group_rules[count.index].cidr_block
  description       = var.security_group_rules[count.index].description
  security_group_id = aws_security_group.this.id
}

## Source_security_group_id Rule

resource "aws_security_group_rule" "source_sg_id_rule" {
  count = length(var.security_group_rules)

  type              = var.security_group_rules[count.index].type
  from_port         = var.security_group_rules[count.index].from_port
  to_port           = var.security_group_rules[count.index].to_port
  protocol          = var.security_group_rules[count.index].protocol
  source_security_group_id = var.security_group_rules[count.index].source_security_group_id
  description       = var.security_group_rules[count.index].description
  security_group_id = aws_security_group.this.id
}

main.tf

module "sample_sg" {
  source            = "./modules/aws_security_group"
  name              = "test-sg"
  vpc_id            = "vpc-xxxxxx"

  security_group_rules = [
    { type = "ingress", from_port = 22, to_port = 22, protocol = "tcp", cidr_block = [var.vpc_cidr], description = "ssh" },
    { type = "ingress", from_port = 80, to_port = 80, protocol = "tcp", cidr_block = [var.vpc_cidr], description = "http" },
    { type = "ingress", from_port = 0, to_port = 0, protocol = "-1", source_sg_id = "sg-xxxx", description = "allow all" }
    { type = "egress",  from_port = 0, to_port = 0, protocol = "-1", source_sg_id = "sg-xxxx", description = "allow all" }
  ]
}

So, the problem statement here is when I call the security group rules in the module with the above list of maps, it should check if it is source_sg_id or cidr.

Then filter those maps & pass it to respective resources in the module.

Ex:

module ""{
...

  security_group_rules = [
    { type = "ingress", from_port = 22, to_port = 22, protocol = "tcp", cidr_block = [var.vpc_cidr], description = "ssh" },
    { type = "ingress", from_port = 0, to_port = 65535, protocol = "-1", source_sg_id = "sg-xxxx", description = "allow all" }
  ]
}

These rules should be looked up & pass the first one to CIDR rule & second one to Source_security_group_id rule.

I'm thinking of making it as below

locals {

  sid_rules = some_function{var.security_group_rules, "source_security_group_id"}
  cidr_rules = some_function{var.security_group_rules, "cidr"}
}


resource "aws_security_group_rule" "cidr_rule" {
  count = count(local.cidr_rules)

  ....
  cidr_blocks       = local.cidr_rules[count.index].cidr_block
  ....
}


resource "aws_security_group_rule" "sid_rule" {
  count = count(local.sid_rules)

  ....
  source_security_group_id  = local.sid_rules[count.index].source_sg_id
  ....
}

So, I'm looking for a way to filter the maps from list based on a key

I have tried lookup but was no help in case of list of string.

Answer 1

I figured out a clever way to do this.

Let's say I am trying to filter only the pets that are cats kind = "cat" from a list of pets.

variable "pets" {
  type = list(object({
    name = string
    kind = string
  }))
  default = [
    {
      name = "Fido"
      kind = "dog"
    },
    {
      name = "Max"
      kind = "dog"
    },
    {
      name = "Milo"
      kind = "cat"
    },
    {
      name = "Simba"
      kind = "cat"
    }
  ]
}

1. First convert the list of pets to a map pets_map of pets using the index tostring(i) as the key. This will be used in step 3 to lookup the filtered pets.

locals {
  pets_map = { for i, pet in var.pets : tostring(i) => pet }
}

2. Next create a filtered list of the keys that respectively matches the condition pet.kind == "cat" by looping over the keys in the pets_map and setting the respective keys that do not match to an empty string. Then compact the list which removes the empty strings from the list.

locals {
  cats_keys = compact([for i, pet in local.pets_map : pet.kind == "cat" ? i : ""])
}

3. Loop over the filtered keys cats_keys and lookup the respective pet from the pets_map. Now you have the filtered list of pets that are cats kind = "cat".

locals {
  cats     = [for key in local.cats_keys : lookup(local.pets_map, key)]
}

You can now access the cats with local.cats, which will give you the following map.

{
  name = "Milo"
  kind = "cat"
},
{
  name = "Simba"
  kind = "cat"
}

Below is the full example.

variable "pets" {
  type = list(object({
    name = string
    kind = string
  }))
  default = [
    {
      name = "Fido"
      kind = "dog"
    },
    {
      name = "Max"
      kind = "dog"
    },
    {
      name = "Milo"
      kind = "cat"
    },
    {
      name = "Simba"
      kind = "cat"
    }
  ]
}

locals {
  pets_map = { for i, pet in var.pets : tostring(i) => pet }
  cats_keys = compact([for i, pet in local.pets_map : pet.kind == "cat" ? i : ""])
  cats     = [for key in local.cats_keys : lookup(local.pets_map, key)]
}

Answer 2

Consider creating another module to handle the rules, and setting the security group resources inside that module.

module "security_groups" {
  count             = length(var.security_group_rules)
  source_sg_id_rule = var.security_group_rules[count.index].source_sg_id_rule
}

Then, in the new module, use a count statement as a test to create optional items:

resource "aws_security_group_rule" "source_sg_id_rule" {
    count = length(var.source_sg_id_rule) == 0 ? 0 : 1

    type              = var.type
    from_port         = var.from_port
    to_port           = var.to_port
    protocol          = var.protocol
    source_security_group_id = var.source_security_group_id
    description       = var.description
    security_group_id = var.security_group_id
}

This will create the resources as an array of one or zero items, and drop any lists of zero.

Answer 3

Thanks for the response @dan-monego.

I sorted it out with single module itslef.

Following is the module file.

aws_sg_module.tf

# Security group
##########################

resource "aws_security_group" "this" {
  name                   = var.name
  description            = var.description
  vpc_id                 = var.vpc_id
  revoke_rules_on_delete = var.revoke_rules_on_delete

  tags = merge(
    {
      "Name" = format("%s", var.name)
    },
    local.default_tags,
    var.additional_tags
  )
}

resource "aws_security_group_rule" "cidr" {
  count = var.create ? length(var.cidr_sg_rules) : 0

  type              = var.cidr_sg_rules[count.index].type
  from_port         = var.cidr_sg_rules[count.index].from
  to_port           = var.cidr_sg_rules[count.index].to
  protocol          = var.cidr_sg_rules[count.index].protocol
  cidr_blocks       = var.cidr_sg_rules[count.index].cidr
  description       = var.cidr_sg_rules[count.index].description
  security_group_id = local.this_sg_id
}

resource "aws_security_group_rule" "source_sg" {
  count = var.create ? length(var.source_sg_rules) : 0

  type                     = var.source_sg_rules[count.index].type
  from_port                = var.source_sg_rules[count.index].from
  to_port                  = var.source_sg_rules[count.index].to
  protocol                 = var.source_sg_rules[count.index].protocol
  source_security_group_id = var.source_sg_rules[count.index].source_sg_id
  description              = var.source_sg_rules[count.index].description
  security_group_id        = local.this_sg_id
}
resource "aws_security_group_rule" "self" {
  count = var.create ? length(var.self_sg_rules) : 0

  self              = true
  type              = var.source_sg_rules[count.index].type
  from_port         = var.source_sg_rules[count.index].from
  to_port           = var.source_sg_rules[count.index].to
  protocol          = var.source_sg_rules[count.index].protocol
  description       = var.source_sg_rules[count.index].description
  security_group_id = local.this_sg_id
}

Call it using following module block.

security_groups.tf

module "stack_sg" {
  source            = "./modules/aws_security_group"
  name                = "stack-sg"

  vpc_id = module.network.vpc_id

  cidr_sg_rules = [
    { type = "ingress", from = 80, to = 80, protocol = "tcp",  cidr = [module.network.vpc_cidr], description = "http" },
    { type = "egress", from = 0, to = 65535, protocol = "-1",  cidr = ["0.0.0.0/0"], description = "allow all " }
  ]

  source_sg_rules = [
    { type = "ingress", from = 0, to = 65535, protocol = "tcp", source_sg_id = module.alb_sg.sg_id, description = "alb" }
  ]
}

Answer 4

In order to filter a list of maps using a specific key value. You can use the following simple statement:

Assuming:

• key is map's key you're filtering on
• val is the value of the key
• list is the original list of maps

element([ 
    for element in list) : env 
      if element.key == "val" 
   ], 0)

the result of the above statement will be a map.

Answer 5

Given:

[{foo: "...", baz: "..."}, ...]

And you want the baz of some element in this list, use:

element([
    for o in [{"foo"="kick","baz"="5"},{"foo"="bar","baz"="100"}] : 
        o if o.foo == "bar"
],0).baz