Terraform does not create a new ec2 instance if I change the AMI ID

Question

I am trying to add instance behind an ASG. This is what I have come up with.

main.tf

data "aws_availability_zones" "available" {
  state = "available"
}


resource "aws_launch_template" "m-web-asg" {
  name = "m-web-asg"

  capacity_reservation_specification {
    capacity_reservation_preference = "open"
  }
  image_id = var.web_image_id
  instance_initiated_shutdown_behavior = "terminate"
  instance_type = "t2.micro"
  key_name = "keyname"

  monitoring {
    enabled = true
  }

  network_interfaces {
    security_groups             = var.m_web_server_security_group_ids
    associate_public_ip_address = true
  }

  # vpc_security_group_ids = var.m_web_server_security_group_ids

  tag_specifications {
    resource_type = "instance"

    tags = {
      Name = "test"
    }
  }

  user_data = filebase64("external-files/instance_provisioner.sh")
}


resource "aws_autoscaling_group" "m-web-asg" {
  name                 = "m-web-asg"
  min_size             = 1
  max_size             = 3
  desired_capacity     = 1
  launch_template {
    name      = aws_launch_template.m-web-asg.name
  }
  vpc_zone_identifier  = var.m_subnet_ids
  tag {
    key                 = "env"
    value               = "testing"
    propagate_at_launch = true
  }
}

resource "aws_autoscaling_attachment" "m-web-asg" {
  autoscaling_group_name = aws_autoscaling_group.m-web-asg.id
  lb_target_group_arn   = var.target_group_arn
}

resource "aws_autoscaling_policy" "scale_down" {
  name                   = "m-web-asg-scale-down"
  autoscaling_group_name = aws_autoscaling_group.m-web-asg.name
  adjustment_type        = "ChangeInCapacity"
  scaling_adjustment     = -1
  cooldown               = 120
}

resource "aws_cloudwatch_metric_alarm" "scale_down" {
  alarm_description   = "Monitors CPU utilization for m-web-asg ASG"
  alarm_actions       = [aws_autoscaling_policy.scale_down.arn]
  alarm_name          = "m-web-asg-scale-down"
  comparison_operator = "LessThanOrEqualToThreshold"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  threshold           = "30"
  evaluation_periods  = "2"
  period              = "120"
  statistic           = "Average"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.m-web-asg.name
  }
}

resource "aws_autoscaling_policy" "scale_up" {
  name                   = "m-web-asg-scale-up"
  autoscaling_group_name = aws_autoscaling_group.m-web-asg.name
  adjustment_type        = "ChangeInCapacity"
  scaling_adjustment     = 1
  cooldown               = 120
}

resource "aws_cloudwatch_metric_alarm" "scale_up" {
  alarm_description   = "Monitors CPU utilization for m-web-asg ASG"
  alarm_actions       = [aws_autoscaling_policy.scale_up.arn]
  alarm_name          = "m-web-asg-scale-up"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  namespace           = "AWS/EC2"
  metric_name         = "CPUUtilization"
  threshold           = "75"
  evaluation_periods  = "2"
  period              = "120"
  statistic           = "Average"

  dimensions = {
    AutoScalingGroupName = aws_autoscaling_group.m-web-asg.name
  }
}


terraform {
  backend "s3" {
    bucket = "terraform-m-backend"
    key = "terraform"
    region = "us-west-2"
    dynamodb_table = "terraform-m-lock"
  }  
}

variables.tf

variable m_vpc_id {
  type        = string
  default     = "vpc-xxxx"
  description = "This is the VPC ID for web servers of m AWS"
}

variable m_subnet_ids {
  type        = list(string)
  default     = ["subnet-xxxx"]
  description = "This is the Subnet ID for web servers of m AWS"
}

variable m_web_server_security_group_ids {
  type        = list(string)
  default     = ["sg-xxxx", "sg-yyyy"]
  description = "This is the Security Groups ID for web servers of m AWS"
}

variable target_group_arn {
  type        = string
  default     = "arn:aws:elasticloadbalancing:us-west-2:xxx:targetgroup/testing/xxxx"
  description = "This is the target_group_arn for web servers of m AWS"
}

variable web_image_id {
  description = "This is the image id that we will use to bring up web server"
  type        = string
  default     = "ami-xxxx"
}

When I initially do a terraform init/plan/apply everything works fine.

Now if I change the variable web_image_id from "ami-xxxx" to "ami-yyyy", terraform plan does shows this;

# aws_launch_template.m-web-asg will be updated in-place

and then upon apply noting changes. The old ec2 instance keep running as it is.

I was expecting that the change in the AMI would kill the existing instance and create a new instance with the new AMI.

What am I missing?

Answer 1

Your code only updates aws_launch_template.m-web-asg[aws_launch_template.m-web-asg when you change AMI. This will not affect currently running instances in your ASG. What's going to happen is then when the ASG scale-up, the new instances will be running with the new AMI, while old ones will be still running the old AMI.

What you could do is setup a null_resource that triggers on your aws_launch_template.m-web-asg[aws_launch_template changes. The null_resource would use local-exec to perform ASG refresh using AWS CLI.

Answer 2

Thanks to @Marcin I read about instance_refresh and added the following piece of code into aws_autoscaling_group

  instance_refresh {
    strategy = "Rolling"
    preferences {
      // You probably want more than 50% healthy depending on how much headroom you have
      min_healthy_percentage = 80
      instance_warmup        = 10

    }
  }

This makes sure that any change that happens in the launch template get reflected into the existing instances.