Amazon ECS: Target Group Health Check vs Container Health Check

Question

Amazon ECS supports two different types of health checks:

• Target Group health checks make a configurable network request
• Container health checks run in the docker container and can be configured to run any shell command that the container supports

If both health checks are configured, which one wins? If either fails is the Service marked as UNHEALTHY? Or both? Can I configure one to override the other?

I'd very much like the Target Group health status to not cause ECS to continually bounce the service and I was hoping the container Health Check could be used to override it.

Answer 1

The AWS documentation is somewhat vague on this topic, but does suggest a high degree of coupling between ALB & ECS when it comes to health checks. i.e. see the documentation for healthCheckGracePeriodSeconds and minimumHealthyPercent for examples of ECS health check behaviour which is influenced by the presence or absence of a load balancer.

The healthCheckGracePeriodSeconds may be useful to avoid a failed ALB health check from causing the ECS container to be restarted (during service startup at least):

The period of time, in seconds, that the Amazon ECS service scheduler should ignore unhealthy Elastic Load Balancing target health checks, container health checks, and Route 53 health checks after a task enters a RUNNING state. This is only valid if your service is configured to use a load balancer. If your service has a load balancer defined and you do not specify a health check grace period value, the default value of 0 is used.

If your service's tasks take a while to start and respond to health checks, you can specify a health check grace period of up to 2,147,483,647 seconds during which the ECS service scheduler ignores the health check status. This grace period can prevent the ECS service scheduler from marking tasks as unhealthy and stopping them before they have time to come up.

Answer 2

In my experience, either one will cause your container to be decommissioned. I would say you probably don't need the container health check if you have a target group performing the check.