Failed to delete ECS service in Cloudformation

Question

I'm running into a small issue deleting a Cloudformation stack that has an ECS cluster and ECS services as a part of it.

If I just delete it manually from the CF console I get a failed delete with the following error:

AWS::ECS::Cluster The Cluster cannot be deleted while Container Instances are active or draining.

Following the AWS docs (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CleaningUp.html) we wrote a script to delete the clusters using the AWS cli, this script has been working great for months until Friday.

stack=$1
services="$(aws ecs list-services --cluster "$stack" | grep "$stack" | sed -e 's/"//g' -e 's/,//')"
for service in $services; do
    aws ecs update-service --cluster "$stack" --service "$service" --desired-count 0
    aws ecs delete-service --cluster "$stack" --service "$service"
done

for id in $(aws ecs list-container-instances --cluster "$stack" | grep container-instance | sed -e 's/"//g' -e 's/,//'); do
    aws ecs deregister-container-instance --cluster "$stack" --container-instance "$id" --force
done

for service in $services; do
    aws ecs wait services-inactive --cluster "$stack" --services "$service"
done

aws ecs delete-cluster --cluster "$stack"
aws cloudformation delete-stack --stack-name "$stack"

However we are now getting this new error:

AWS::ECS::Service: Service was not ACTIVE. (Service: AmazonECS; Status Code: 400; Error Code: ClientException; Request ID: xxxx-xxxx-xxxx-xxxx)

So I removed the ECS update and delete service calls thinking maybe they just need to be left ACTIVE and when we delete the ECS cluster it will take care of them. But when i do that i get this conflicting error:

AWS::ECS::Cluster The Cluster cannot be deleted while Services are active. (Service: AmazonECS; Status Code: 400; Error Code: ClusterContainsServicesException)

I assume I'm just doing something wrong, so any insight would be appreciated.

Answer 1

Raised issue with AWS support, was informed it was a bug on their end, was resolved shortly there after.

Answer 2

I've been scratching my head around this for a long time and never found a viable solution anywhere until last week. AWS has just release its new API where they have --force option for service removal. This task and service is corrupted now, only way to deal is to delete it, you cannot update it anymore.

You can use this command to delete your service now; it was impossible last week! {code}aws ecs delete-service --service my-http-service --force true{code}

Hope this helps

Answer 3

If unable to delete from console, try CLI:

 aws ecs delete-cluster --cluster <cluster_name>

This would give an error back, for example in my case -

An error occurred (ClusterContainsContainerInstancesException) when calling the DeleteCluster operation: The Cluster cannot be deleted while Container Instances are active or draining.

Check of you can delete the resources blocking deletion of cluster, in my case the EC2 instances were already terminated, but the cluster was still not delete-able !

Tried same command after few mins (5-10 mins), now it resulted in the cluster deletion giving a JSON response back indicating cluster status as "INACTIVE"