CloudFormation stuck in UPDATE_ROLLBACK_FAILED after trying to update to latest Node version

Question

I find myself in a bit of situation here.

I have a CloudFormation with 32 nested stacks and 200+ Lambda functions. I am attempting to update all my Lambda functions from a runtime of NodeJs 6.10 to the latest NodeJs 12.X.

Near the end of the update, one of my other resources failed and caused the stack to rollback. So now, CloudFormation is attempting to rollback all my Lambda functions to Node 6.10 causing the following error:

which now results in being in an UPDATE_ROLLBACK_FAILED state.

I tried to skip all the lambda functions with a continue-update-rollback command, but I can't skip resources that haven't already failed yet, as you can see by the error message below.

Is there a way to force update_rollback the whole thing and just skip all resources? I cannot delete the stack and re-install, as this is a production instance with lots of data.

Any help is greatly appreciated.

Answer 1

For anyone who is having similar problems, I was able to get a response from AWS and the summary of it is: it cannot be done.

I was able to get my stack back to a UPDATE_ROLLBACK_COMPLETE by running continue-update-rollback until it failed on the lambdas, and then re-running it. I just kept repeating this process until all of the lambda functions were able to roll back.

It's unfortunate there is not a better solution, but I can confirm this is how it is done for the time being. I will update this if a better feature gets implemented in AWS.