How can I get my CloudFormation stack to update if it's stuck in the UPDATE_ROLLBACK_FAILED state?

3 minute read
1

My AWS CloudFormation stack is stuck in the UPDATE_ROLLBACK_FAILED state.

Short description

If your stack is stuck in the UPDATE_ROLLBACK_FAILED state after a failed update, then the only actions that you can perform on the stack are the ContinueUpdateRollback or DeleteStack operations. This is because CloudFormation requires further input from you to acknowledge that the stack is out of sync with the template that the stack is attempting to roll back to. To retry the rollback and resolve the error, you can use ContinueUpdateRollback.

Tip: To resolve the error, you might need to raise limits, change permissions, or modify other settings. For information on common errors that cause update rollback failures, see Update rollback failed.

In some cases, retrying the rollback doesn't resolve the error. In these cases, you must skip resources, and also acknowledge that these resources no longer reflect the desired state in the CloudFormation template. To skip the resources that caused the rollback failures, complete the following steps in the Resolution section.

Resolution

You can roll the stack back to a working state with either the CloudFormation console or AWS Command Line Interface (AWS CLI).

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.

CloudFormation console

1.    Open the CloudFormation console.

2.    From the navigation pane, choose Stacks.

3.    From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.

4.    If you don't want to skip resources, choose Stack Actions, and then choose Continue update rollback.

-or-

If you want to skip FAILED resources during rollback, complete the following:

1.    From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.

2.    Choose Stack Actions, and then choose Continue update rollback.

3.    In the Continue update rollback dialog box, expand Advanced troubleshooting.

4.    In the Resources to skip - optional section, select the resources that you want to skip.

5.    Choose Continue update rollback.

AWS CLI

In the AWS CLI, run the continue-update-rollback command. For example:

$ aws cloudformation continue-update-rollback --stack-name awsstackname123 --resources-to-skip awsfaultyresource123

Note: Replace awsstackname123 with the name of your stack. Replace awsfaultyresource123 with the logical IDs of your resources.

Important: CloudFormation sets the status of specified resources to UPDATE_COMPLETE. Then, CloudFormation continues to roll back the stack. After the rollback is complete, the state of the skipped resources is inconsistent with the state of the resources in the stack template. Before performing another stack update, update the stack or resources to be consistent with each other. If you don't do this, subsequent stack updates can fail, and the stack becomes unrecoverable.

If you have nested stacks that must be rolled back to a working state, see Using ResourcesToSkip to recover a nested stacks hierarchy.


AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago
6 Comments

What to do in cases where it doesn't work, loops between 'Only the resources failed during UpdateRollback are allowed to be skipped' and 'The following resource(s) failed to update'.

replied a year ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied a year ago

Hello, when I do this, I get Currently in UPDATE_COMPLETE with reason: null for the resource when I do not skip it and "Nested stack resources can only be skipped when their embedded stack statuses are one of [DELETE_COMPLETE, DELETE_IN_PROGRESS, DELETE_FAILED]" when I select it to be skipped.

What can I do?

Jonas
replied 9 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 9 months ago

Hello, I aslo meet the error: null for the resource when I do not skip it and "Nested stack resources can only be skipped when their embedded stack statuses are one of [DELETE_COMPLETE, DELETE_IN_PROGRESS, DELETE_FAILED]" when I select it to be skipped. If I tried CLI, I had error: An error occurred (ValidationError) when calling the ContinueUpdateRollback operation: 1 validation error detected: Value '[monitoringServer, ec2-sap-ecc-landscape-shared-monitoringServer-6NE49OJUSLVE.ec2-sap-ecc-landscape-shared-monitoringServer-6NE49OJUSLVE]' at 'resourcesToSkip' failed to satisfy constraint: Member must satisfy constraint: [Member must satisfy regular expression pattern: [a-zA-Z0-9]+|[a-zA-Z][-a-zA-Z0-9]*.[a-zA-Z0-9]+] How can I can solve this ?

Kas
replied 2 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 2 months ago