How do I prevent "Rate exceeded" errors in CloudFormation?

Short description

The Rate exceeded error occurs when API calls to an AWS service exceed the maximum allowed API requests, and the API calls are throttled. Generally, these errors are temporary and resolve themselves as the API calls lessen.

To prevent Rate exceeded errors, you can:

• Implement exponential backoff
• Create or update stacks one at a time
• Use the DependsOn attribute
• Request a quota increase

Resolution

Implement exponential backoff

When using AWS API endpoints, implement exponential backoff to decrease the number of API calls being made.

The following example pseudo code for a DescribeStacks API call is configured to retry the API call after specific amounts of time:

{
Make ‘DescribeStacks’ API call 

if throttled: wait 2 sec; Make ‘DescribeStacks’ API call 
if throttled: wait 4 sec; Make ‘DescribeStacks’ API call 
if throttled: wait 8 sec; Make ‘DescribeStacks’ API call 
if throttled: wait 16 sec; Make ‘DescribeStacks’ API call 
if throttled: wait 32 sec; Make ‘DescribeStacks’ API call 
}

Create or update stacks one at a time

Creating or updating multiple CloudFormation stacks concurrently can result in many API calls being made at the same time. To prevent the API calls from exceeding the maximum allowed API requests, create or update one stack at a time.

Use the DependsOn attribute

Unless a dependency is defined between resources, CloudFormation creates and updates resources at the same time. The DependsOn attribute defines dependencies between resources to control concurrent updates.

The DependsOn attribute lets you specify when each dependent resource is created or updated. For example, if resource B is dependent on resource A, then you can specify that resource A be created or updated before resource B. This limits the number of API calls being made at at the same time and reduces the occurrence of throttling. You can also use the DependsOn attribute with nested stacks.

Request a quota increase

If the preceding resolutions don't work for your situation, then you can request a quota increase. Before you request a quota increase, identify the API call to determine the one that is exceeding the call rate.

In your request for a quota increase, include your AWS Region, the time frame of the API throttling, and the reason for the increase.