Why isn't my Lambda function with an Amazon SQS event source optimally scaling?

Resolution

Note: When you configure an Amazon SQS queue as an event source, Lambda functions can optimally scale up to 300 more instances per minute. The maximum number of concurrent invocations is 1,000 for standard queues. If you use FIFO event source mapping, then functions can scale in concurrency to the number of active message groups. For more information see, Scaling and processing.

Identify and resolve any Lambda function invocation errors

To prevent errors at scale, Lambda throttles function scaling when invocation errors occur. When the errors are resolved, Lambda continues to scale the function. For more information, see Backoff strategy for failed invocations. For best practices on how to resolve Lambda function invocation errors, see Troubleshooting issues in Lambda and How do I troubleshoot Lambda function failures?

Configure your Lambda function with optimal concurrency for your use case

Reserved concurrency

If you configured reserved concurrency on your function, then Lambda throttles your function when the function reaches the reserved value. Event source mappings don't account for reserved concurrency and can process more messages off your queue than can be sent to your function.

The amount of concurrency that's reserved for your function must have the following minimum values:

• For standard Amazon SQS queues: 1,000
• For FIFO queues: The number of active message groups

It's not a best practice to have reserved concurrency lower than the number of message groups in the FIFO queue. A lower reserved concurrency can cause processing delays from the FIFO queue and cause your function to be throttled.

Important: To limit the number of concurrent invokes, use the maximum concurrency setting for Amazon SQS event sources instead of reserved concurrency.

Unreserved concurrency

If you don't configure reserved concurrency on your function, then your function has a default unreserved concurrency quota of 1,000. This default quota applies to other functions in the same AWS account and AWS Region. If you have at least 1,000 unreserved concurrency available in your function's Region, then the function scales until it reaches the maximum available concurrency. When all your account's concurrency is in use, Lambda throttles invocations.

Note: Lambda functions initially scale based on the concurrency scaling rate.

To guard against excessive scaling because of sudden spikes in traffic, Lambda has restrictions on how fast your functions can scale. The concurrency scaling rate defines the upper limit on how rapidly functions within your account can increase in response to heightened requests.

Important: The concurrency scaling rate is a function-level limit. Each function in your account can scale independently of other functions. The concurrency scaling rate also differs from the account-level concurrency limit that provides the total amount of concurrency available to your functions.

Maximum concurrency setting

The maximum concurrency setting limits the maximum concurrent executions of your Lambda function to not exceed a configured value. When you set the maximum concurrency for one event source, that value applies only to that specific event source. Any additional event sources without maximum concurrency use the remaining account's concurrency quota or the reserved concurrency.

The maximum concurrency setting and reserved concurrency can be used together. It's a best practice to not set the maximum concurrency setting greater than the function's reserved concurrency, as it can cause throttling.

Confirm that there are enough messages in your Amazon SQS queue to allow your Lambda function to scale

If an Amazon SQS queue is configured to invoke a Lambda function, then Lambda scales invocations only when there are messages in the queue.

To check how many messages in your Amazon SQS queue need to be processed, review your ApproximateNumberOfMessagesVisible metric.

If the metric is low or at 0, then your function can't scale.

If the metric is high and there are no invocation errors, then increase the batch size on your event notification. Increase the batch size until the duration metric increases faster than the batch size metric. For more information, see Monitoring functions on the Lambda console.

Note: The maximum batch size for a standard Amazon SQS queue is 10,000 records. For FIFO queues, the maximum batch size is 10 records. For more information, see ReceiveMessage.

Related information

Using Lambda with Amazon SQS

Managing AWS Lambda function concurrency