Why do I see duplicate messages in Amazon SQS for the same Amazon S3 event?

2 minute read
0

I see duplicate messages in Amazon Simple Queue Service (Amazon SQS) for the same Amazon Simple Storage Service (Amazon S3) event.

Resolution

Amazon S3 uses built-in backoff and retry mechanisms to deliver notifications. In rare occasions, the retry mechanism might cause duplicate notifications for the same object event.

Amazon S3 delivers event notifications as JSON objects that contain a sequencer key. The sequencer key is a hexadecimal value that allows you to determine the sequence of events for an object key. Use the sequencer key to identify the event sequence of PUTs and DELETEs for the same object. Duplicate event notifications for a specific object event have the same value for the sequencer key.

To determine the order that events occurred, compare the sequencer strings from two event notifications on the same object key. The event notification with the greater sequencer hexadecimal value is the event that occurred later.

Note:

  • You can't use the sequencer key to determine the order of events between different object keys.
  • The sequencers can be different lengths. To compare these values, right-pad the shorter value with zeros, and then do a lexicographical comparison.

For applications that identify duplicate notifications, maintain a secondary database or index of S3 objects that use event notifications. Then, store and compare the sequencer key values to check for duplicates when each event notification processes.

AWS OFFICIAL
AWS OFFICIALUpdated 4 months ago
2 Comments

Can two messages have same sequencer key but still have different values for other attributes (JSON key-value in message)? For example, different eventTime?

replied 5 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 4 months ago