Why is there a discrepancy in storage and size metrics between the Amazon S3 console, CloudWatch, and S3 Storage Lens?

6 minute read
1

I’m comparing “Calculate total size” in the Amazon Simple Storage Service (Amazon S3) console, daily bucket storage in Amazon CloudWatch, and "Total storage" in Amazon S3 Storage Lens. I see a discrepancy in these metrics.

Short description

When you see this discrepancy, verify the following conditions:

  • You turned on object versioning.
    Note: The object versioning feature in Amazon S3 retains multiple versions of an object in your bucket. By default, Amazon S3 object versioning is turned off on buckets, and you must explicitly turn this feature on.
  • Incomplete multipart uploads exist.
    Note: Incomplete multipart uploads aren't visible in the Amazon S3 console, but are calculated as part of the storage in CloudWatch and Storage Lens metrics.

To identify the cause of the reporting discrepancy, check if you turned on object versioning. Also, look for any multipart uploads in your bucket. These two factors might result in an increased value of the calculated bucket size in CloudWatch and the Total storage metric S3 Storage Lens. These factors don't apply when you calculate the total size of the object in the Amazon S3 console. For more information, see Amazon S3 daily storage metrics for buckets in CloudWatch.

Tip: If you have incomplete multipart uploads in Amazon S3, then consider creating a lifecycle configuration rule. This lifecycle configuration rule automatically cleans up any incomplete parts, lowering the cost of data storage. Note that lifecycle rules operate asynchronously, so there might be a delay with the operation. However, as soon as the objects are marked for deletion, you are no longer billed for storage (even if the object isn't removed yet).

Note that the Amazon S3 monitoring metrics are recorded once per day. Therefore, these metrics might not display the most updated information. However, CloudWatch monitors your AWS resources and applications in real time. Also, S3 console and Storage Lens use base 2 conversion (/1024) to report storage metrics, and CloudWatch by default uses base 10 conversion (/1000).

Resolution

Daily storage metrics in CloudWatch

In CloudWatch, the BucketSizeBytes metric captures all Amazon S3 and Amazon S3 Glacier storage types, object versions, and any incomplete multipart uploads. This value is calculated by summing up all object sizes, metadata in your bucket (both current and noncurrent objects), and any incomplete multipart upload sizes. For example, the BucketSizeBytes metric calculates the amount of data (in bytes) that's stored in an Amazon S3 bucket in all the following object storage classes:

  • S3 Standard
  • S3 Intelligent-Tiering
  • S3 Standard-IA
  • S3 One Zone-IA
  • S3 Reduced Redundancy Storage
  • S3 Glacier Deep Archive
  • S3 Glacier Flexible Retrieval
  • S3 Glacier Instant Retrieval

The NumberOfObjects metric in CloudWatch counts the following values in your bucket:

  • The total number of objects among all storage classes. For buckets with versioning, this includes both current and noncurrent object versions as well as delete markers.
  • The total number of parts for any incomplete multipart uploads

For example, if you have two versions of the same object, then the two versions count as two separate objects.

"Total storage" metric in S3 Storage Lens dashboard

You can see the Total storage metric in the S3 Storage Lens dashboard under the Bucket tab. This metric captures the total storage, including incomplete multipart uploads, object metadata, and delete markers. To see the exact composition of space that's occupied by noncurrent versions, incomplete multipart uploads, or delete markers, check individual metrics:

  • Current version bytes
  • Noncurrent version bytes
  • Incomplete multipart upload bytes

These metrics are available under AWS Free Tier.

Also, the Object count metric includes information on current and noncurrent versions, delete markers, and incomplete multipart upload object counts. For granular level information, check metrics:

  • Current version object count
  • Noncurrent version object count
  • Delete marker object count
  • Incomplete multipart upload object count.

For more information, see Amazon S3 Storage Lens metrics glossary.

"Calculate total size" in Amazon S3 console

To calculate the size of your bucket from the Amazon S3 console, you can use the Calculate total size action. Amazon S3 then calculates your bucket's storage size. However, note that multipart uploads, and previous or noncurrent versions aren't calculated in the total bucket size. Amazon S3 calculates only the total number of objects for the current or newest version of each object that is stored in the bucket. For example, if there are two versions of an object in your bucket, then Amazon S3's storage calculator counts them as only one object. As a result, the number that the Amazon S3 console calculates is smaller than the one that CloudWatch reports.

Incomplete multipart uploads

To check the amount of size that's occupied by incomplete multipart uploads, check the Incomplete multipart upload bytes metric in the S3 Storage Lens dashboard under the Bucket tab.

To review the list of incomplete multipart uploads, run the list-multipart-uploads command:

aws s3api list-multipart-uploads --bucket BUCKET_EXAMPLE

Note: Replace BUCKET_EXAMPLE with your bucket name.

Then, list all the objects in the multipart upload, using the list-parts command and your UploadId value:

aws s3api list-parts --bucket BUCKET_EXAMPLE --key large_test_file --upload-id EXAMPLE_VALUE

Note: Replace BUCKET_EXAMPLE with your bucket name and EXAMPLE_VALUE with your UploadId value.

Creating a lifecycle rule

To automatically delete multipart uploads, create a lifecycle configuration rule:

  1. Open the Amazon S3 console.
  2. Choose the Management tab.
  3. Choose Create new policy.
  4. Add the name of the policy.
  5. Choose Select - Delete expired delete markers or incomplete multipart uploads.
  6. (Optional) If your bucket isn't versioned, then choose Delete incomplete multipart uploads.

Object versioning

To review and audit your Amazon S3 bucket for different versions of objects, use the Amazon S3 inventory list. An Amazon S3 inventory list file contains a list of the objects in the source bucket and metadata for each object. The inventory list file capture metadata information such as bucket name, object size, storage class, and version ID. To calculate the size that current and noncurrent versions occupy, check the S3 Storage Lens dashboard metrics Current-version bytes and Non-current version bytes, respectively.

Related information

Example 8: Lifecycle configuration to abandon multipart uploads

Expiring objects

Amazon S3 daily storage metrics for buckets in CloudWatch

AWS OFFICIAL
AWS OFFICIALUpdated 10 months ago