Why does the new Amazon EBS volume that I create from an encrypted snapshot not exist?

Short description

When you create an Amazon EBS volume from a snapshot, two operations run:

1. Amazon EBS initiates the volume creation. This returns a volume ID, and sets the volume state to creating, as returned in the API or AWS CLI output. This means that the createVolume API is valid and registered successfully.
2. An asynchronous call initiates to validate the AWS Key Management Service (AWS KMS) key that's used to encrypt and decrypt the volume that you create.

If the AWS KMS validation succeeds, then the volume state is set to available, and the EBS volume becomes accessible. If the specified AWS KMS key ID, Alias, or ARN aren't valid, then the action appears complete. However, the volume creation eventually fails and doesn't return any errors.

Note: The AWS KMS validation for createVolume is asynchronous. For more information, see create-volume and Encrypt Amazon EBS resources.

You might notice the problem when you attach or access the EBS volume. Though the createVolume API returns a volume ID, the EBS volume doesn't exist and the AWS CloudTrail logs doesn't show any errors.

Example of issue

This example shows the createVolume API in use with a not valid alias for the AWS KMS key. The createVolume API succeeds and returns a volume ID, and then sets the volume state to creating. Because the alias for the AWS KMS key isn't valid, the asynchronous authentication fails. This causes the whole operation to fail. When you check the AWS CloudTrail logs for the create volume event, no errors are found because the createVolume operation succeeded.

$ aws ec2 create-volume --volume-type gp2 --availability-zone eu-west-1c --encrypted --kms-key-id hana --snapshot-id snap-0a27fe340500641d9 {
     "AvailabilityZone": "eu-west-1c",
     "MultiAttachEnabled": false,
     "Tags": [],
     "Encrypted": true,
     "VolumeType": "gp2",
     "VolumeId": "vol-043fe27d0ccf74b36",
     "State": "creating",
     "KmsKeyId": "hana",
     "SnapshotId": "snap-0a27fe340500641d9",
     "Iops": 100,
     "CreateTime": "2020-10-06T18:03:09.000Z",
     "Size": 8
}

AWS CloudTrail logs:

responseElements": {        "requestId": "8677d3cd-ad1d-4866-95f5-375d92a35813",
        "volumeId": "vol-043fe27d0ccf74b36",
        "size": "8",
        "snapshotId": "snap-0a27fe340500641d9",
        "zone": "eu-west-1c",z
        "status": "creating",
        "createTime": 1602007389000,
        "volumeType": "gp2",
        "iops": 100,
        "encrypted": true,
        "masterEncryptionKeyId": "hana",
        "tagSet": {},
        "multiAttachEnabled": false
    },
    "requestID": "8677d3cd-ad1d-4866-95f5-375d92a35813",
    "eventID": "bd4216df-ba39-425e-b272-936212ae6699",
    "eventType": "AwsApiCall",
    "recipientAccountId": "864258534754"
}

When you run the describe-volume-status, you find that the volume doesn't exist:

$ aws ec2 describe-volume-status --volume-ids vol-043fe27d0ccf74b36An error occurred (InvalidVolume.NotFound) when calling the DescribeVolumeStatus operation: 
The volume 'vol-043fe27d0ccf74b36' does not exist.

Resolution

Note: If you receive errors when running AWS CLI commands, make sure that you're using the most recent version of the AWS CLI.

Use the describe-volume-status API

To check if the EBS volume exists, use the describe-volume-status API.

Subscribe to the CloudWatch createVolume events

Subscribe to the public CloudWatch createVolume events for more information on volume creation failure. In this example, the notification shows the createVolume CloudWatch event. The notification shows that the createVolume result fails because of a not valid keyId.

AWS Notification MessageCreateVolume <no-reply@sns.amazonaws.com>
{
"version":"0","id":"192e695f-2387-1cf0-fb1c-1cb32f047212",
"detail-type":"EBS Volume Notification","source":"aws.ec2",
"account":"12345678","time":"2020-10-06T18:03:10Z",
"region":"eu-west-1",
"resources":["arn:aws:ec2:eu-west-1:864258534754:volume/vol-043fe27d0ccf74b36"],
"detail":
{"result":"failed","cause":"Invalid keyId hana","event":"createVolume",
"request-id":"8677d3cd-ad1d-4866-95f5-375d92a35813"}
}

Note: If you create an EBS volume from an encrypted snapshot, then the snapshot creation can also fail to create for these reasons:

• The AWS Identity and Access Management (IAM) user or role that creates the volume doesn't have sufficient permissions. The IAM user or role must have permissions to access the AWS KMS key that's used to encrypt the snapshot.
• The AWS KMS key that's used to encrypt the snapshot is turned off, deleted, or isn't in the AWS Region.

Related information

How do I optimize the performance of my Amazon EBS volumes?

Why can't I find the username that created an EBS volume when I search CloudTrail events logs?