How do I troubleshoot issues related to AWS Secrets Manager secrets in Amazon ECS?

5 minute read
1

When I try to place a task that uses AWS Secrets Manager secrets in Amazon Elastic Container Services (Amazon ECS), I get an error.

Short description

When you fail to place a task with a Secrets Manager secret in Amazon ECS, you might receive one of the following errors:

"ResourceInitializationError error on AWS Fargate AccessDenied error on Amazon Elastic Compute Cloud (Amazon EC2)"

"The task is unable to retrieve secrets from Secrets Manager and fails."

You get these error messages due to one of the following reasons:

  • The Amazon ECS task execution role doesn't have the required permissions to access Secrets Manager secrets.
  • Secrets Manager can't find the specified secret.
  • The name of the secret ends with a hyphen followed by six characters. This returns unexpected results when you search for a secret using partial ARN.
  • There are issues with your Amazon Virtual Private Cloud (Amazon VPC) networking configuration.

To troubleshoot the errors for Amazon ECS tasks that fail to start, use the AWSSupport-TroubleshootECSTaskFailedToStart runbook. Then, refer to the relevant troubleshooting steps for your issue.

Resolution

Important:

  • Use the AWSSupport-TroubleshootECSTaskFailedToStart runbook in the same AWS Region where your ECS cluster resources are located.
  • When using the runbook, you must use the most recently failed Task ID. If the failed task is part of an Amazon ECS service, then use the most recently failed task in the service. The failed task must be visible in ECS:DescribeTasks during the automation execution. By default, stopped ECS tasks are visible for 1 hour after entering the Stopped state. Using the most recently failed task ID prevents the task state cleanup from interrupting the analysis during the automation.

For instructions on how to initiate the runbook, see AWSSupport-TroubleshootECSTaskFailedToStart. Based on the output of the automation, use one of the following manual troubleshooting steps.

Check that the ECS task execution role has the required permissions

  1. Open the IAM console.
  2. In the navigation pane, choose Roles.
  3. Search the list of roles for the task execution role that your ECS tasks are using.
  4. Confirm that the task execution role has the required permissions to access the necessary Secrets Manager resource.

Check that the Secrets Manager secret exists

  1. Open the Secrets Manager console.
  2. On the Secrets list page, choose the secret that your ECS tasks are using.
  3. Confirm that your ECS task definition includes these specific secret names. If you specified additional parameters in the secret, confirm that your task definition includes json-key/version-stage/version-id that's specified in the secret.

Check the name and ARN of the Secrets Manager secret

Check if your secret name ends with a hyphen followed by six characters (example: myappsecret-xxxxxx). Secrets Manager automatically adds a hyphen and six random characters after the secret name at the end of the ARN. Therefore, it's a best practice not to end your secret name with a hyphen followed by six characters. Doing so might return unexpected results when you search for a secret using partial ARN. To resolve this issue, be sure that you use the full ARNs, including the additional hyphen and six characters, of the secrets from Secrets Manager.

Verify that your VPC networking configuration allows your Amazon ECS infrastructure to reach Secrets Manager

Create the interface VPC endpoints for Secrets Manager when the following conditions are true:

  • You're using a task definition that references the Secrets Manager secrets to retrieve sensitive data for your containers.
  • You're using interface VPC endpoints.

Also, be sure that the security groups for your VPC endpoints allow the ECS infrastructure to use these endpoints.

To check if the VPC endpoint for Secrets Manager exists, do the following:

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose Endpoints.
  3. Be sure that the VPC endpoint com.amazonaws.example-region.secretsmanager for Secrets Manager is on the list of endpoints.
  4. Select the endpoint from the list of endpoints, and then choose the Subnets tab. Be sure that the subnets for this endpoint include the ones used by your ECS task.
    Note: If no subnets are listed, then choose Manage Subnets. Select the subnet based on its Availability Zone, and then choose Modify Subnets.

To confirm that the security group attached to the endpoint com.amazonaws.example-region.secretsmanager allows incoming connections on port 443 from the Amazon ECS tasks, do the following:

  1. Select the endpoint from the list of endpoints.
  2. Choose the Security Groups tab.
  3. Choose the Group ID for the security group that you want to check.
  4. Choose the Inbound rules tab.
  5. Verify that the list of inbound rules includes a rule that allows 443 connections from your ECS tasks.

Related information

How can I pass secrets or sensitive information securely to containers in an Amazon ECS task?

Troubleshooting AWS Secrets Manager