How do I resolve the error "The specified queue does not exist or you do not have access to it." when running my AWS Glue job to send messages to Amazon SQS in a different Region?

Short description

If the Amazon SQS queue is in a different Region than the Glue ETL job, then you must pass the Region information when sending messages to the Amazon SQS queue. Otherwise, your ETL job fails with the following error:

ERROR [main] glue.ProcessLauncher (Logging.scala:logError(70)): Exception in User Class: com.amazonaws.services.sqs.model.QueueDoesNotExistException :The specified queue does not exist or you do not have access to it. (Service: AmazonSQS; Status Code: 400; Error Code: AWS.SimpleQueueService.NonExistentQueue; Request ID: 3861e4c0-9b49-5404-a4c6-bcd3ed43fe20)

Resolution

To create an AWS Glue Spark job for Account A in us-west-2 to send messages to Amazon SQS for Account B in us-east-1, do the following:

1.    Create an Amazon SQS queue in Account B with the following access policy. This access policy provides access to the AWS Identity and Access Management (IAM) role that would be attached to the AWS Glue Spark job in Account A. You can also grant these required permissions to a specific IAM user (for example, testuser) in Account A. For more information, see Basic examples of Amazon SQS policies.

{
    "Version": "2008-10-17",
    "Id": "__default_policy_ID",
    "Statement": [
        {
            "Sid": "__owner_statement",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::111122223333444:role/GlueSparkJobIAMRole",
                    "arn:aws:iam::111122223333444:user/testuser"
                ]
            },
            "Action": "sqs:SendMessage",
            "Resource": "arn:aws:sqs:us-east-1:5555666677778888:test-queue"
        }
    ]
}

Replace the following in the above policy:

• 111122223333444 with the AWS account ID for Account A.
• 5555666677778888 with the AWS account ID for Account B.
• testuser with the name of the IAM user in Account A.
• GlueSparkJobIAMRole with the IAM role attached to the AWS Glue spark job in Account A.
• test-queue with the name of the queue created in Account B.

2.    Create an AWS Glue ETL job in Account A. For more information, see Adding jobs in AWS Glue. On the Configure the job properties page, select A new script to be authored by you. Include the Python script in the job to send a message to the Amazon SQS queue in Account B:

import boto3
sqs = boto3.client('sqs', region_name="us-east-1")
queue_url = 'https://sqs.us-east-1.amazonaws.com/5555666677778888/glue-queue'
response = sqs.send_message(
    QueueUrl=queue_url,
    DelaySeconds=10,
    MessageAttributes={
        'Title': {
            'DataType': 'String',
            'StringValue': 'The Whistler'
        },
        'Author': {
            'DataType': 'String',
            'StringValue': 'John Doe'
        },
        'WeeksOn': {
            'DataType': 'Number',
            'StringValue': '6'
        }
    },
    MessageBody=('Example message'))
print(response['MessageId'])

Replace the following in the above script:

• us-east-1 with the Region where the Amazon SQS queue is present
• 5555666677778888 with the AWS account ID of Account B
• glue-queue with the name of the Amazon SQS queue
• Example message with the message to be sent to the SQS queue

Replace the message attributes and corresponding values in the script with your desired message attributes and values.

3.    Identify the AWS Identify Access Management (AWS IAM) role attached to the AWS Glue Spark job in Account A. Then, grant Amazon SQS required permissions to that role. For simplicity, you can attach the AWS managed policy AmazonSQSFullAccess to this AWS IAM role. For more information, see Setting up IAM permissions for AWS Glue.

4.    Run the AWS Glue ETL job created in Account A.

5.    Verify that the job completed successfully by checking that the job sent the message to the Amazon SQS queue in Account B.

6.    To receive the message in the Amazon SQS queue in Account B, poll for the message in the queue. For more information, see Receiving and deleting messages (console).

7.    Verify that you can view the messages sent from Account A in the queue.

Related information

Managing Amazon SQS queues (console)