How can I troubleshoot AWS DMS endpoint connectivity failures?

8 minute read

I can't connect to my AWS Database Migration Service (AWS DMS) endpoints. Why is my test connection failing, and how can I troubleshoot these connectivity issues?

Short description

There are two types of errors typically seen when testing the connection from the replication instance to the source or target endpoint:

1. If the error occurred because of connectivity issue between replication instance and source or target, then you see errors similar to the following:

"Application-Status: 1020912, Application-Message: Failed to connect Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: HYT00 NativeError: 0 Message: [unixODBC][Microsoft][ODBC Driver 13 for SQL Server]Login timeout expired ODBC general error."
"Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]timeout expired ODBC general error."
"Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider ODBC general error., Application-Detailed-Message: RetCode: SQL_ERROR SqlState: HY000 NativeError: 2005 Message: [unixODBC][MySQL][ODBC 5.3(w) Driver]Unknown MySQL server host 'mysql1.xxxxx.us-east-1.rds.amazonaws.com' (22) ODBC general error."

2. If the error occurred because of a native database error, such as database permission or authentication error, then you see errors similar to the following:

"Application-Status: 1020912, Application-Message: Cannot connect to ODBC provider Network error has occurred, Application-Detailed-Message: RetCode: SQL_ERROR SqlState: 08001 NativeError: 101 Message: [unixODBC]FATAL: password authentication failed for user "dmsuser" ODBC general error."

Based on the type of error you receive and your network configuration, see the appropriate resolution section.

Note: It's a best practice to test the connectivity from the AWS DMS replication instance to the endpoints after creating your AWS DMS source and target endpoints. Do this before starting the AWS DMS migration task. Otherwise, the task can fail because of connectivity issues with the endpoint.

Resolution

Resolve connectivity issues for AWS hosted resources

Verify that connectivity can be established between the source or target database and the replication instance. Depending on your use case and network infrastructure, connect your source or target database to a replication instance in public subnet or private subnet. For more information, see Setting up a network for a replication instance.

Note: AWS DMS versions 3.4.7 and later require that you configure AWS DMS to use VPC endpoints or to use public routes to all your source and target endpoints that interact with certain Amazon Web Services. If your DMS endpoint tests are successful on earlier versions but failing on later, see Configuring VPC endpoints as AWS DMS source and target endpoints.

Check your replication instance configuration

In your replication instance, confirm that your configuration includes the following:

An Outbound Rule for the IP address with the port of the source or target database in the security group. By default, the Outbound Rule of a security group allows all traffic. Security groups are stateful, so you don't need to modify the Inbound Rule from the default.
An Outbound Rule for the IP address with the port of the source or target database in the network ACL. By default, the Outbound Rule of network access control list (ACL) allows all traffic
An Inbound Rule for the IP address with the ephemeral ports of the source or target database in the network ACL. By default, the Inbound Rule of a network ACL allows all traffic.

Check your source or target database configuration

In your source or target database, confirm that your configuration includes the following:

An Inbound Rule for the IP address of the replication instance or the CIDR of the subnet group of the replication instance with the port of the source or target database in the security group. Security groups are stateful, so you don't need to modify the Outbound Rule from the default.
Note: To find the IP addresses and CIDRs, see the Determine the IP addresses and CIDR of a subnet group section.
An Inbound Rule for the IP address of the replication instance or the CIDR of the subnet group of the replication instance with the port of the source or target database in the network ACL. Confirm that there is no explicit deny rule for the IP address and port allowed.
An Outbound Rule for the IP address or the CIDR of the subnet group of the replication instance with ephemeral ports in the network ACL. By default, the Outbound Rule of a network ACL allows all traffic.
As a best practice, configure your network to allow the CIDR of the subnet group of the replication instance. The IP address of the replication instance changes during a failover or host replacement event.

Determine the IP addresses and CIDR of a subnet group

To determine the IP addresses and CIDR of the subnet group to set up inbound and outbound rules, use the AWS DMS console or the CLI.

Using the AWS console:

Access the AWS DMS console.
From the navigation pane, choose Replication instances.
Choose the name of your replication instance.
Under Details, note the Public IP address, Private IP address, and the Replication subnet group of your replication instance.
Under Replication subnet group, choose the link to access the subnet group page. Note the name of each subnet in the subnet group.
To verify the CIDR of each subnet access the Amazon Virtual Private Cloud (Amazon VPC) console.
In the Subnets tab, search for the subnets noted in step 5. For each subnet, note the CIDR.

Using the AWS CLI:

Run the describe-subnets command to determine the CIDR of each subnet. For replication-instance-name, enter the name of your replication instance.

aws ec2 describe-subnets --filters Name=subnet-id,Values="$(aws dms describe-replication-instances --filters "Name=replication-instance-id,Values=replication-instance-name" --query "ReplicationInstances[*].ReplicationSubnetGroup.Subnets[*].SubnetIdentifier" --output text | sed -e 's/\t/,/g')" --query "Subnets[*].{SubnetId:SubnetId,CidrBlock:CidrBlock}" --output table

Run the describe-replication-instances command to determine the IP addresses of the replication instance. For replication-instance-name, enter the name of your replication instance.

aws dms describe-replication-instances --filters "Name=replication-instance-id,Values=replication-instance-name" --query "ReplicationInstances[*].{ReplicationInstancePublicIpAddresses:ReplicationInstancePublicIpAddresses,ReplicationInstancePrivateIpAddresses:ReplicationInstancePrivateIpAddresses}" --output table

Resolve connectivity issues (on-premises resources)

If your source or target database is hosted on-premises, then confirm the following:

Check with your network administrator to confirm that the database allows incoming connections from the AWS DMS replication instance.
Confirm that a firewall isn't blocking communication to the source or target database.
Confirm that the DNS configuration is set up correctly. If you require DNS resolution, then use the Amazon Route 53 Resolver. For information on using an on-premises name server to resolve endpoints using the Amazon Route 53 Resolver, see Using your own on-premises name server.
-or-
Create a new DMS instance through the AWS CLI to use a customer DNS name server (--dns-name-servers) to resolve DNS issues. By default, DMS instances use Amazon-provided DNS for resolutions. DMS endpoints can fail if the source or target is configured to use custom DNS. For more information, see create-replication-instance.

Be sure that your Amazon EC2 instance has the same network configurations as the AWS DMS replication instance with the connectivity issues. Run the following commands on the new EC2 instance to troubleshoot network connectivity:

telnet <database_IP_address_or_DNS> <port_number>
nslookup <domain_name>

For database_IP_address_or_DNS, use the IP address or domain name of the database specified for the DMS source or target endpoint. For port_number, use the port number of the database specified for the DMS source or target endpoint. For domain_name, use the domain name of the database specified for the DMS source or target endpoint.

Resolve native database errors

To resolve native database errors, confirm that the following endpoint configurations are set correctly:

Username
Password
Set the ServerName to the DNS or IP of the on-premises database, or Amazon Relational Database Service (Amazon RDS) endpoint
Port
Database name
Note: Don't specify a Database name for MySQL source or target.

Note: If any of these fields are specified with AWS Secrets Manager, then see Using secrets to access AWS Database Migration Service endpoints.

For native database errors related to the source or target database, refer to the resolution from the specific database documentation. Use the error code and the error message on the DMS console.

For more information, check error, trace, alert, or other logs from the source or target database.

For database access errors, confirm the permissions required by DMS for the specific source or target.

For more information on encrypting connections for source and target endpoints using Secure Sockets Layer (SSL), see Using SSL with AWS Database Migration Service.