How do I troubleshoot authentication errors when I use RDP to connect to an EC2 Windows instance?

7 minute read

I receive authentication errors when I use Remote Desktop Protocol (RDP) to try logging in to my Amazon Elastic Compute Cloud (Amazon EC2) Windows instance.

Resolution

You might receive the following authentication errors when you use RDP to log in to an Amazon EC2 Windows instance:

"An authentication error has occurred. The Local Security Authority cannot be contacted."
"The remote computer that you are trying to connect to requires Network Level Authentication (NLA), but your Windows domain controller cannot be contacted to perform NLA. If you are an administrator on the remote computer, you can disable NLA by using the options on the Remote tab of the System Properties dialog box."

These errors might occur in the following scenarios:

Network Layer Authentication (NLA) is turned on for the server.
The trust relationship between your domain and the EC2 instance joined to this domain fails when RDP logs in.

NLA is turned on for the server

NLA errors occur when an instance loses connectivity to a domain controller because domain credentials aren't authenticated. To fix this issue, use the AWS Systems Manager AWSSupport-TroubleshootRDP automation document to modify instance settings or deactivate NLA on the instance.

The AWSSupport-TroubleshootRDP automation document allows you to modify common settings on an instance that can impact RDP connections.

Use one of the following methods to deactivate NLA on an unreachable instance:

Configure AWS Systems Manager Session Manager.
Run the AWS-RunPowerShellScript command document.
Manually change the registry offline.

Note: You must change the registry when you change the NLA. Before you start, create an Amazon Machine Image (AMI) from your instance. This creates a backup before you change the registry.

Deactivate NLA with Systems Manager Session Manager

To deactivate NLA with Session Manager, add registry keys by completing these steps:

Important: The instance must have the Systems Manager Agent (SSM Agent) installed and the instance must be online. The instance must also have an AWS Identity and Access Management (IAM) role that grants permissions for Session Manager. For more information, see Session Manager prerequisites.

Open the Systems Manager console.
In the navigation pane, choose Fleet Manager.
Choose the managed instance that you want to connect to.
In the Node actions menu, choose Start terminal session, Connect. You're connected to the instance using Session Manager.

Run the following commands in the terminal session:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v SecurityLayer /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v UserAuthentication /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v fAllowSecProtocolNegotiation /t REG_DWORD /d 0 /f

Deactivate NLA with the AWS-RunPowerShellScript command document

To deactivate NLA with the AWS:RunPowerShellScript command document, add registry keys by completing these steps:

Important: The instance must have the SSM Agent installed and it must be online. The instance must also have an IAM role that grants permissions for Session Manager. For more information, see Session Manager prerequisites.

Open the Systems Manager console.
In the navigation pane, choose Run Command, and then choose Run a Command.
For Command document, choose AWS-RunPowerShellScript.

For Command parameters, enter the following commands:

reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v SecurityLayer /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v UserAuthentication /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp" /v fAllowSecProtocolNegotiation /t REG_DWORD /d 0 /f

For Target selection, choose Choose instances manually, and then select your instance.
Choose Run.
Wait until the Overall status changes to Success. Refresh the page after two minutes.
Restart the instance.
Use RDP to log in to the instance.

Manually change the registry offline

Stop the unreachable instance and detach the root volume.
Launch a new instance in the same Availability Zone as the unreachable instance that you stopped. The new instance becomes your rescue instance.

Important: It's a best practice to launch a Windows instance that's different from the unreachable instance to avoid disk signature issues.
Attach the detached volume to the rescue instance as /dev/xvdf.
Use RDP to connect to the rescue instance, and then bring the volume that you just attached online in Disk Manager.
In a command prompt, type regedit.exe, and then press Enter to open the Registry Editor.
Select HKEY_LOCAL_MACHINE and then select File, Load Hive.
Navigate to the Windows folder on the attached volume, and then select the SYSTEM file. The default path is D:\Windows\System32\config.
Name the SYSTEM file. For example, badsys.
The badsys system file now appears under HKEY_LOCAL_MACHINE. Under badsys, navigate to ControlSet001, Control, Terminal Server, WinStations, RDP-Tcp.
Double-click SecurityLayer and set the value data to 0. Select UserAuthentication and set the value data to 0. Then, select AllowSecProtocolNegotiation and set the value data to 0.
Scroll up and select badsys, File, Unload Hive.
After the hive unloads, open Disk Manager and take the disk offline.
Detach the volume from the rescue instance and attach the volume to the unreachable instance as the root volume (/dev/sda1).
Start the instance and test RDP.

The trust relationship between your domain and the EC2 instance joined to this domain fails during RDP log in

Use cached user credentials to try to log in to the unreachable instance.

Prerequisites

A local account that can successfully authenticate to the EC2 instance.
(Optional) At least one domain account that was logged in when the instance was communicating with the domain controller. For domain account to work, the domain account credentials must be cached in the server.

Note: It's a best practice to use a local account.
When the domain controller isn't available, make sure that the setting for the number of previous logins to cache is set to at least 1. This must be done to use interactive logins. The policy can be set to the default value of 10. By default, the policy isn't defined and you can use the server's local policy.

To log in using cached user credentials, complete these steps:

Open the EC2 console. and then select Security groups.
In the navigation pane, choose Security Groups.
Choose Create security group.
Add a security group name and description.
Under Inbound rules, choose Add rule.
For Type, choose RDP. Then, provide information for the source you want to use RDP to connect from.
Under Outbound rules, remove all outbound access.
Choose Create security group.
In the navigation pane, choose Instances, and then select the unreachable instance.
Choose Actions, Security, Change security groups. Remove all the existing security groups and then assign the security group that you just created.
Use the regular domain account to use RDP to connect to the EC2 instance. Because all outbound access is removed from Amazon EC2, RDP uses the cached credentials stored on the server.

Note: Authentication is initially attempted against the domain controller. But because there's no outbound access from Amazon EC2, authentication eventually checks the cached credentials stored on the server. Authentication is reattempted with the cached credentials and the login succeeds. After you log in, you can change the security group settings back to the original state and then continue to fix any issues with your domain.