How do I access Amazon MSK brokers from custom ports?

Short description

You can dynamically update the advertised.listener configuration in Amazon MSK from the default Kafka broker port 9094/9092 to 443 or any other port of your choice. The dynamic configurations are stored in Apache ZooKeeper. Therefore, these configurations don't get updated when you restart the broker.

For example, suppose that you have an Amazon MSK cluster with two brokers on Apache Kafka version 2.4.1 with TLS encryption and plaintext traffic enabled. To update the broker configuration by modifying the advertised.listener property, be sure that your client machine is in the same virtual private cloud (VPC) as the Amazon MSK cluster. To listen and connect from port 443, you must configure a Network Load Balancer, target group, and an Amazon Route 53 private hosted zone on kafka.amazon.com.

You can create these configurations using either of the following approaches:

• If you need to use single port for all brokers, then you must configure separate Network Load Balancers for each broker. For example, to use a single port 443 for two brokers, then the communication pattern between the client and broker is similar to the following:
  Client - Network Load Balancer 1 for port 443 - target group for broker 1 - broker 1 Client - Network Load Balancer 2 for port 443 - target group for broker 2 - broker 2
• If you need to use one Network Load Balancer for all brokers, then you must use a separate port for each broker. For example, to use one Network Load Balancer for two brokers, then the communication pattern between the client and broker is similar to the following:
  Client - Network Load Balancer listener port 9001 - target group for broker 1 - broker 1 Client - Network Load Balancer listener port 9002 - target group for broker 2 - broker 2

This article discusses the first approach.

Resolution

Update the broker port

Update the port for broker 1 from 9094 to 443 by running a command similar to the following from the client machine:

[ec2-user@ip-aa-a-a-aaa bin]$./kafka-configs.sh --bootstrap-server 
b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9094 --entity-type 
brokers --entity-name 1 --alter --command-config client.properties 
--add-config 
advertised.listeners=[CLIENT_SECURE://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443,REPLICATION://b-1-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9093,REPLICATION_SECURE://b-1-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9095]

Completed updating config for entity: broker 1

Update the port for broker 2 to 443 by running a command similar to the following from the client machine:

[ec2-user@ip-aa-a-a-aaa bin]$./kafka-configs.sh --bootstrap-server 
b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9094 --entity-type 
brokers --entity-name 2 --alter --command-config client.properties 
--add-config 
advertised.listeners=[CLIENT_SECURE://b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443,REPLICATION://b-2-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9093,REPLICATION_SECURE://b-2-internal.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:9095]

Completed updating config for entity: broker 2

Note:

• You must run the preceding command for each broker separately from the same client machine. This configuration update doesn't require a broker restart.
• When you modify the values for REPLICATION and REPLICATION_SECURE, be sure to add -internal to the name of the broker.
• Be sure not to replace ports 9093 and 9095 in the commands.
• Be sure to use the correct values for the client field when you update the broker port configuration. Examples: For TLS - CLIENT_SECURE, use [CLIENT_SECURE://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443]. For SASL - CLIENT_SASL_SCRAM, use [CLIENT_SASL_SCRAM://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443]. For IAM - CLIENT_IAM, use [CLIENT_IAM://b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com:443].

You can check the value for the client field by doing the following:

1.    Enter into the Apache ZooKeeper shell by running the following command:

./zookeeper-shell.sh <ZooKeeper_connection_string>

2.    List the configuration for broker 1 by running the following command:

get /brokers/ids/1
{"listener_security_protocol_map":{"REPLICATION_SECURE":"SSL","CLIENT_SASL_SCRAM":"SASL_SSL","REPLICATION":"PLAINTEXT","CLIENT":"PLAINTEXT","CLIENT_SECURE":"SSL"},"endpoints":["CLIENT://b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9092","CLIENT_SASL_SCRAM://b-1.dns.691zg3.c14.kafka.us-east-1.amazonaws.com:9096","CLIENT_SECURE://b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9094","REPLICATION://b-1-internal.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9093","REPLICATION_SECURE://b-1-internal.testdns.xx.c14.kafka.us-east-1.amazonaws.com:9095"],"rack":"use1-az2","jmx_port":9099,"port":9092,"host":"b-1.testdns.xx.c14.kafka.us-east-1.amazonaws.com","version":4,"timestamp":"1645081828199"}

Create two target groups

Create two target groups (example: TG1 and TG2) in the same VPC as your Amazon MSK cluster with the following options:

• ip for target type
• TCP for Protocol
• 9094 for Port

Then, register the Amazon MSK broker to the respective target group using the IP address of the broker.

Run a command similar to the following to get the IP address of the broker:

[ec2-user@ip-aa-a-a-aaa bin]$ dig +short b-1.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com
10.0.3.xx

[ec2-user@ip-aa-a-a-aaa bin]$ dig +short b-2.testdns.mswah7.c1.kafka.us-east-1.amazonaws.com
10.0.4.xx

Create two Network Load Balancers

Create two Network Load Balancers (example: NLB1 and NLB2) with the following settings:

• For Listeners, modify the port to point to the broker’s advertised port 443.
• For Default action, select the respective target group to forward traffic.
• For Mappings, be sure to select the subnets where the Amazon MSK brokers exist.

Be sure to enable cross-zone load balancing for the Network Load Balancers that you created.

Create a Route 53 private hosted zone

Create a Route 53 private hosted zone with the following options:

• For Domain Name, enter the domain name from your broker string (example: kafka.us-east-1.amazonaws.com).
• For VPC ID, enter the VPC ID of the clients.

Create two A-type alias records

Using the Amazon Route 53 console, create two A-type alias records with record names that are identical to the respective DNS names of the brokers. This is required to prevent any TLS handshake failures.

Be sure that the inbound rules for cluster's security group allow traffic from the IP address of the Network Load Balancers to port 9094.

After performing all the preceding steps, you can connect to the broker advertised port 443.

Note: This solution works for all types of access control methods that are enabled for Amazon MSK clusters.

---

Related information

How Goldman Sachs builds cross-account connectivity to their Amazon MSK clusters with AWS PrivateLink

Custom MSK configurations