AWS MSK Can not Connect from Azure

Question

I have set up in Aws inside 1 VPC and configured AWS MSK(Kafka) across the AWS VPC I am able to connect the MSK using bootstrap URL. But When I try to connect from Azure VM I am not able to connect.

I have created a site-to-site VPN connection between AWS to AZURE and in azure setup transit gateway also.

Other AWS services like AWS RDS, EC2 machines etc I am able to connect (using site-to-site) from Azure VM but only AWS MSK I am not.

Public access is disabled in AWS MSK and RDS (still can connect RDS).

I have all the AWS services inside 1 VPC only and inside 2 security groups. I have already whitelisted IP range of Azure in site-to-site VPN of AWS.

How can I connect AWS MSK from Azure VM?

Thanks

Answer 1

Investigate the problem

Try using nslookup your.kafka.host from inside your AWS VPC and your Azure VNET. (Copy kafka host from your connection string to run nslookup command)

Then

1. validate if you can resolve the hostname from the Azure VNET, not only from AWS VPC
2. validate if your Azure network reaches the IP addresses that those hostnames are resolved into

At this point, you have a lot of information on what is/are your problem(s).

Solution possibilities:

If #1 failed, but then #2 succeeded directly to the IP addresses

If #1 validation failed, try to come up with a distribution scheme for your DNS entries. It might be tricky to update them if they are dynamic and managed by AWS. You can use public DNS to enable name resolution in your remote network, or for better safety, you can use 2 private dns zones in both of your networks. The underlying IP addresses may also be changed dynamically, so if you are going with a DNS based problem resolution, it's best to use an automated solution which adapts to changes made by AWS.

If #2 failed

• Either try to make MSK IP addresses reachable by configuring your Azure VNET, and VPN gateways on both sides
• Or build a Kafka proxy. You will have the best adaptability to the AWS platform provided features in this case, but it may introduce an overhead in terms of building and maintaining the solution.