'ECS instance can't connect to itself behind network load balancer

We've got a service (java-based) and mysql both running as ECS services. The service wants to communicate to a mysql database through a network ELB.

When our service and the mysql database are running on the same EC2 instance, it can't connect to the mysql database. I can reproduce this using telnet - if I telnet to the mysql database through the ELB from another computer I can connect, but if I ssh to the EC2 instance that mysql is running on and then try to telnet to it through the ELB, it just hangs.

What's going on?



Solution 1:[1]

If an instance needs to connect to itself via a network load balancer I think the only way to get it to work is to disable client IP preservation on the load balancer target group.

From https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html

Check whether client IP preservation is enabled on your target group. Load balancers with client IP preservation enabled do not support hairpinning or loopback. If an instance is a client of a load balancer that it's registered with, and it has client IP preservation enabled, the connection succeeds only if the request is routed to a different instance. Otherwise, the source and destination IP addresses are the same and the connection times out.

If an instance must send requests to a load balancer that it's registered with, do one of the following:

Disable client IP preservation.

Ensure that containers that must communicate, are on different container instances.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Mike