Docker Kafka client to Docker Kafka broker connection refused [duplicate]

Question

As the title states I am having issues getting my docker client to connect to the docker broker

Errors:

%3|1647161877.851|FAIL|fc24c271e73f#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)
2022-03-13T08:57:57.851241551Z %3|1647161877.851|ERROR|fc24c271e73f#producer-1| [thrd:app]: fc24c271e73f#producer-1: localhost:9092/bootstrap: Connect to ipv4#127.0.0.1:9092 failed: Connection refused (after 0ms in state CONNECT, 1 identical error(s) suppressed)

Server setup: Community docker-compose.yml from step 1 of guide:

https://docs.confluent.io/platform/current/quickstart/ce-docker-quickstart.html

Client setup:

.NET Confluent.Kafka producer

Procedures so far:
I initially thought it would be an issue with localhost in the docker container and using docker option:

--net=host

also fixed the issue. However this has the side effect of removing published ports and is no good.

I then tried using

--add-host host.docker.internal:host-gateway

which I found in another post (cannot remember which) however this does not work. I initially thought there was an issue with it since the --net=host works but when I tried it with a SignalR sample project it worked straight away.

I have come across multiple posts suggesting changes to

KAFKA_ADVERTISED_LISTENERS

However I find it odd that the host forwarding is not working and changes to this would make a difference. I have nonethelss tried out multiple combinations of it whether it be the broker ip of localhost to 0.0.0.0 etc. (stuff I found across different posts) but with no success.

Trying to get a better understanding of what is going on I came across this post:

https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/

Which explains the 2 step process involved in connecting to a broker. Looking at the errors I received with the /bootstrap it seems to already fail on the initial connection. This however should be an indication that the immediate issue is not with with KAFKA_ADVERTISED_LISTENERS however I may be wrong in that assumption.

The post does have a docker to docker scenario but that is being done using custom network bridge which I do not want to have to use for this.

Anybody has any ideas of what I should do? Perhaps knows the exact changes to make and where.

Answer 1

It appears your code is trying to connect to port 9092.

PLAINTEXT://broker:29092 is the listener you want to connect to, assuming your own code is in a container, as well.

If it isn't, then you can use localhost:9092 without host networking, but you need to have these variables

KAFKA_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://0.0.0.0:9092 
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092 

done using custom network bridge

Compose defaults to create its own network bridge unless you define one separately and link all services to it.

not want to have to use for this.

You'll need a bridge unless you're running the code on a Linux host where --net=host will actually work. Then you should be able to use localhost:9092.

You shouldn't need host networking, though, if you understand how the listeners work

has the side effect of removing published ports

Not exactly. The exposed ports of the container should automatically be exposed on the host