'Kafka Server issue in Docker

I am using docker for my sample Spark + Kafka project in windows machine. I am facing

WARN ClientUtils: Couldn't resolve server kafka:9092 from bootstrap.servers as DNS resolution failed for kafka
[error] (run-main-0) org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
[error] org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
------
Caused by: org.apache.kafka.common.config.ConfigException: No resolvable bootstrap urls given in bootstrap.servers

Below is my docker-compose.yml

version: '2'
services:
    test1:
        build: test1service/.
        depends_on:
            - kafka
    test2:
        build: test2/.
        depends_on:
            - kafka
            - test1
    zookeeper:
        image: confluentinc/cp-zookeeper:5.1.0
        ports:
         - 2181:2181
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        extra_hosts:
          - "localhost: 127.0.0.1"
    kafka:
        image: confluentinc/cp-kafka:5.1.0
        ports:
         - 9092:9092
        depends_on:
          - zookeeper
        environment:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
          KAFKA_DELETE_TOPIC_ENABLE: "true"
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        extra_hosts:
          - "localhost: 127.0.0.1"

Below is my sample code in test2 service

val inputStreamDF = spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9092")
                           .option("subscribe", "test1")
                           .option("startingOffsets", "earliest")
                           .load()

docker ps command output is

CONTAINER ID   IMAGE                             COMMAND                  CREATED             STATUS          PORTS                                        NAMES
4c8feb49e12b   test1                             "/usr/bin/start.sh"      About an hour ago   Up 50 minutes                                                test1
4535ce246541   test2                             "/usr/bin/myservice-…"   About an hour ago   Up 50 minutes                                                test2
733766f72adb   confluentinc/cp-kafka:5.1.0       "/etc/confluent/dock…"   About an hour ago   Up 51 minutes   0.0.0.0:9092->9092/tcp       kafka_1
d915e25cb226   confluentinc/cp-zookeeper:5.1.0   "/etc/confluent/dock…"   About an hour ago   Up 51 minutes   2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   zookeeper_1

Has anyone faced similar issue, how did you resolve?

Solution 1:^[1]

You try to reach kafka:9092, but docker compose has generated the container kafka_1, which is why there is no name resolution.

Docker gives internal ip to your containers, and use their container name to create an internal DNS on this network (with the embbed dns server)

Your environment variables are not changed by Docker Compose to fit the name it gives to your container.

You should use "container_name: kafka" in your container description to get a static container name.

Solution 2:^[2]

Only thing you need is network. After that all services will be connected and you can use kafka:9092.

version: '2'
services:
    test1:
        build: test1service/.
        depends_on:
            - kafka
        networks:
          - app-network
    test2:
        build: test2/.
        depends_on:
            - kafka
            - test1
        networks:
          - app-network
    zookeeper:
        image: confluentinc/cp-zookeeper:5.1.0
        ports:
         - 2181:2181
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
    kafka:
        image: confluentinc/cp-kafka:5.1.0
        ports:
         - 9092:9092
        depends_on:
          - zookeeper
        environment:
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
          #KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka:9092
          KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
          KAFKA_DELETE_TOPIC_ENABLE: "true"
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
        extra_hosts:
          - "localhost: 127.0.0.1"
        networks:
          - app-network
networks:
  app-network:
    driver: bridge

Solution 3:^[3]

Try with localhost and then the container port:

localhost:9092

Here is an example command for creating a topic named "orders"

docker exec kafka kafka-topics --bootstrap-server localhost:9092 --create --topic orders --partitions 1 --replication-factor 1

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1
Solution 2	Mustafa Güler
Solution 3	Gian Eric

'Kafka Server issue in Docker

Solution 1:[1]

Solution 2:[2]

Solution 3:[3]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]

Solution 3:^[3]