Kafka consumer does not print anything

Question

I am following this tutorial: https://towardsdatascience.com/kafka-docker-python-408baf0e1088 in order to run a producer-consumer example using Kafka, Docker and Python. My problem is that my terminal prints the iterations of the producer, while it does not print the iterations of consumer. I am running this example in local, so:

1. in one terminal tab I have done: docker-compose -f docker-compose-expose.yml up where my docker-compose-expose.yml is this:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    ports:
     - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
     - "9092:9092"
    expose:
     - "9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "topic_test:1:1"
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock

I see the logs from kafka and zookeeper, so I assume that it is working. 2. In another tab I run python producer.py where my producer.py is this:

from time import sleep
from json import dumps
from kafka import KafkaProducer
producer = KafkaProducer(
    bootstrap_servers=['localhost:9092'],
    value_serializer=lambda x: dumps(x).encode('utf-8')
)
for j in range(9999):
    print("Iteration", j)
    data = {'counter': j}
    producer.send('topic_test', value=data)
    sleep(0.5)

It prints the iteration, so I think it is working. 3. Finally, in another tab I run python consumer.py where my consumer.py is:

from kafka import KafkaConsumer
from json import loads
from time import sleep
consumer = KafkaConsumer(
    'topic_test',
    bootstrap_servers=['localhost:9092'],
    auto_offset_reset='earliest',
    enable_auto_commit=True,
    group_id='my-group-id',
    value_deserializer=lambda x: loads(x.decode('utf-8'))
)
for event in consumer:
    event_data = event.value
    # Do whatever you want
    print(event_data)
    sleep(2)

So, here is the problem. According to the tutorial, it should print something like this:

{'counter': 0}
{'counter': 1}
{'counter': 2}
{'counter': 3}
{'counter': 4}
{'counter': 5}
{'counter': 6}
[...]

However, my terminal does not print anything. Moreover, If I solve this problem I want to run the producer on one machine and the consumer on another machine. I need to change only KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092 in the docker-compose-expose.yml writing the IP address of the producer in place of localhost, right?

UPDATE: I tried on another machine, and these steps are now working. Maybe during my experiments I changed something about kafka or docker, so for this reason it is not working. however I do not know why

Answer 1

Basically I had understood that the problem was in some images/processes that were in execution. With docker-compose stop and docker-compose rm -f I solved.