'why my kafka has message in one partition?

I have two kafka brokers with two partitions on local machine, and use the following tool to write one local file into kafka test2 topic.

# create topic
  ./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test2
  Created topic "test2".
  # write 15MB file to kafka, very fast!!
  kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --topic test2 < data.txt
  # read data from kafka
  ./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test2 --from-beginning

Then I find all messages are in one partition, how to debug this?

$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9093,localhost:9094 --topic test2 --time -1
    SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
    test2:0:68263
    test2:1:0

The status of partition is:

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2
Topic:test2 PartitionCount:2    ReplicationFactor:2 Configs:
    Topic: test2    Partition: 0    Leader: 1   Replicas: 1,2   Isr: 1,2
    Topic: test2    Partition: 1    Leader: 2   Replicas: 2,1   Isr: 2,1


Solution 1:[1]

I will try the following:

  1. Use kcat(formerly kafkacat tool, makes producing/consumer significantly easier)
  2. Try publishing 10s of messages
  3. Try publishing message with key, have observed key with null were getting published to single partition

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Akhil Jain