'why my kafka has message in one partition?
I have two kafka brokers with two partitions on local machine, and use the following tool to write one local file into kafka test2 topic.
# create topic
./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 2 --topic test2
Created topic "test2".
# write 15MB file to kafka, very fast!!
kafka-console-producer.sh --broker-list localhost:9093,localhost:9094 --topic test2 < data.txt
# read data from kafka
./kafka-console-consumer.sh --zookeeper localhost:2181 --topic test2 --from-beginning
Then I find all messages are in one partition, how to debug this?
$ kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9093,localhost:9094 --topic test2 --time -1
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
test2:0:68263
test2:1:0
The status of partition is:
$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic test2
Topic:test2 PartitionCount:2 ReplicationFactor:2 Configs:
Topic: test2 Partition: 0 Leader: 1 Replicas: 1,2 Isr: 1,2
Topic: test2 Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
Solution 1:[1]
I will try the following:
- Use kcat(formerly kafkacat tool, makes producing/consumer significantly easier)
- Try publishing 10s of messages
- Try publishing message with key, have observed key with null were getting published to single partition
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Akhil Jain |
