Spring kafkaListener Consumer GroupID for a Partition

Question

Currently, We create multiple kafka consumers for a given topic and all the messages in the topic are together processed by all the consumers in the same consumer group.

We want to enhance this now. Instead of creating multiple kaka consumers/listeners reading from the same topic, we want multiple consumers in the same consumer group to read from a specified partition of the topic.

Is it possible? Do we have any reference where we can specify both the consumer group id and partition together in each of our kafka listeners?

Answer 1

If I understand your requirements, you have to use manual partition assignment. See https://docs.spring.io/spring-kafka/docs/current/reference/html/#manual-assignment

You can also configure POJO listeners with explicit topics and partitions (and, optionally, their initial offsets). The following example shows how to do so:

@KafkaListener(groupId = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "0", "1" }),
          @TopicPartition(topic = "topic2", partitions = { "0", "1" })
        })

@KafkaListener(groupId = "thing2", topicPartitions =
        { @TopicPartition(topic = "topic1", partitions = { "2", "3" }),
          @TopicPartition(topic = "topic2", partitions = { "2", "3" })
        })

etc.