Kafka consumer speed increases with session timeout/max poll records

Question

I'm currently trying to read from one Kafka topic, doing some transformation and producing the messages to another topic. However, i am having a lot of issues with the Consumer. First of all, if we set reasonable session timeout/max poll records values (like 10 s), the consumer takes super long, constantly rebalances and sometimes sends duplicated messages. If we increase the params to crazy values like 30 min, the speed increases dramatically. But the problem is once it reaches the 30 min mark, it rebalances and takes around 30 min to start up again.

I have been playing with a lot of different params but still lost on how to fix this. Any ideas? Thanks

Answer 1

This may be due to some configuration issues. Based on your question I would suggest to please check your auto commit property. Because ideally kafka does rebalance if it does not recieve the acknowledgement of the message read before session timeout happens. If it is set as false then either set it to true or make sure to commit to kafka once you are done processing the message