'Spring Kafka configuration for high consumer processing time
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>2.8.2</version>
</dependency>
I have a kafka consumer that takes anywhere from 30-120 seconds to process a record.
Configuration
@Bean
public ConcurrentKafkaListenerContainerFactory<String, GGP> kafkaListenerContainerFactory() {
ConcurrentKafkaListenerContainerFactory<String, GGP> factory =
new ConcurrentKafkaListenerContainerFactory<>();
factory.setConsumerFactory(consumerFactory());
factory.setConcurrency(10);
return factory;
}
What are some considerations to make when setting the following config vals for consumers that can take multiple minutes to complete the consumption of a record.
heartbeat.interval.ms # default 3 seconds
session.timeout.ms # default 45 seconds
request.timeout.ms # default 30 seconds
max.poll.interval.ms # default 5 mins
As I currently understand max.poll.interval.ms should be longer than the processing time of a single record but by how much (is there a good rule of thumb)? I understand that the larger this value the longer it will take to recover from an actual failure. However if this value is too low then it will fail prematurely. Do any of the other configurations need to change because of such a long processing time?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
