'Separate kafka servers for retry topics and DLT?
Can we use separate kafka clusters for the main topic and retry topics? The idea is to not keep a single point of failure which can occur if kafka is down, hence we need 2 different kafka servers, so that if one is down then at least we can pause the consumption for the main one.
I am trying to implement spring kafka non blocking retries as mentioned here https://docs.spring.io/spring-kafka/reference/html/#retry-topic.
Solution 1:[1]
You can provide separate KafkaListenerContainerFactory and KafkaTemplate instances configured with different broker settings to be used with with the RetryTopic feature.
These will be used for forwarding the message to the retry topics, and for creating the retry topics' containers.
The main topic container will still be created with the factory specified in the @KafkaListener annotation, or the factory provided by Spring Boot's autoconfiguration.
Refer to this part of the documentation for specifying a different factory for the retry topic containers.
The retry topic KafkaTemplate can be specified both in the @RetryableTopic annotation or the RetryTopicConfigurationBuilder. You can also provide a bean with name RetryTopicInternalBeanNames.DEFAULT_KAFKA_TEMPLATE_BEAN_NAME if you're using the annotation.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 |