'Throttling Kafka Messages to downstream service
I am currently trying to throttle my Kafka consumption. My application consumes messages and the hits another application endpoints to persist some data. The persisting application gets hit by many other applications and based on the health of it I would like to slow down messages so that I am not overwhelming it.
I currently am running a cron which reaches out to the persisting application for a health check at a given interval. If the health is bad I pause the consumers and put a timer task for when they should be resumed.
Another way I have tested out is by just pausing and resuming based on the health endpoints response. If it comes back healthy I would resume the listeners that are paused. If it comes back unhealthy I would pause any listeners that are currently active.
My current implementation uses the KafkaListenerEndpointRegistry to grab the ConcurrentMessageListenerContainers that I need. From there I loop through the containers and pause the containers individually. Then based on the two approaches I either set a timertask to resume the specific container or when the health comes back green I do the same strategy mentioned above to pause them but this time I resume the containers.
My question is if anyone has come across a better solution to a similar problem or finds gaps in my approach?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|