'Kafka streams change thread count based on replica count
I have a relatively simple setup where I'm running a dynamic number of instances of my stateless Kafka streams app based on my consumer group lag. This is achieved using an HPA, monitoring my consumer group lag.
I am trying to choose the right number of active threads to run my app with but seems like this will depend on the number of replicas running. The ideal situation is when I have the same number of threads on an instance as the number of partitions assigned to that instance, but this changes with traffic. Seems like there is no built in way to change the number of threads based on the assigned number of partitions so the question is:
- do I set
num.stream.threads
to equal double my CPU count and let some threads be sometimes idle? Not sure how big of a penalty this is. - Or do I keep the threads count to be 1 which will not utilize multithreading? So I might as well use a single CPU for each instance.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|