'kafka and parallel consumer: why order is important into a microservice architecture
I started to dive into kafka ecosystem.
I was surprised to find out that by default, each consumer only digests one "event" at a time, sequentially!
It's given by offset acknowledgement, unit of parallelism is at partition-level and some other stuff... you can find nice details here.
If I need to consume received messages in parallel into my application node thread pool, I need to use and make some non-default development effort to get it.
By other hand, several technologies have their own recipes to get it: quarkus/smallrye, confluentinc has a parallel-consummer, spring, ...
I hope to find an by-default code configuration in order to get it.
This suggests me that perhaps, some other technologies are more suitable in order to consume messages straightforwardly...
- Why parallel consumer is not given by default into libraries?
- Why order is important into a microservice architecture?
Solution 1:[1]
Why parallel consumer is not given by default into libraries?
Kafka clients are a largely pluggable ecosystem. The core developers are focused on optimizing the server code, and the built-in client libraries (and serializers) work "well-enough" (TM). So, therefore, a "by default code configuration" for parallel-consumption doesn't exist.
Why order is important into a microservice architecture
That completely depends on your app, but one example is payment-processing or handling any sort of ledger system (after all, Kafka is a sort of distributed ledger). You cannot withdraw money without first depositing a balance. This is not unique to microservices.
Solution 2:[2]
KafkaConsumer is a relatively low-level object, that's basically capable of reading records from given offset position, seeking to a particular offset, reading and saving that offset in existing Kafka store (called __consumer_offsets). Similarly, the receive API is fully synchronous with its poll(Duration) signature.
If more custom, e.g. asynchronous behaviour is desired, then you can use the wrappers like parallel-consumer or spring-kafka.
When it comes to library design, very often it is preferable to do only one thing (basically an applied single responsibility principle).
As an example, consider that if the "main" library were to be asynchrous, the library providers would need to provide thread creation and maintaining semantics, what happens when there are no records (compare to spring-kafka's listeners), and so on. By exposing low-level API these concerns that are not immediately relevant to Kafka these concerns can be avoided.
Solution 3:[3]
If I need to consume received messages in parallel into my application node thread pool, I need to use and make some non-default development effort to get it.
Kafka is built mostly on the concept to allow increase in performance by scale out, not by scale up.
Scale out means you increase performance by adding more components in the architecture of your system where components are complete applications which behave like black boxes. You just add more instances of your application or other applications that you depend upon (ex kafka brokers).
Following this pattern kafka has the consumer-group which allows you to have multiple applications spawn each one having a separate kafka-consumer but all belonging in the same consumer-group.
Only 1 kafka-consumer of the consumer-group will consume an element that needs to be consumed from the source. So if you have 10 kafka-consumer in one consumer-group they will consume in parallel elements from the source. This is how kafka enhances performance by allowing you to scale-out.
It seems that what you need is more of scale up since you want to spawn multiple threads to be able to consume more elements in parallel. When you do this you will face hardware requirements where you will be able to reach a specific performance by by adding more resources but then the costs will increase geometrically and also there will be a point where it would be difficult to increase the performance even more.
So to sum up, re-consider your approach to use multiple threads which means scale up and think about multiple instances of applications using apache-kafka so that you can easily scale out.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | OneCricketeer |
| Solution 2 | Adam Kotwasinski |
| Solution 3 | Panagiotis Bougioukos |