Need to Consume Multiple Kafka Topics

Question

We have created a .NET Worker.cs to consume kafka topics. What should be the best way to consume multiple topics?

Our requirement is there are multiple topics, some are used daily and some rarely. Should we subscribe rarely used topics to one consumer and daily used topics individually to each consumer(as they are having heavy data load.)

Image to see the requirement architecture which we created

In this image, TopicName(D) shows daily used topics and TopicName(R) shows rarely used topics.

Should this be a best possible way to consume our kafka topics or some other way?

Answer 1

The only rule is that you have to check that what Kafka does and doesn't not guarantee while consuming multiple topics:

• Kafka only guarantees message order for a single topic/partition, this also means you can get messages out of order if your single topic Consumer switches partitions for some reason.

• When you subscribe to multiple topics with a single Consumer, that Consumer is assigned a topic/partition pair for each requested topic.

• That means the order of incoming messages for any one topic will be correct, but you cannot guarantee that ordering between topics will be chronological.

• You also can't guarantee that you will get messages from any particular subscribed topic in any given period of time.

If processing all messages is important, you'll need to be certain that each Consumer can process messages from all of its subscribed topics faster than the messages are created.

If it can't, you'll either need more Consumers committing reads in the same group, or you'll have to be OK with the fact that some messages may never be processed.

Obviously one Consumer/topic is the simplest, but it does add some overhead to have the additional Consumers. You'll have to determine whether that's important based on your needs.

The only way to correctly answer your question is to evaluate your application's specific requirements and capabilities, and build something that works within those and within Kafka's limitations.