How to filter messages on kafka topic based on hostnames under message properties , where as consumers are on 2 different servers?

Question

I have a requirement to consume messages from kafka based on hostname under message properties. Messages are getting dropped under Kafka topic from downstream and on the listener side 2 process are configured to same kafka queue but running on 2 different servers/hostnames. What is the procedure to filter out messages based on hostnames under properties to route messages to the right server. For example message 1 got property with hostname name as serverA and message 2 got property with hostname serverB . Message 1 should be picked up by server A and message 2 should be picked up by serverB.

Answer 1

I think the best approach is to have 2 different topics for server A and server B. That way those can also scale independently. If you want those services to have one interface you can do it as follows with 3 topics:

1. services A + B interface - you will have a router that listens to this topic and route the message based on your business logic to topics A/B.
2. topic A - service A will listen to that one.
3. topic B - service B will listen to this one.

Both service A and B will write the output to your response topic. In that approach topics A and B are internal mechanism of your backend system, your clients will only write to the interface topic.

Another approach would be to implement custom partitioner - I think that's an over kill for your use case and there are a lot of things to take into account such as skewed data.

Answer 2

You could set the hostname as the record key in the producer, and define a static partitioner that routes certain host name patterns to specific partitions.

Then, the consumer would need to use assign rather than subscribe to read specific partitions.

Alternatively, just make different topics unless you plan on having thousands of them