Avoid re-consumption with Apache Kafka MirrorMaker 2 in an active-active configuration?

Question

I am running Apache MirrorMaker 2.7 on multiple active Kafka (2.6) clusters (named prod1, prod2). So topic on prod1 is replicated by MirrorMaker2 as prod1.topic on the prod2 cluster.

I have a Kafka Consumer service running on both prod1 and prod2 using the same kafka consumer group-id.

I have emit.checkpoints.interval.seconds=1 in my MirrorMaker2 config, resulting in offsets being translated every second from topic to prod1.topic.

The problem is, everytime my producer produces to topic on prod1, the data is replicated over to prod1.topic on prod2, and consumed by my consumer on prod2 BEFORE the latest-commit offset from my prod1 consumer is translated over to prod2 by MirrorMaker2.

This does not happen if I start my prod2 consumer a few seconds after data is produced to prod1, as latest-commit offsets would have arrived on prod2 by then, however, both my consumers on prod1/2 need to be running live as I am using prod1/2 as an active-active deployment configuration.

So how could I ensure any data produced is consumed only once, by either of the prod1/ prod2 consumers?

Answer 1

I been trying to achieve a similar setup for an application running in 2 datacenters, regretfully, looks like for the moment MM2 does not guarantee exactly one consumption, as messages and corresponding offsets are not replicated at the same time

You can decrease the offset commit interval with this setting: emit.checkpoints.interval.seconds=1

Here is some reference about this: https://ibm-cloud-architecture.github.io/refarch-eda/technology/kafka-mirrormaker/#replication-considerations

Exactly-once delivery is difficult to achieve in distributed system. In the case of Kafka, producer, brokers, and consumers are working together to ensure only one message is processed end to end. With coding practice and configuration settings, within a unique cluster, Kafka can guarantee exactly once processing. No duplicated records between producer and broker, and committed reads, on consumer side, are not reprocessed in case of consumer restarts.

But for cross cluster replications, the semantic is based on at least once approach. Duplicates can happen when the mirror maker source task stops before committing its offset to the source topic. A restart will load records from the last committed offset which can generate duplicates.

The duplicate handling should be done by the application. You could use different consumer groups ids, or configure your app connected to the destination not to consume from the replicated topics and in case source goes down reconfigure your app to start consuming from those too (or deploy an instance that consumes the remaining messages in those topics).