Difference between kafka idempotent and transactional producer setup?

Question

When setting up a kafka producer to use idempotent behaviour, and transactional behaviour:

I understand that for idempotency we set: enable.idempotence=true and that by changing this one flag on our producer, we are guaranteed exactly-once event delivery?

and for transactions, we must go further and set the transaction.id=<some value> but by setting this value, it also sets idempotence to true?

Also, by setting one or both of the above to true, the producer will also set acks=all.

With the above should I be able to add 'exactly once delivery' by simply changing the enable idempotency setting? If i wanted to go further and enable transactional support, On the Consumer side, I would only need to change their setting, isolation.level=read_committed? Does this image reflect how to setup the producer in terms of EOS?

Answer 1

Actually idemnpotency by itself does not always guarantee exactly once event delivery. Let's say you have a consumer that consumes an event, processes it and produces an event. Somewhere in this process the offset that the consumer uses must be incremented and persisted. Without a transactional producer, if it happens before the producer sends a message, the message might not be sent and its at most once delivery. If you do it after the message is sent you might fail in persisting the offset and then the consumer would read the same message again and the producer would send a duplicate, you get an at least once delivery. The all or nothing mechanism of a transactional producer prevents this scenario given that you store your offset on kafka, the new message and the incrementation of the offset of the consumer becomes an atomic action.