QuickDevGuide

Home › Apache Kafka

Exactly-Once with Transactions — Apache Kafka

For consume → transform → produce pipelines, Kafka transactions make the output records and the consumer offsets commit atomically. How it works The producer has a stable transactional.id and calls initTransactions() (fences zombie instances). beginTransaction(), send output records. sendOffsetsToTransaction(offsets, consumer.groupMetadata()) — offsets become part of the transaction. commitTransaction() — or abortTransaction() on error. Downstream consumers use isolation.level=readcommitted so they never see aborted data. Limits Exactly-once covers Kafka-to-Kafka. Side effects in external systems (emails, REST calls, DB writes) still need idempotency keys or the outbox pattern. Kafka Streams enables all of this with one setting: processing.guarantee=exactlyoncev2.

Key Takeaways