The Real Cost of Exactly-Once in Kafka

A billing and transaction processing system was configured to use Apache Kafka. To ensure that payment events were neither duplicated nor lost during network partitions, the development team enabled Exactly-Once Semantics (EOS) by configuring a `transactional.id` on their producers and setting the consumer isolation level to `read_committed`. They expected a minor performance penalty in exchange for strong consistency. Instead, performance testing showed that producer write latency jumped by 2x to 3x, and overall pipeline throughput dropped by nearly 35%. The team was forced to delay their production release to re engineer their batching and commit patterns.

Exactly-Once Semantics in Kafka is an engineering achievement, but it is not a free performance upgrade. Achieving exactly-once guarantees across distributed systems requires coordination. To evaluate whether EOS is appropriate for your workloads, you must understand the mechanical costs under the hood: the two-phase commit protocol, epoch based producer fencing, and the impact of read isolation levels on consumer latency.

The Foundation: Idempotent Producers

Before examining transactions, we must look at the foundation of Kafka's consistency model: the idempotent producer. In a standard non-idempotent setup, if a producer sends a batch of messages to a broker and the network connection drops before the broker returns an acknowledgment, the producer is forced to retry the write. If the broker successfully wrote the messages before the connection failed, the retry will write the duplicate messages again, resulting in duplicate data in the log.

Enabling idempotence (via the producer configuration enable.idempotence=true) prevents this duplication. When the producer initializes, the broker assigns it a unique Producer ID (PID). For each message batch sent by the producer, it attaches the PID and a monotonically increasing sequence number. The broker tracks the highest sequence number successfully written for each PID on a per partition basis:

If a network retry occurs and the broker receives a sequence number that has already been written, the broker discards the duplicate write and returns a success acknowledgment to the producer. This guarantees idempotence for a single producer session writing to a single partition, with minimal performance overhead. The only cost is a small metadata header attached to each message batch.

The Mechanism: Two-Phase Commit and the Transaction Coordinator

To guarantee exactly-once delivery across multiple partitions and topics, Kafka builds on top of the idempotent producer by introducing a Transaction Coordinator and a transaction log topic named __transaction_state. When a producer executes writes within a transaction, it coordinates with the broker using a two phase commit protocol:

This process consists of seven distinct stages:

1. Producer Initialization: The producer registers its configured transactional.id with the Transaction Coordinator using the InitTransactions() API. The coordinator maps this ID to a PID, increments the producer's epoch counter, and writes this mapping to the __transaction_state topic.
2. Beginning the Transaction: The producer calls beginTransaction(). This is a local operation that initializes the internal transaction state buffer.
3. Registering Partitions: Before sending messages to a topic partition, the producer sends an AddPartitionsToTxnRequest to the coordinator. The coordinator writes a log entry marking those partitions as part of the active transaction. This write must be replicated and committed to disk before the producer is permitted to send the actual data.
4. Writing Data: The producer sends the message batches to the target brokers using standard produce requests. These messages contain the PID and epoch headers, and the broker writes them to the partition logs. However, these messages are not yet visible to consumers configured with `read_committed` isolation.
5. Initiating Commit: The producer calls commitTransaction(). It sends an EndTxnRequest to the coordinator. The coordinator writes a `PrepareCommit` state transition to the __transaction_state log.
6. Writing Transaction Markers: The coordinator writes a special control batch called a **transaction marker** (either COMMIT or ABORT) to every topic partition involved in the transaction. This marker tells downstream consumers whether to emit or discard the preceding message batches.
7. Completing the Transaction: Once all transaction markers are acknowledged by the partition leaders, the coordinator writes a `CompleteCommit` entry to the transaction log, completing the lifecycle.

This multi step protocol is what guarantees atomic writes across partitions. If a producer crashes mid transaction, the coordinator will eventually time out the session and write ABORT markers to the target partitions, ensuring that partial writes are discarded by downstream consumers.

Zombie Fencing and Epoch Management

A major design challenge in distributed systems is the zombie producer problem. This occurs when a producer process freezes (due to a garbage collection pause or network timeout), causing the cluster coordinator to assume the producer is dead and spin up a new instance. If the old producer wakes up, it may continue writing data, leading to duplicate or out of order writes.

Kafka solves this using epoch management. When a producer initializes via InitTransactions(), the coordinator increments the producer epoch linked to that transactional.id. If a zombie producer attempts to send a write or request a transaction commit using its old epoch, the broker rejects the request with a ProducerFencedException. The old producer is fenced out, protecting log consistency.

The Consumer Latency Tax: Read Isolation Levels

While the producer pays a write latency tax to coordinate transactions, the consumer also pays a latency tax to read them. This behavior is governed by the isolation.level consumer configuration.

read_uncommitted (Default)

The consumer reads messages sequentially as soon as they are written to the partition log. The consumer ignores transaction boundaries and commits. It will return aborted messages and uncommitted messages immediately. While this mode has zero latency overhead, it provides no consistency guarantees.

read_committed

The consumer is restricted. It will only return messages that are part of committed transactions, or messages that are written without a transaction. If the consumer encounters a message batch from an open transaction, it will buffer that message in memory and block. The consumer cannot progress past the **Last Stable Offset (LSO)**, which is the offset of the first active transaction currently open in the partition.

This blocking behavior introduces a significant latency tax. If your producers commit transactions every 1 second, a consumer operating with read_committed must wait up to 1 second before it can read any messages written after the start of that transaction, even if the messages themselves were processed in milliseconds. The end to end latency of your pipeline becomes bound to the commit frequency of your producers.

Performance Overhead Benchmarks

The performance cost of Kafka transactions is a direct consequence of the extra round trips and disk writes required by the commit protocol. The table below compares throughput and latency characteristics across different producer and consumer configurations based on benchmark studies published in early 2026:

The data reveals a critical trade off. Committing frequently (every 100 milliseconds) keeps consumer read latency low but degrades throughput to 65% because the broker is flooded with metadata writes and transaction markers. Committing less frequently (every 1 second) recovers throughput to 90% but increases consumer read latency to over 1 second because consumers block at the LSO waiting for commits.

The Mitigation and Optimization Checklist

If you require exactly-once guarantees but must minimize the performance tax, implement these optimizations:

1. Optimize Batching Parameters

Because transactions add commit overhead, you must increase batch sizes to amortize the write cost. Configure your transactional producers to use larger batch sizes and enable compression to reduce raw byte footprint:

properties
# Producer properties for transactional batching
enable.idempotence=true
transactional.id=billing-producer-1
linger.ms=50
batch.size=65536
compression.type=snappy

2. Balance Commit Interval with SLA requirements

Do not commit too frequently. A commit interval of under 100 milliseconds will degrade broker performance. Choose a commit interval that represents the maximum latency your application SLA can tolerate on the read side (for example, 500ms or 1000ms), and batch your writes accordingly.

3. Monitor the Last Stable Offset (LSO)

Monitor the difference between the High Watermark (the latest replicated offset) and the Last Stable Offset (LSO) on your brokers. A growing delta between these values indicates a stalled transaction. If a producer crashes without committing or aborting, the LSO will block, preventing all read_committed consumers from progressing. Configure transaction.timeout.ms to a tight but safe value (for example, 15000ms) to ensure zombie transactions are aborted quickly:

properties
# Set transaction timeout to abort stuck sessions
transaction.timeout.ms=15000

4. Increase Partition and Broker Counts

The transaction coordinator writes all states to the __transaction_state internal topic. By default, this topic has 50 partitions. Ensure that these partitions are distributed evenly across your brokers to prevent the coordinator writes from bottlenecking a single CPU.

When to Avoid Transactions

Exactly-Once Semantics is essential for financial transaction pipelines, ledger accounting, and key value enrichment stores where duplicate records corrupt the system state.

However, for telemetry data, web analytics, metric collection, or logging pipelines, the cost of transactions is not justified. In these environments, you should use an idempotent producer (to prevent duplicates on network retries) and accept at-least-once consumer delivery. This combination keeps throughput high and latency low while avoiding the two-phase commit overhead.