2024 · 06 • 13 min read

Raft Is Not Magic: What the Paper Does Not Explain

The original Raft paper by Diego Ongaro and John Ousterhout, titled "In Search of an Understandable Consensus Algorithm," is a model of clarity in computer science literature. It can be read and fully understood in a single afternoon. The paper describes a system for leader election, log replication, and safety proofs, providing a clean algorithm for maintaining a replicated state machine across independent network hosts. It makes consensus feel approachable.

However, running that same algorithm in production at scale teaches you lessons that are omitted from the academic text. The paper is designed for correctness and understandability. It is not an operations manual. When you build or run a Raft based consensus engine (such as etcd, JRaft, or TiKV), you quickly discover operational gaps. Production systems require extensions like pre-vote protocols, leader leases, and chunked, resumable snapshot transfers. The fact that every major production implementation has built these same extensions independently shows that these gaps are real operational requirements, not minor details.

The Disruptive Partition Problem

In the basic Raft algorithm, any node that observes a higher term immediately updates its current term and transitions to a follower state. If that node is currently the leader, it must step down. This simple rule is a core safety mechanism designed to ensure that if a new leader is elected during a partition, the old leader will step down as soon as it reconnects. However, in a real network, this rule introduces a serious vulnerability known as the disruptive partition storm.

Consider a healthy five node Raft cluster operating in term 1. Node A is the leader, and nodes B, C, D, and E are followers. A network partition occurs, completely isolating node E from the rest of the cluster. The main cluster (A, B, C, D) maintains a majority and continues to process writes normally.

Meanwhile, node E no longer receives heartbeats from the leader. Its election timer expires. Node E increments its term to 2, transitions to a candidate state, and broadcasts a RequestVote RPC. Because it is partitioned, it receives no responses. Its election timer expires again. Node E increments its term to 3, enters a new election, and fails. This cycle repeats. During a long partition, node E's term will inflate. It can easily reach term 50 while the main cluster remains in term 1.

When the partition heals, node E reconnects to the cluster and sends a message containing its inflated term. The current leader, node A, receives this packet, detects term: 50, and immediately steps down to a follower state. The entire cluster halts write operations and enters an unnecessary election cycle. Because node E's log is stale (it missed all writes during the partition), node E cannot be elected leader. The cluster must wait for an election timeout before a healthy node is re elected, causing a brief period of write unavailability. If the network is unstable, this cycle can repeat, leading to a partition storm.

UNOPTIMIZED DISRUPTIVE PARTITION: [Node E (isolated)] -> Timeout -> Term 2 -> Timeout -> Term 3 -> Timeout -> Term 50 Partition Heals: [Node E (Term 50)] -- Sends AppendEntries reply/RequestVote --> [Leader Node A (Term 1)] Result: Leader Node A steps down. Cluster write operations halt. OPTIMIZED WITH PRE-VOTE: [Node E (isolated)] -> Timeout -> Pre-Vote request (Term 1) -> No majority response Result: Term remains at 1. No disruption occurs when partition heals.

To resolve this, production implementations use the pre-vote extension, as documented in the SOFAStack JRaft manual in 2025. Before a follower increments its term and declares a formal candidacy, it enters a pre-candidate state. In this state, the node broadcasts a PreVote RPC to the cluster, asking "would you vote for me if I started an election?" The key difference is that the node does not increment its term during the pre-vote phase. Followers evaluate the pre-vote request using standard eligibility rules, checking if the sender's log is at least as up to date as their own. If the sender is partitioned, it will not receive a majority of positive responses. Its term remains unchanged, and when the partition heals, it reconnects quietly without disrupting the active leader.

Leader Leases and the Linearizable Read Problem

The base Raft paper states that a leader can serve reads by returning the state from its local state machine. However, if a network partition occurs, a leader can be isolated with a minority of nodes while the majority elects a new leader. If the partitioned leader continues to serve local reads, it will return stale data. This violates linearizable read consistency.

To guarantee linearizability without the pre-vote extension, a leader must replicate a read request as a dummy log entry through the full consensus pipeline before returning the result. This approach ensures the leader is still valid, but it reduces read performance. A system designed to handle thousands of reads per second cannot afford to write a consensus log entry for every read operation.

Production consensus engines address this using two main techniques: time bounded leader leases and etcd's ReadIndex mechanism.

1. Time-Bounded Leader Leases

A leader lease is a time window during which the leader is guaranteed to be valid. When a leader successfully sends a heartbeat to a majority of nodes, it establishes a lease (for example, 500 milliseconds). The followers promise not to vote for any other candidate during this lease window. As long as the leader's local clock has not exceeded the lease duration, it can serve linearizable reads directly from memory without consulting other nodes.

The lease model is highly performant, but it introduces a dependency on bounded clock drift. If the leader's CPU experiences a long pause (such as a garbage collection cycle or a virtualization freeze) and its clock drifts, it can serve stale reads after its lease has actually expired on the followers. Systems using leases must calibrate their lease windows with safe margins to tolerate clock drift.

2. The ReadIndex Mechanism

To avoid clock drift assumptions, etcd's raft library implements the ReadIndex protocol, described in deepwiki's etcd implementation notes. When a read request arrives, the leader records its current commit index as the ReadIndex. Before serving the read, the leader must confirm its authority by sending a low overhead heartbeat to the cluster. Once a majority of nodes acknowledge the heartbeat, the leader waits for its local state machine to apply up to the recorded ReadIndex, and then returns the value. This ensures linearizability without clock dependencies, though it adds a network round trip for the heartbeat confirmation.

The Hidden Cost of Log Divergence Repair

The Raft paper describes log alignment using the AppendEntries consistency check. If a follower's log diverges from the leader's, the leader decrements the follower's nextIndex pointer and retries the AppendEntries RPC. This backtracking continues until the leader finds a common prefix in the logs. Once found, the leader overwrites the follower's divergent entries with its own.

In a production system, this backtracking repair process can be expensive. If a follower has been offline or partitioned for a long time, the gap between the leader's log and the follower's log can span millions of entries. If the leader has already compacted its log (by generating a snapshot and deleting old log segments), it cannot backtrack far enough to find the common prefix. In this case, the leader must trigger a full snapshot transfer.

Even if the log has not been compacted, sending a long sequence of backtracking RPCs introduces CPU and network overhead. In a Go implementation writeup by Viraj Bhartiya in 2025, the author notes that backtracking one entry at a time under high write amplification can stall the consensus loop. Production systems optimize this by implementing fast recovery backtracking. When a follower rejects an `AppendEntries` RPC, it returns additional metadata, such as the term of the conflicting entry and the first index of that term. This allows the leader to decrement `nextIndex` by entire terms in a single round trip, reducing the time to align logs.

The Commit-Current-Term-Only Rule

One of the most common implementation errors in custom Raft engines involves the safety rule governing log commits. The rule states that a leader can only commit a log entry from its current term by counting replicas. If an entry is from a previous term, the leader cannot commit it based on replica count alone. It must instead commit an entry from its current term, which indirectly commits the older entries through Raft's log matching property.

The reason for this rule is detailed in Section 5.4.2 of the Raft paper. If a leader were allowed to commit an old term entry directly, a subsequent leader election could overwrite that entry, violating the safety guarantee. The following Go code snippet shows the correct validation logic in the consensus event loop, where matchIndex is tracked per node:

go
// Correct implementation of commit index advancement in Raft
func (rf *Raft) updateCommitIndex() {
    for N := rf.commitIndex + 1; N <= len(rf.log); N++ {
        // Safety Rule: Only commit entries from the current term directly
        if rf.log[N].Term != rf.currentTerm {
            continue
        }
        
        matches := 1
        for peerId, matchIdx := range rf.matchIndex {
            if peerId != rf.me && matchIdx >= N {
                matches++
            }
        }
        
        // Check if a majority of replicas have acknowledged the index
        if matches > len(rf.peers)/2 {
            rf.commitIndex = N
        }
    }
}

Failing to include the term check in the guard condition is a classic correctness trap. If this check is omitted, a leader can falsely mark an inherited write as committed, returning a success status to the client, only to have that write overwritten by a subsequent leader election under specific network failure scenarios.

Snapshot Transfer Is Not a Single Atomic Step

The Raft paper describes log compaction and snapshotting in Section 7. When a follower falls too far behind, the leader compacts its local log and sends a snapshot to the follower using the InstallSnapshot RPC. The paper describes this process conceptually as a single RPC transfer.

In a production database, a snapshot contains the entire data state, which can span gigabytes or terabytes. Sending this volume of data as a single atomic payload is fragile. A network interruption mid transfer forces the leader to restart the transfer from byte zero, which can prevent a lagging follower from ever catching up.

To build a reliable recovery path, production systems stream snapshots in sequential, chunked payloads, as documented in the SOFAStack JRaft architecture guide. The leader splits the snapshot file into chunks (for example, 1 megabyte blocks) and tracks the offset in the InstallSnapshot request. The follower acknowledges each chunk and writes it to a temporary file. If a network failure occurs, the leader can resume the transfer from the last acknowledged offset. If a node is down for a long period, an interrupted snapshot transfer will stall the replica indefinitely unless the system supports resumable chunking.

Raft Operational Failure Reference

The table below summarizes how a Raft cluster behaves under different failure scenarios based on operational manuals from etcd and JRaft in 2025:

Failure Scenario Cluster Impact Observable System Signatures Recovery Pathway
Minority Node Failure (e.g. 2 of 5 down) None. The remaining 3 nodes maintain a majority and process writes. Minor latency increase due to loss of parallel replica confirmations. Log divergence metrics rise. Reschedule failed containers. The leader will automatically catch up the new nodes.
Leader Failure Brief write unavailability (typically 100 to 500 milliseconds) during election. Write timeouts at client level. Term count increments. Connection drops on client sockets. The remaining nodes elect a new leader. Uncommitted writes on the old leader are discarded.
Majority Failure (e.g. 3 of 5 down) Total write unavailability. The remaining 2 nodes cannot achieve a quorum. All write requests fail with timeout or leadership errors. ReadIndex calls block. Requires restoring the failed nodes, or manually resetting the cluster size to a lower value.
Network Partition (3 vs 2 split) The majority partition (3 nodes) continues writes. The minority partition blocks writes. Client routing errors on minority nodes. High election attempt metrics in the minority. Heals automatically when the partition closes. Pre-vote prevents term inflation.

The Production Checklist

If you are deploying etcd, CockroachDB, or a custom Raft implementation in production, implement these configuration guards: