raft

Active contributors: José Armando García Sancio, Colin Patrick McCabe, Calvin Liu, Alyssa Huang, PoAn Yang

Purpose

The raft/ module is Kafka's implementation of the Raft consensus protocol, used to replicate the cluster's metadata log (__cluster_metadata) across the controller quorum. It was introduced as part of KIP-595 in late 2020 and is the foundation of KRaft mode — the only supported control plane in Kafka 4.x. The metadata/ module sits on top of this; brokers consume the same log as observers.

Directory layout

raft/src/main/java/org/apache/kafka/raft/
├── KafkaRaftClient.java             ← (~4,400 lines) the Raft state machine + network driver
├── RaftClient.java                  ← interface implemented by KafkaRaftClient
├── RaftManager.java                 ← higher-level lifecycle helper used by Server modules
├── KafkaRaftClientDriver.java       ← single-threaded IO loop ("kafka-raft-io-thread")
├── KafkaNetworkChannel.java         ← bridges Raft RPCs onto the regular client/broker NetworkClient
├── QuorumState.java                 ← persistent voted/leader/epoch state
├── LeaderState.java / FollowerState.java / CandidateState.java / ProspectiveState.java / UnattachedState.java / ResignedState.java
├── Endpoints.java / VoterSet.java / DynamicVoter.java(s)   ← voter set + reconfiguration (KIP-853)
├── ElectionState.java
├── RequestManager.java              ← outbound request scheduling, per-node retry
├── KRaftConfigs.java / QuorumConfig.java / MetadataLogConfig.java
├── RaftUtil.java / RaftMessage.java / RaftRequest.java / RaftResponse.java
├── Batch.java / BatchReader.java / ControlRecord.java
├── FileQuorumStateStore.java        ← persists QuorumState to disk
├── ExternalKRaftMetrics.java
├── TimingWheelExpirationService.java
└── internals/                       ← KafkaRaftMetrics, fetch/append helpers, voters table

Key abstractions

Type	File	Purpose
RaftClient	raft/src/main/java/org/apache/kafka/raft/RaftClient.java	Public-ish interface: register, prepareAppend, scheduleAppend, poll.
KafkaRaftClient	raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java	The implementation — election timer, log replication, snapshot management, listener dispatch.
KafkaRaftClientDriver	raft/.../KafkaRaftClientDriver.java	Owns the kafka-raft-io-thread; calls client.poll() in a loop.
QuorumState	raft/.../QuorumState.java	The persisted election state (epoch, voted-for, leader, etc.) plus role transitions.
LeaderState, FollowerState, CandidateState, ProspectiveState, UnattachedState, ResignedState	raft/.../*State.java	Per-role state; the union forms the Raft state machine.
VoterSet / Endpoints	raft/.../VoterSet.java, Endpoints.java	Voter directory used for reconfiguration.
KafkaNetworkChannel	raft/.../KafkaNetworkChannel.java	Sends Raft RPCs (Vote, BeginQuorumEpoch, EndQuorumEpoch, Fetch, FetchSnapshot).
FileQuorumStateStore	raft/.../FileQuorumStateStore.java	Stores QuorumState in quorum-state file under the metadata log dir.
RaftLog / LogAppendInfo / LogFetchInfo	raft/.../RaftLog.java	Storage interface; backed by KafkaMetadataLog (in metadata/) which uses UnifiedLog.
MetadataLogConfig / QuorumConfig / KRaftConfigs	raft/.../*Config.java	Configuration of the Raft client (election timeout, fetch timeout, snapshot frequency, ...).

State machine

stateDiagram-v2
    [*] --> Unattached
    Unattached --> Prospective: random election timer elapsed
    Prospective --> Candidate: pre-vote granted by quorum
    Candidate --> Leader: votes >= majority
    Candidate --> Unattached: term advanced / lost
    Leader --> Resigned: stepDown / shutdown
    Resigned --> Unattached: timeout
    Follower --> Unattached: leader timeout
    Unattached --> Follower: discover higher term + leader
    Leader --> Follower: discover higher term
    Candidate --> Follower: discover higher term + leader

The Prospective state was added to support pre-vote (avoid unnecessary term bumps when a partitioned voter rejoins). Resigned allows a leader that has stepped down to drain in-flight requests cleanly.

Replication

sequenceDiagram
    participant L as Leader
    participant F as Follower (voter)
    participant O as Observer (broker)
    L->>F: BeginQuorumEpoch
    F->>L: Fetch(metadata-log, fetchOffset)
    L-->>F: Fetch response with records
    F-->>F: append + advance HW (when 1+f voters acked)
    O->>L: Fetch(metadata-log, fetchOffset)
    L-->>O: Fetch response (read-only)

Voters, observers, and brokers all use the same Fetch RPC — voters' fetches additionally count toward HW advancement and update the leader's notion of "follower caught up". Followers and observers periodically fetch a Snapshot (KIP-630) when their offset falls below the leader's start offset.

Listener model

Consumers of the Raft log register a RaftClient.Listener<T>:

interface Listener<T> {
    void handleCommit(BatchReader<T> reader);
    void handleSnapshot(SnapshotReader<T> reader);
    void handleLeaderChange(LeaderAndEpoch leaderAndEpoch);
}

The single-threaded driver invokes the listener for every committed batch in order. The two main listeners are:

• QuorumController (controller role) — applies metadata records as state changes (creates partitions, updates ACLs, etc.).
• MetadataLoader (broker role) — replays records into MetadataImage/MetadataDelta; downstream subscribers (BrokerMetadataPublisher, group coordinator, replica manager, ...) update local state.

KRaft configuration

The most important options (declared in KRaftConfigs.java and QuorumConfig.java):

Config	Effect
process.roles	broker, controller, or broker,controller
node.id	Stable identifier across roles
controller.quorum.voters	(Static) node-id@host:port list of voters
controller.quorum.bootstrap.servers	Bootstrap list when not using static voters
metadata.log.dir	Optional dedicated log dir for __cluster_metadata
controller.listener.names	Listeners only the controller role uses
metadata.log.segment.bytes, ...	Snapshot / segment knobs for the metadata log

Dynamic voter reconfiguration (KIP-853) lets the cluster add or remove controller voters without restarting; DynamicVoter.java and the AddRaftVoter / RemoveRaftVoter RPCs implement it.

Threading model

KafkaRaftClient is single-threaded by design. KafkaRaftClientDriver runs client.poll() on the kafka-raft-io-thread and is the only thread that touches any state. External callers (scheduleAppend, listener registration) hand off via thread-safe queues. This makes the state machine easier to reason about than a multi-threaded Raft implementation, at the cost of needing the IO loop to be fast.

Snapshots

When the metadata log grows past metadata.log.max.snapshot.interval.ms of records (or a size threshold), the controller writes a Snapshot containing the current MetadataImage. Subsequent followers / observers can fetch the snapshot via FetchSnapshot instead of replaying from the beginning. Snapshot files live alongside the log under __cluster_metadata-0/.

Entry points for modification

• Change election or replication: KafkaRaftClient.java plus the per-state classes (LeaderState, FollowerState, CandidateState, ProspectiveState). State transitions go through QuorumState. Be cautious — any change is protocol-affecting and needs a KIP.
• Add or change a Raft RPC: schema lives in clients/src/main/resources/common/message/Vote{Request,Response}.json, etc. New RPCs are then handled in KafkaRaftClient.handle* methods.
• Add a listener that consumes the metadata log: implement RaftClient.Listener<ApiMessageAndVersion> and register it via KafkaRaftClient.register.
• Tune timers: QuorumConfig.java has the timeouts; defaults are deliberate, change with care.

Related pages

• Modules: metadata — QuorumController, MetadataImage, MetadataLoader are the main consumers.
• Modules: core (broker) — KafkaRaftServer, SharedServer, ControllerServer wire this in.
• Features: KRaft mode — operator-level view.