KRaft mode

Active contributors: José Armando García Sancio, Colin Patrick McCabe, Calvin Liu, David Arthur, Alyssa Huang

KRaft (Kafka Raft) is the consensus protocol that replaced ZooKeeper as Kafka's control plane. As of Kafka 4.x it is the only supported control plane — ZooKeeper code paths were removed in early 2025. This page explains how KRaft fits together at runtime, where in the source tree the relevant code lives, and how operators interact with it.

Why KRaft

ZooKeeper was Kafka's metadata store from the project's inception. It was an external dependency operators had to deploy, monitor, and upgrade separately, and the design fundamentally limited the cluster's scalability (write throughput on metadata, time-to-replay during controller failover). KIP-500 introduced a new control plane in which:

• Cluster metadata is itself stored as a Kafka log (__cluster_metadata) replicated by Raft.
• A small number of broker processes participate as voters in the Raft quorum and host the controller.
• Other brokers replicate the metadata log as observers and apply the records into local in-memory state.

The result: one fewer piece of infrastructure to run, faster controller failover, and a metadata model (MetadataImage / MetadataDelta) that scales to millions of partitions.

Architecture

graph TD
    subgraph Quorum[Controller quorum - 3 voters]
      C1[Controller 1<br/>Leader]
      C2[Controller 2<br/>Follower]
      C3[Controller 3<br/>Follower]
      C1 -.replicate.-> C2 & C3
    end
    subgraph Brokers[Brokers - observers]
      B1[Broker 100]
      B2[Broker 101]
      B3[Broker 102]
    end
    Quorum -->|Fetch metadata| B1 & B2 & B3
    Client -->|admin RPC| B1
    B1 -->|forward to controller| C1
    C1 -->|append metadata records| Quorum
    C1 -->|response| B1 -->|response| Client

Three roles, configured via process.roles:

• controller — voter; participates in Raft, runs QuorumController. Implementation: core/src/main/scala/kafka/server/ControllerServer.scala.
• broker — observer; replicates the metadata log read-only and reacts to deltas via BrokerMetadataPublisher. Implementation: core/src/main/scala/kafka/server/BrokerServer.scala.
• broker,controller — combined mode; one JVM hosts both. Common for development and small clusters. Wired up via SharedServer.scala.

Lifecycle

Bootstrapping

Before first start, an operator runs:

KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"
./bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties

StorageTool.java (tools/) writes meta.properties to each log directory and bootstraps the metadata log with a BootstrapMetadata record set (cluster ID, initial metadata.version, initial voter set). For a multi-node cluster, every node must be formatted with the same cluster ID and the same controller.quorum.voters.

Boot

sequenceDiagram
    participant K as Kafka.main
    participant SS as SharedServer
    participant Raft as KafkaRaftClient
    participant ML as MetadataLoader
    participant CTRL as ControllerServer (if controller role)
    participant BRK as BrokerServer (if broker role)

    K->>SS: startForBroker / startForController
    SS->>Raft: load QuorumState; election timer starts
    SS->>ML: register publishers
    Raft->>Raft: discover leader / become candidate / become leader
    Raft-->>ML: handleCommit(records since last checkpoint)
    ML->>ML: apply records → new MetadataImage
    ML-->>BRK: BrokerMetadataPublisher.onMetadataUpdate
    ML-->>CTRL: QuorumController catches up
    BRK->>BRK: bind listeners; advertise itself via Heartbeat

Broker registration: a broker sends BrokerRegistration and periodic BrokerHeartbeat RPCs to the active controller. ClusterControlManager writes a RegisterBrokerRecord and BrokerRegistrationChangeRecords into the metadata log; brokers see the changes via MetadataLoader and update their Cluster snapshot accordingly.

Operator interactions

Action	RPC / Tool
Inspect quorum status	kafka-metadata-quorum.sh describe --status
Add / remove a controller voter (KIP-853)	kafka-metadata-quorum.sh add-voter / remove-voter
Force preferred leader election	kafka-leader-election.sh --election-type preferred
Change cluster-wide config	kafka-configs.sh --entity-type brokers --entity-default
Upgrade metadata.version	kafka-features.sh upgrade --metadata <version>
Inspect the metadata log	kafka-dump-log.sh --files .../__cluster_metadata-0/*.log
Browse metadata interactively	kafka-metadata-shell.sh --snapshot ...

Code layout

Concern	Where
Raft state machine	raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java
Controller event loop	metadata/src/main/java/org/apache/kafka/controller/QuorumController.java
Metadata records (schemas)	metadata/src/main/resources/common/metadata/
Image / delta types	metadata/src/main/java/org/apache/kafka/image/
Loader → publisher fan-out	metadata/src/main/java/org/apache/kafka/image/loader/MetadataLoader.java
Broker side reactor	core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala
Lifecycle wrapper	core/src/main/scala/kafka/server/{KafkaRaftServer,SharedServer,BrokerServer,ControllerServer}.scala
Format tool	tools/src/main/java/org/apache/kafka/tools/StorageTool.java
Quorum CLI	tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java

Dynamic voter reconfiguration (KIP-853)

Clusters can change the voter set without restart. DynamicVoter, DynamicVoters, and the AddRaftVoter / RemoveRaftVoter RPCs in raft/ implement this. Operators run kafka-metadata-quorum.sh add-voter --replica-id=<n> --replica-uuid=<uuid> --replica-directory=<dir> and the controller writes a VotersRecord that the quorum atomically applies via the standard joint-consensus pattern.

Snapshots

The metadata log is finite-but-large; new brokers can't replay from the beginning forever. The controller periodically writes MetadataImage snapshots (KIP-630) using the Raft Snapshot API. New brokers fetch the latest snapshot via FetchSnapshot and then replay only the tail of the log. Snapshot files live alongside the log under __cluster_metadata-0/.

Failure modes

• Quorum loss — if a majority of voters are down, the cluster cannot make metadata changes (no topic create / config change / broker registration). Brokers continue to serve existing partitions because client RPCs are handled locally and don't require controller synchronicity.
• Lagging broker — a broker that fetches the metadata log slowly is fenced by the controller (fenced=true in its BrokerRegistration); leadership avoids it. Once it catches up and heartbeats again, it is unfenced.
• Split log dirs — brokers can have multiple log dirs; each gets its own meta.properties. If they disagree, the broker refuses to start.
• Stale node ID — KRaft stores the broker's incarnation in BrokerRegistration. If a node is restarted with a duplicate ID, the controller rejects the registration until the previous incarnation is fenced.

ZK migration (historical)

For Kafka 3.x there was a "KRaft migration" path that ran ZK and KRaft in parallel during the transition. As of 4.x this code is gone — clusters are required to be all-KRaft. The remaining MigrationDriver references in metadata/ are stubs preserved for upgrade tests.

Related pages

• Modules: raft — the Raft client itself.
• Modules: metadata — controller, image / delta, loader.
• Modules: core (broker) — broker side wiring.
• Glossary — voter / observer / metadata.version vocabulary.