metadata

Active contributors: José Armando García Sancio, Colin Patrick McCabe, Calvin Liu, Alyssa Huang, David Arthur

Purpose

The metadata/ module owns the cluster's metadata model: the controller's authoritative state machine, the immutable in-memory snapshot used by brokers (MetadataImage / MetadataDelta), and the replayer that turns Raft-replicated records into local state changes (MetadataLoader). It is the layer between the raw KRaft log (raft/) and the broker's per-component reactions (in core/, group-coordinator/, etc.).

Directory layout

metadata/src/main/java/org/apache/kafka/
├── controller/
│   ├── QuorumController.java              ← controller event loop + metadata-record producer
│   ├── ReplicationControlManager.java     ← topics, partitions, ISR, reassignment
│   ├── ConfigurationControl(Manager).java ← topic / broker / cluster configs
│   ├── ClientQuotaControlManager.java     ← KIP-546 client quotas
│   ├── ClusterControlManager.java         ← broker registrations, fencing
│   ├── ScramControlManager.java           ← SCRAM credentials
│   ├── ProducerIdControlManager.java
│   ├── DelegationTokenControlManager.java
│   ├── FeatureControlManager.java         ← MetadataVersion + feature levels
│   └── ...
├── image/
│   ├── MetadataImage.java                 ← immutable cluster-state snapshot
│   ├── MetadataDelta.java                 ← record-by-record diff applied to an image
│   ├── TopicsImage.java / FeaturesImage.java / ConfigurationsImage.java / ...
│   ├── loader/
│   │   ├── MetadataLoader.java            ← replays records, fans out to publishers
│   │   ├── MetadataPublisher.java         ← SPI; one impl per consumer (broker, controller, ...)
│   │   └── ...
│   └── publisher/
└── metadata/
    ├── KafkaConfigSchema.java             ← config validation schema for ConfigRecord
    ├── BrokerRegistration.java            ← broker registry entry
    ├── PartitionRegistration.java         ← per-partition replica + ISR + leader epoch
    ├── ReplicaAssignment.java
    ├── authorizer/
    │   ├── StandardAuthorizer.java        ← default Authorizer impl, sourced from MetadataImage
    │   └── ...
    └── ...

The actual records that flow over Raft live as JSON schemas under metadata/src/main/resources/common/metadata/ (e.g., TopicRecord.json, PartitionRecord.json, ConfigRecord.json, AccessControlEntryRecord.json); the generator/ module compiles them into Java classes just like the broker RPCs.

Key abstractions

Type	File	Purpose
QuorumController	metadata/src/main/java/org/apache/kafka/controller/QuorumController.java	Controller event loop; receives admin RPCs and emits metadata records.
ReplicationControlManager	metadata/.../controller/ReplicationControlManager.java	Topic create/delete, partition reassignment, ISR updates, leader election.
ClusterControlManager	metadata/.../controller/ClusterControlManager.java	Broker registrations, fencing, controlled shutdown.
MetadataImage	metadata/src/main/java/org/apache/kafka/image/MetadataImage.java	Immutable snapshot composed of TopicsImage, ClusterImage, ClientQuotasImage, ...
MetadataDelta	metadata/.../image/MetadataDelta.java	Applies a sequence of metadata records to produce a new image.
MetadataLoader	metadata/.../image/loader/MetadataLoader.java	KRaft listener; replays records, maintains current image, fans out to MetadataPublishers.
MetadataPublisher	metadata/.../image/loader/MetadataPublisher.java	SPI implemented by consumers (e.g., BrokerMetadataPublisher in core/).
BrokerRegistration	metadata/.../metadata/BrokerRegistration.java	Per-broker registration record (incarnationId, listeners, fenced/in-controlled-shutdown flags).
PartitionRegistration	metadata/.../metadata/PartitionRegistration.java	Per-partition replica set, ISR, leader epoch.
KafkaConfigSchema	metadata/.../metadata/KafkaConfigSchema.java	Validates broker / topic config records before they are appended.
StandardAuthorizer	metadata/.../metadata/authorizer/StandardAuthorizer.java	Default Authorizer; reads ACL records out of the metadata image.

How a CreateTopics flows

sequenceDiagram
    participant Client
    participant Broker as Broker (KafkaApis)
    participant Fwd as ForwardingManager
    participant Ctrl as Controller (QuorumController)
    participant Raft as KafkaRaftClient
    participant ML as MetadataLoader (broker)

    Client->>Broker: CreateTopicsRequest
    Broker->>Fwd: forward to controller
    Fwd->>Ctrl: Envelope(CreateTopicsRequest)
    Ctrl->>Ctrl: ReplicationControlManager.createTopics<br/>builds TopicRecord + PartitionRecord(s)
    Ctrl->>Raft: scheduleAppend(records)
    Raft-->>Ctrl: committed at offset N
    Ctrl-->>Broker: CreateTopicsResponse (after commit)
    Raft-->>ML: handleCommit(records)
    ML->>ML: build MetadataDelta + new image
    ML->>BrokerMetadataPublisher: onMetadataUpdate(delta, image)

The same pattern applies to AlterConfigs, AlterPartitionReassignments, CreateAcls, AlterClientQuotas, etc.: the broker forwards to the controller; the controller computes records and appends to the Raft log; brokers eventually replay via MetadataLoader.

MetadataImage / MetadataDelta

The image is immutable — every committed batch of records produces a new MetadataImage. A MetadataDelta accumulates the changes from one or more record batches and exposes them as a typed diff (e.g., topicsDelta()). Publishers subscribe to deltas, not images, so they can react to specific events:

graph LR
    Recs((records)) --> Delta[MetadataDelta]
    Image1[MetadataImage A] --> Delta
    Delta --> Image2[MetadataImage B]
    Delta --> P1[BrokerMetadataPublisher]
    Delta --> P2[GroupCoordinatorMetadataPublisher]
    Delta --> P3[KRaftMigrationDriver - legacy]

BrokerMetadataPublisher (core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala) is the broker-side reactor — it tells ReplicaManager about partition changes, DynamicConfigManager about config changes, the authorizer about ACL changes, and the dynamic config manager about cluster-wide config changes.

MetadataVersion and feature levels

FeatureControlManager and MetadataVersion (in server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java) gate behavior changes by an integer version. Each new RPC, on-disk format, or controller feature is introduced under a new MetadataVersion; clusters upgrade by issuing an UpdateFeatures RPC, which causes the controller to write a FeatureLevelRecord and brokers to switch on the new behavior. This is also how dynamic voter reconfiguration (KIP-853), KRaft migration, and tiered storage features are gated.

Snapshots

When the metadata log gets long, the controller writes a MetadataImage snapshot via the Raft Snapshot API; new brokers can join by fetching the snapshot first and then catching up via Raft Fetch. SnapshotReader / SnapshotWriter types in metadata/src/main/java/org/apache/kafka/metadata/util/ encode/decode the records.

Authorizer

StandardAuthorizer indexes ACL records pulled from MetadataImage.acls(). The AclControlManager on the controller side adds/removes ACL records when CreateAcls / DeleteAcls RPCs arrive. Because authorization data is part of the same metadata log, broker authorization decisions are eventually consistent across the cluster. See features/acls-and-security.md.

Entry points for modification

• Add a new metadata record type: define a JSON schema under metadata/src/main/resources/common/metadata/, add a corresponding *ControlManager to produce it, and update the relevant *Image/*Delta to consume it. KIP required.
• Change controller behavior for an existing RPC: edit the relevant *ControlManager (e.g., ReplicationControlManager for topic operations) and the corresponding image/delta. Tests live in metadata/src/test/.
• Hook into metadata events as a new component: implement MetadataPublisher and register it through MetadataLoader. This is how new broker subsystems (e.g., share coordinator) wire themselves up.
• Bump MetadataVersion: add a new value to MetadataVersion.java, update upgrade tests, and gate the new behavior on metadataVersion.isAtLeast(...).

Related pages

• Modules: raft — the underlying log replication.
• Modules: core (broker) — BrokerMetadataPublisher, ControllerServer.
• Features: KRaft mode.
• Features: ACLs and security.