Architecture

Kafka is a distributed system with three logical roles — clients, brokers, and the controller quorum — running the same broker binary configured for different process.roles. This page sketches how they connect, where each piece lives in the source tree, and how a record flows from producer.send() to a replicated log on disk.

Logical topology

graph TD
    subgraph Clients
      P[Producer<br/>clients/.../producer]
      C[Consumer<br/>clients/.../consumer]
      A[Admin<br/>clients/.../admin]
      ST[Streams app<br/>streams/]
      CN[Connect worker<br/>connect/runtime]
    end

    subgraph Broker[Broker process - core/]
      SN[SocketServer + RequestChannel<br/>core/.../network]
      KA[KafkaApis<br/>handles all RPCs]
      RM[ReplicaManager<br/>read/write log, ISR]
      LOG[(LogManager + Log<br/>storage/)]
      GC[Group Coordinator<br/>group-coordinator/]
      TC[Transaction Coordinator<br/>transaction-coordinator/]
      SC[Share Coordinator<br/>share-coordinator/]
      ML[MetadataLoader<br/>metadata/]
    end

    subgraph Controller[Controller quorum - raft + metadata]
      RC[KafkaRaftClient<br/>raft/]
      CTRL[QuorumController<br/>metadata/controller]
      MLOG[(__cluster_metadata log)]
    end

    P -->|Produce| SN
    C -->|Fetch / OffsetCommit| SN
    A -->|Admin RPCs| SN
    ST --> P
    ST --> C
    CN --> P
    CN --> C
    SN --> KA
    KA --> RM
    KA --> GC
    KA --> TC
    KA --> SC
    RM --> LOG
    KA -->|forward write ops| RC
    RC <-->|Raft RPCs| CTRL
    CTRL --> MLOG
    CTRL -->|metadata records| ML
    ML -.->|apply delta| RM
    ML -.->|apply delta| GC

Every broker contains a MetadataLoader that subscribes to the controller's metadata log; cluster state (topics, partitions, configs, ACLs, quotas) is therefore eventually consistent across the cluster as records are appended on the controller and replayed locally.

Process roles

The same JVM binary is launched as one of three role combinations, controlled by the process.roles config:

Role	Components active	Source entry point
broker	BrokerServer (request handling, log, coordinators)	core/src/main/scala/kafka/server/BrokerServer.scala
controller	ControllerServer (KRaft + QuorumController)	core/src/main/scala/kafka/server/ControllerServer.scala
broker,controller (combined / single-node)	both, sharing one SharedServer	core/src/main/scala/kafka/server/SharedServer.scala

KafkaRaftServer (core/src/main/scala/kafka/server/KafkaRaftServer.scala) wires these together and is what kafka.Kafka.main instantiates.

Request lifecycle (Produce)

sequenceDiagram
    participant App as App (KafkaProducer)
    participant Net as NetworkClient
    participant Sock as SocketServer (broker)
    participant Api as KafkaApis
    participant Repl as ReplicaManager
    participant Log as Log (storage)

    App->>Net: send(ProducerRecord)
    Net->>Net: RecordAccumulator batches
    Net->>Sock: ProduceRequest (binary RPC)
    Sock->>Api: handle(request)
    Api->>Repl: appendRecords(...)
    Repl->>Log: append() to leader segment
    Repl-->>Repl: wait for follower fetch / ISR ack
    Repl-->>Api: ProduceResponse
    Api-->>Sock: send response
    Sock-->>Net: response bytes
    Net-->>App: callback / Future

Producer batching and retry logic live in clients/src/main/java/org/apache/kafka/clients/producer/internals/ (Sender, RecordAccumulator, TransactionManager). The broker side is in core/src/main/scala/kafka/server/KafkaApis.scala and ReplicaManager.scala.

Storage layer

Each partition replica is an ordered log of segment files plus index files (.log, .index, .timeindex, .snapshot, .txnindex). The on-disk format and segment management live in storage/src/main/java/org/apache/kafka/storage/internals/log/ and Scala code under core/src/main/scala/kafka/log/. Tiered storage (KIP-405) introduces a RemoteLogManager (storage/) that offloads cold segments to a pluggable RemoteStorageManager.

Control plane (KRaft)

The controller quorum runs KafkaRaftClient (raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java) on top of a dedicated __cluster_metadata topic. Cluster operations (CreateTopics, AlterConfigs, ACL changes) are handled by QuorumController (metadata/src/main/java/org/apache/kafka/controller/QuorumController.java), which appends metadata records to the Raft log. Brokers replay those records via MetadataLoader and MetadataDelta/MetadataImage types in metadata/src/main/java/org/apache/kafka/image/.

For a deeper dive see features/kraft-mode.md, modules/raft.md, and modules/metadata.md.

Coordinators

A subset of brokers host group state for consumer groups, share groups, transactions, and Streams group protocol participants:

• Group coordinator — group-coordinator/src/main/java/org/apache/kafka/coordinator/group/. Implements both the classic and consumer-group (KIP-848) protocols.
• Share coordinator — share-coordinator/ (KIP-932 share groups).
• Transaction coordinator — transaction-coordinator/ and core/src/main/scala/kafka/coordinator/transaction/.

All three sit on top of a shared replicated state-machine runtime in coordinator-common/src/main/java/org/apache/kafka/coordinator/common/runtime/CoordinatorRuntime.java.

Higher-level frameworks

• Kafka Streams (streams/) — a JVM library that compiles a Topology of processors and state stores into Kafka consumer/producer pairs, internal repartition topics, and changelog topics. See modules/streams.md.
• Kafka Connect (connect/runtime/) — a worker framework that runs source/sink connector plugins, distributes tasks, manages offsets, and exposes a REST API. See modules/connect.md.

Wire protocol and code generation

Every RPC is described by a JSON message schema under clients/src/main/resources/common/message/ (e.g. ProduceRequest.json, FetchRequest.json). The generator/ module compiles these schemas into Java request/response classes during the build, so adding a new RPC means editing a .json file rather than hand-writing serialization code.

Build system

A multi-module Gradle build (build.gradle, settings.gradle) drives compilation, test execution, packaging, and quality checks (Checkstyle, SpotBugs, Spotless, JaCoCo/Scoverage). The same Gradle task graph also generates the RPC stubs and runs the full unit/integration suite. See Getting started and how-to-contribute/tooling.md.