Modules

Kafka is a multi-module Gradle project. settings.gradle declares about fifty modules; this section walks the ones a contributor will encounter most often. Use Kafka's own vocabulary — these are referred to as "modules" in the build, not "packages" or "apps".

Top-level layout

clients/                   Java producer/consumer/admin + shared common/* types
core/                      Scala broker (request handling, replication)
connect/                   Kafka Connect runtime + bundled converters/transforms
streams/                   Kafka Streams library
raft/                      KRaft (Raft) consensus implementation
metadata/                  Metadata image, controller, MetadataLoader
storage/                   Log storage, segments, indexes, tiered storage
group-coordinator/         Consumer group coordinator (classic + KIP-848)
share-coordinator/         Share group coordinator (KIP-932)
transaction-coordinator/   Transaction coordinator state and protocol
coordinator-common/        Replicated state-machine runtime for coordinators
server/                    Newer Java broker infra split out of core
server-common/             Shared broker types (configs, metadata version, etc.)
tools/                     Java CLIs (kafka-topics, kafka-console-consumer, ...)
shell/                     Interactive kafka-metadata-shell
trogdor/                   Distributed test/fault-injection framework
jmh-benchmarks/            JMH microbenchmarks
generator/                 RPC schema → Java code generator
test-common/               Shared JUnit utilities used across modules

Per-module pages

Page	Module(s)	What it covers
clients	clients	Producer, consumer, admin, common protocol types and the wire-protocol code generator's output.
core (broker)	core	The Scala broker: KafkaApis, ReplicaManager, request pipeline, log/network layer.
storage	storage, storage-api	On-disk log format, segments, indexes, tiered storage.
raft	raft	KRaft consensus: KafkaRaftClient, election, replication.
metadata	metadata	QuorumController, MetadataImage/MetadataDelta, MetadataLoader.
coordinators	group-coordinator, share-coordinator, transaction-coordinator, coordinator-common	The three coordinators and their shared runtime.
streams	streams, streams-scala, test-utils	DSL, processor API, state stores, partition assignors.
connect	connect/runtime plus api, json, mirror, transforms	Connect framework: workers, herders, REST API, plugins.
server / server-common	server, server-common	Cross-module broker infra (config, metadata version, share/coordinator helpers, etc.).
tools	tools, tools-api	The Java implementations of the kafka-*.sh CLIs.
shell, trogdor, generator, jmh	shell, trogdor, generator, jmh-benchmarks, test-common	Smaller modules grouped onto one page.

How modules depend on each other

graph TD
    GEN[generator] --> CLIENTS[clients]
    CLIENTS --> SERVERCOMMON[server-common]
    CLIENTS --> STORAGEAPI[storage-api]
    SERVERCOMMON --> STORAGE[storage]
    STORAGE --> METADATA[metadata]
    SERVERCOMMON --> METADATA
    METADATA --> RAFT[raft]
    SERVERCOMMON --> RAFT
    SERVERCOMMON --> SERVER[server]
    METADATA --> SERVER
    STORAGE --> SERVER
    METADATA --> COORD[coordinator-common]
    COORD --> GROUP[group-coordinator]
    COORD --> TXN[transaction-coordinator]
    COORD --> SHARE[share-coordinator]
    SERVER --> CORE[core]
    GROUP --> CORE
    TXN --> CORE
    SHARE --> CORE
    RAFT --> CORE
    CLIENTS --> STREAMS[streams]
    CLIENTS --> CONNECT[connect/runtime]
    CLIENTS --> TOOLS[tools]
    METADATA --> SHELL[shell]

A few practical implications of that graph:

• clients/ is the bottom of the dependency tree (after generator/). A change there can rebuild almost everything.
• core/ is at the top of the broker stack and depends on most other broker modules; nothing depends on core/.
• The streams and Connect modules depend only on clients/, never on broker internals — the same constraint that applies to user code.
• server-common/ is the dumping ground for utilities that need to be shared between core/, the coordinators, and metadata/.

Reading order for a new contributor

1. clients/ — start with KafkaProducer.java and KafkaConsumer.java to understand the wire protocol and client state machines.
2. core/src/main/scala/kafka/server/KafkaApis.scala — the broker's RPC dispatcher; every API is a method here.
3. core/src/main/scala/kafka/server/ReplicaManager.scala — the replication and read/write path.
4. storage/src/main/java/org/apache/kafka/storage/internals/log/ — the on-disk log format.
5. raft/ + metadata/ — KRaft and the controller.
6. group-coordinator/ — for anything consumer-group related.
7. streams/ or connect/ — depending on which higher-level framework you care about.