Apache Kafka

Apache Kafka is a distributed event streaming platform written in Java and Scala. It runs as a cluster of broker processes that durably store ordered streams of records (events) across replicated, partitioned topics, and exposes producer, consumer, admin, Connect, and Streams APIs for moving and processing those events.

This wiki maps the source tree at https://github.com/apache/kafka to the major subsystems that make up Kafka — the broker (core/), the Java clients (clients/), the KRaft consensus layer (raft/, metadata/), the Streams library (streams/), and Connect (connect/) — and explains how to build, test, and contribute changes.

What it does

A Kafka cluster provides:

• Durable, partitioned, replicated logs. Each topic is split into partitions; each partition is an append-only log replicated across brokers. The on-disk layout and replication protocol live in storage/ and core/src/main/scala/kafka/server/.
• Producer / consumer / admin APIs. Java clients that talk a binary protocol (the Kafka Wire Protocol) to brokers. Implementations live in clients/src/main/java/org/apache/kafka/clients/.
• A consensus-driven control plane (KRaft). Cluster metadata (topics, partitions, ACLs, configs, quotas) is itself stored as a Raft-replicated log on the controller quorum. See raft/ and metadata/.
• Group coordination. Consumer groups, share groups, transactions and Streams rebalances are coordinated by per-broker coordinator components in group-coordinator/, share-coordinator/, and transaction-coordinator/.
• Higher-level streaming. streams/ is a Java library for building stream processing topologies on top of Kafka topics. connect/ is a framework for moving data between Kafka and external systems via reusable connectors.

Quick links

• Architecture — top-level diagram of how brokers, controllers, and clients fit together.
• Getting started — clone, build, run a broker, run the test suite.
• Glossary — Kafka-specific vocabulary (KRaft, ISR, KIP, EOS, etc.).
• How to contribute — KIP process, dev workflow, code style, testing.
• Modules — the Gradle modules and what each one is for.
• Features — cross-cutting capabilities like exactly-once, tiered storage, and KRaft.

Repository layout at a glance

clients/                  Java producer, consumer, admin and shared protocol code
core/                     Scala broker (request handling, replication, KafkaApis)
connect/                  Kafka Connect framework + bundled connectors
streams/                  Kafka Streams library (DSL + processor API)
raft/                     KRaft (Raft) consensus implementation
metadata/                 Cluster metadata image, controller, MetadataLoader
storage/                  Log storage, segments, indexes, remote storage interfaces
group-coordinator/        Consumer group + Streams group coordinator
share-coordinator/        Share group coordinator (KIP-932)
transaction-coordinator/  Transaction coordinator state and protocol
coordinator-common/       Shared coordinator runtime (state machine + replicated log)
server/, server-common/   Shared broker infrastructure used outside core
tools/                    CLI tools (`kafka-topics`, `kafka-console-consumer`, …)
shell/                    Interactive `kafka-metadata-shell`
trogdor/                  Distributed test/fault-injection framework
jmh-benchmarks/           JMH microbenchmarks
generator/                Code generator for RPC message JSON schemas
docs/                     User documentation source
config/                   Sample broker, controller, producer, consumer configs
bin/                      Shell entry points for all CLI tools
tests/                    Python (ducktape) system tests
docker/                   Docker images and compose files

For a deeper tour, see Architecture and the Modules section.