Lore

A timeline of the Kafka codebase, derived from git log and the project's KIP record. Dates are taken from the earliest commit that introduced the change on the default branch; where a feature shipped over many months, the range covers the bulk of the work.

Eras

LinkedIn origins (Jan 2011 – Jul 2011, pre-Apache)

Kafka was originally developed inside LinkedIn. The Apache repository itself starts on 2011-08-01 with a single import commit:

Initial checkin of Kafka to Apache SVN. This corresponds to https://github.com/kafka-dev/kafka/commit/709afe4ec75489bc00a44335de8821fa726bb97e ... — 642da2f28c

There is no individual commit history before that point in this repository.

Incubation and first releases (2011 – 2014)

Kafka entered the Apache Incubator and shipped the 0.7.x and 0.8.x lines. The 0.8.x branch introduced replication and the partition leader/follower model that still underpins the broker today (core/src/main/scala/kafka/server/). Activity stayed modest, ~300 commits/year.

Streams and Connect (Sep 2015 – 2017)

Two major frameworks were added in late 2015 and stabilized through 2017:

• Kafka Streams — first commit 263c10ab7c on 2015-09-25 (KIP-28). The library was built directly on top of the Java consumer/producer and grew the DSL and processor API in streams/.
• Kafka Connect — initially developed under the name "Copycat"; renamed to Kafka Connect on 2015-11-08 (f2031d4063, KAFKA-2774). The connect/runtime/ worker, Herder, and REST layer date from this era.

Annual commit counts roughly doubled (1,150 in 2016, 1,507 in 2017) reflecting the additional surface area.

Exactly-once and the binary protocol generator (2017 – 2019)

KIP-98 (idempotent + transactional producer) landed in this window, introducing the TransactionManager (clients/.../producer/internals/) and the transaction coordinator (core/src/main/scala/kafka/coordinator/transaction/). The RPC schema generator (generator/) was introduced to replace hand-written serialization — every RPC under clients/src/main/resources/common/message/ is now a JSON schema.

KRaft and the controller rewrite (Sep 2020 – 2024)

By far the largest architectural change in Kafka's history was replacing the ZooKeeper-based controller with an internal Raft-replicated metadata log:

• 2020-09-22 b7c8490cf4 — Core Kafka Raft Implementation (KIP-595) lands KafkaRaftClient and the raft/ module.
• 2021-01-15 — BrokerServer.scala, ControllerServer.scala, and SharedServer.scala are introduced, splitting the old monolithic KafkaServer into role-specific servers.
• 2021 – 2023 — The metadata/ module grows around QuorumController, MetadataImage, MetadataDelta, and MetadataLoader. Brokers learn to bootstrap from metadata records instead of ZooKeeper watches.
• 2024 – 2025 — Production rollout, ZK→KRaft migration tooling, and finally removal of ZooKeeper code paths (see next era).

The 2024 commit count (2,448) is the highest in the project's history, driven largely by KRaft maturation.

ZooKeeper removal (2025)

A series of "remove zookeeper X" commits landed throughout 2025, finishing the transition:

• b6e6a3c68a — KAFKA-18360 Remove zookeeper configurations
• cf8d3ac49e — KAFKA-18530 Remove ZooKeeperInternals
• 85bd059559 — KAFKA-18545 Remove Zookeeper logic from LogManager

After these, the code paths that connected brokers to ZK no longer exist; KRaft is the only supported control plane in 4.x.

New consumer protocol and share groups (2024 – 2026)

Two large client-facing changes are in flight on trunk:

• KIP-848 — new consumer group protocol. Moves assignment logic from clients to the broker. Implementation in group-coordinator/src/main/java/org/apache/kafka/coordinator/group/ and consumer-side membership in clients/.../consumer/internals/. GroupMetadataManager.java is currently the most-touched file in the repository.
• KIP-932 — share groups (queue-style consumption). A separate share-coordinator/ module, plus KafkaShareConsumer and KafkaShareGroup types in clients/.../consumer/. Marked generally available in Nov 2025 (20856 — Mark KIP-932 as generally available).

Longest-standing code

A few directories and types have survived from the very early Apache years:

• core/src/main/scala/kafka/log/ — segment-based log storage was present in the original 2011 import and still backs the broker, though much of the cold-path code has been ported to Java in storage/src/main/java/org/apache/kafka/storage/internals/log/.
• core/src/main/scala/kafka/network/ — the SocketServer / Acceptor / Processor request pipeline is conceptually unchanged since 0.8.x.
• KafkaApis.scala — has existed under its current name since 0.8 and still dispatches every broker RPC. It has grown into one of the largest files in the repo (~5,200 lines).
• clients/src/main/java/org/apache/kafka/common/Cluster.java and Node.java — the in-memory cluster topology types have been around since the unified Java clients shipped in 0.8.2 (2014).

Major rewrites and migrations

Year	Change
2014	Unified Java producer (KafkaProducer) replaces the original Scala producer
2015 – 2016	Java consumer (KafkaConsumer) replaces the high-level + simple Scala consumers
2017 – 2018	Idempotent + transactional producer (KIP-98) — adds TransactionManager, transaction coordinator
2017+	RPC code generator (generator/) replaces hand-written request/response code
2020 – 2024	KRaft replaces ZooKeeper as the control plane
2023 – 2025	KIP-405 tiered storage moves cold segments to a pluggable RemoteStorageManager
2024 – 2026	KIP-848 broker-driven consumer protocol replaces the legacy "embedded" protocol
2025	All ZooKeeper code paths removed

Deprecated / removed

• MirrorMaker 1 — the original kafka.tools.MirrorMaker Scala tool was removed; only Streams system-test stubs that exercise older versions remain. MM2 (Connect-based) under connect/mirror/ is the supported replacement.
• kafka-storage ZK-based formatting — the legacy "register broker" path that wrote into ZK is gone; only KRaft formatting remains in tools/ (kafka-storage.sh).
• Zookeeper internals (zk package) — entirely removed in early 2025.
• kafka-preferred-replica-election.sh — replaced by kafka-leader-election.sh.

Growth trajectory

• 2011-2014: foundational broker, mostly Scala, ~300 commits/year.
• 2015-2017: Streams + Connect added; commit count more than doubles.
• 2018-2020: protocol versioning, transactions, and exactly-once shipping. ~1,200/year.
• 2021-2023: KRaft incubation. 1,200 – 1,500/year.
• 2024-2025: KRaft GA + ZK removal + KIP-848 + KIP-932 — the busiest years on record (~2,300/year average).

The repository has 17,420 commits in total at the time of this wiki.