Glossary

Kafka has accumulated a lot of project-specific vocabulary across protocols, clients, and the broker. This page is a quick reference; deeper treatment lives on the linked pages.

Cluster topology

• Broker — a JVM process that stores partition replicas and serves client RPCs. Implementation: core/src/main/scala/kafka/server/BrokerServer.scala.
• Controller — broker(s) that own cluster metadata. Under KRaft, controllers form a Raft quorum. Implementation: core/src/main/scala/kafka/server/ControllerServer.scala, metadata/src/main/java/org/apache/kafka/controller/QuorumController.java.
• Combined mode — a process running both broker and controller roles in one JVM (typical for single-node dev setups). See SharedServer.scala.
• Voter / Observer — KRaft terms. Voters can vote in elections; observers (brokers reading the metadata log) cannot.
• Node ID — the broker's node.id. Replaces the older "broker.id" / "controller.id" pair.

Logs and storage

• Topic — a named stream of records.
• Partition — a topic is split into one or more partitions, each an independent ordered log.
• TopicIdPartition — (topicId, partition); the broker's preferred identifier since topic IDs were introduced. See clients/.../common/TopicIdPartition.java.
• Replica — a copy of a partition on a particular broker. One replica is the leader; the rest are followers.
• ISR (In-Sync Replicas) — followers that have caught up to the leader within replica.lag.time.max.ms. Only ISR members are eligible for leader election.
• LEO (Log End Offset) — the offset of the next record that will be appended.
• HW (High Watermark) — the highest offset replicated to all ISR followers; consumers can only read up to the HW.
• Segment — a log file plus its .index and .timeindex companions. Once a segment exceeds segment.bytes or segment.ms, a new one rolls.
• Tiered storage — offloading old segments to a remote store via RemoteLogManager and a pluggable RemoteStorageManager. See features/tiered-storage.md.

Protocol and clients

• API key — a numeric identifier for a request type (Produce = 0, Fetch = 1, etc.). See clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java.
• API version — every API key has a range of versions; clients negotiate via ApiVersions.
• Wire protocol — the binary framing defined by JSON schemas under clients/src/main/resources/common/message/. The generator/ module compiles them to Java classes.
• Bootstrap server — initial broker addresses a client connects to in order to discover the rest of the cluster.

Producers, consumers, and groups

• Producer ID (PID) — assigned to a producer by the transaction coordinator; combined with an epoch and sequence number to deduplicate retries (idempotent producer).
• EOS (Exactly-Once Semantics) — transactions that span produce + consume + offset commit. See features/exactly-once.md.
• Consumer group — a set of consumers that share subscription and have partitions assigned among them.
• Generation — monotonically-increasing identifier per rebalance round.
• Classic protocol vs consumer protocol (KIP-848) — the original "embedded" assignment protocol versus the newer broker-driven protocol; both implemented in group-coordinator/.
• Share group / Queue (KIP-932) — a queue-style alternative to consumer groups, coordinated by share-coordinator/.

Control plane

• KRaft — Kafka Raft, the consensus protocol that replaced ZooKeeper for metadata. See raft/.
• MetadataImage / MetadataDelta — immutable in-memory snapshot of cluster metadata, plus a delta computed by replaying records. See metadata/src/main/java/org/apache/kafka/image/.
• MetadataLoader — broker-side replayer that subscribes to the metadata log and applies images to local components.
• KIP (Kafka Improvement Proposal) — Kafka's design RFC process. Major behavior changes require a numbered KIP discussed on the dev@ list and tracked at https://cwiki.apache.org/confluence/display/KAFKA/KIPs.

Streams and Connect

• Topology — a Kafka Streams DAG of source nodes, processor nodes, and sink nodes. Built via StreamsBuilder (DSL) or Topology (PAPI).
• State store — a key-value or window/session store backing a stream processor; backed by a changelog topic for fault tolerance.
• Standby task — a hot replica of a stateful task on another instance to speed up rebalances.
• Connect worker — a JVM running connector and task instances. Standalone or distributed.
• Source / sink connector — pulls data into Kafka / pushes data out of Kafka.
• Herder — the Connect component that owns task assignment and coordinates workers.

Build and test

• Gradle module — a subproject in settings.gradle (e.g., clients, core, streams:streams-scala).
• Spotless / Checkstyle / SpotBugs — the three required code quality tools; run as part of ./gradlew check.
• Trogdor — Kafka's distributed test/fault-injection framework (trogdor/).
• Ducktape — Python framework used to drive system tests under tests/.
• JMH — JVM microbenchmark harness; benchmarks live in jmh-benchmarks/.

Acronyms in passing

• ACL — Access Control List entry. See clients/.../common/acl/.
• SASL — Simple Authentication and Security Layer. Mechanisms live in clients/.../common/security/.
• MM2 — MirrorMaker 2, the Connect-based cross-cluster replication tool in connect/mirror/.
• MM1 — original MirrorMaker tool (removed; only system-test stubs remain).