Other modules

A few smaller modules don't warrant a dedicated page but are worth knowing about. They live alongside the bigger modules under the repo root.

shell/ — interactive metadata shell

shell/ provides kafka-metadata-shell.sh, an interactive REPL that mounts a metadata log (__cluster_metadata-0 directory or a snapshot file) as a virtual file system you can cd and ls through. It is the easiest way to inspect what the controller actually wrote when investigating a metadata bug.

shell/src/main/java/org/apache/kafka/shell/
├── MetadataShell.java         ← entry point
├── command/                   ← cd, ls, cat, find, exit, ...
├── glob/                      ← path globbing
└── state/                     ← in-memory tree built from metadata records

It depends on metadata/ for the record schemas and the MetadataLoader machinery.

trogdor/ — distributed test framework

trogdor/ is Kafka's own multi-host test/fault-injection harness. A coordinator process accepts workload specifications via REST; agents on each host execute them. It powers long-running ducktape system tests but is also useful standalone for benchmarking with controlled load.

trogdor/src/main/java/org/apache/kafka/trogdor/
├── coordinator/               ← Coordinator daemon (REST + scheduler)
├── agent/                      ← Agent daemon (executes tasks)
├── task/                       ← TaskController, TaskWorker SPIs
├── workload/                   ← ProduceBench, ConsumeBench, ConnectionStressSpec, ...
├── fault/                      ← NetworkPartitionFault, ProcessStopFault, ...
├── basic/                      ← simple HTTP scaffolding
└── rest/

bin/trogdor.sh launches both coordinator and agent. See trogdor/README.md for the workload spec language.

generator/ — RPC schema → Java

generator/ is a one-off build-time tool that compiles JSON message schemas into Java request/response classes. It runs as part of the processMessages Gradle task and produces sources under <module>/src/generated/java/.

generator/src/main/java/org/apache/kafka/message/
├── MessageGenerator.java
├── ApiMessageType.java         (template that becomes the master ApiMessageType.java in clients)
├── MessageDataGenerator.java
└── ...

Adding a new RPC means adding a JSON schema, not editing this module — but if you're changing schema syntax (e.g., adding a new field type), generator/ is where you'd do it. KIP required.

jmh-benchmarks/ — microbenchmarks

JMH harness benchmarks for hot paths:

jmh-benchmarks/src/main/java/org/apache/kafka/jmh/
├── consumer/                  ← consumer-side fetch-path benchmarks
├── producer/
├── record/                    ← compression, RecordBatch decode
├── server/                    ← broker-side benchmarks (KafkaApis, log append)
├── streams/
├── partitioner/
├── network/
└── ...

Build a fat JAR via ./gradlew jmh-benchmarks:shadowJar and run with standard JMH command-line options. See jmh-benchmarks/README.md.

test-common/ — shared test utilities

test-common/ is split into three submodules that don't ship in releases but are dependencies of every other module's testRuntime:

• test-common-internal-api/ — base classes and annotations shared by JUnit tests.
• test-common-util/ — common assertions, builders, helpers.
• test-common-runtime/ — JUnit 5 extension that wires up KafkaClusterTestKit.

test-common/test-common-runtime/src/main/java/org/apache/kafka/common/test/api/ defines annotations like @ClusterTest and @ClusterTestExtensions that drive parameterized integration tests against multiple cluster topologies (single-broker, multi-broker, multi-controller, etc.).

coordinator-common/

Already covered under Modules: coordinators — the shared replicated-state-machine runtime that the group, share, and transaction coordinators sit on top of.

streams/upgrade-system-tests-*/

Each of these modules pins an older Streams release (0.11, 1.0, ..., 4.0, 4.1) and is loaded by the ducktape upgrade tests so that a current trunk build can be tested against a topology produced by an older binary. They are not consumed at runtime by users.

Related pages

• Modules: index — the bigger modules.
• How to contribute: Testing — when to use each test infra.
• How to contribute: Tooling — JMH usage.