tools

Purpose

The tools/ module contains the Java implementations of the kafka-*.sh command-line scripts shipped under bin/. The shell scripts under bin/ are thin wrappers — they simply call bin/kafka-run-class.sh <java-class> after building a classpath. This page is the bridge between "the command on the operator's terminal" and "the Java class doing the work".

tools-api/ is a small companion module that exposes a public Java SPI (PartitionReassignment, RecordReader, etc.) for tools that accept user-supplied plugins.

Directory layout

tools/
├── src/main/java/org/apache/kafka/tools/
│   ├── TopicCommand.java                ← kafka-topics.sh
│   ├── ConsoleProducer.java             ← kafka-console-producer.sh
│   ├── ConsoleConsumer.java             ← kafka-console-consumer.sh
│   ├── ConsumerGroupCommand.java        ← kafka-consumer-groups.sh
│   ├── DelegationTokenCommand.java      ← kafka-delegation-tokens.sh
│   ├── ReassignPartitionsCommand.java   ← kafka-reassign-partitions.sh
│   ├── LeaderElectionCommand.java       ← kafka-leader-election.sh
│   ├── DumpLogSegments.java             ← kafka-dump-log.sh
│   ├── GetOffsetShell.java              ← kafka-get-offsets.sh
│   ├── BrokerApiVersionsCommand.java    ← kafka-broker-api-versions.sh
│   ├── MetadataQuorumCommand.java       ← kafka-metadata-quorum.sh
│   ├── ConfigCommand.java               ← kafka-configs.sh
│   ├── AclCommand.java                  ← kafka-acls.sh
│   ├── ClientMetricsCommand.java        ← kafka-client-metrics.sh
│   ├── FeatureCommand.java              ← kafka-features.sh
│   ├── StorageTool.java                 ← kafka-storage.sh
│   ├── E2ELatencyTest.java              ← kafka-e2e-latency.sh
│   ├── perf/
│   │   ├── ProducerPerformance.java     ← kafka-producer-perf-test.sh
│   │   ├── ConsumerPerformance.java     ← kafka-consumer-perf-test.sh
│   │   └── ShareConsumerPerformance.java← kafka-share-consumer-perf-test.sh
│   ├── reassign/
│   ├── streams/                         ← kafka-streams-application-reset, kafka-streams-groups
│   ├── share/                           ← kafka-share-groups
│   ├── transactions/                    ← kafka-transactions
│   └── ...
└── tools-api/                            ← public SPI shared by tools and external plugins

How the bin scripts dispatch

$ cat bin/kafka-topics.sh
exec $(dirname $0)/kafka-run-class.sh kafka.admin.TopicCommand "$@"

kafka-run-class.sh builds a classpath from <module>/build/libs/ (or libs/ in a release), launches the JVM, and forwards arguments. The class name is the only thing that varies between scripts.

A few CLIs still point at Scala classes under kafka.tools / kafka.admin — these are the legacy entry points that have not yet been ported to tools/. Most new tools land in Java in this module.

Common patterns

• Argument parsing. Most tools use joptsimple (OptionParser / OptionSet). New tools are encouraged to migrate to Java's argparse4j style helper in org.apache.kafka.server.util.CommandLineUtils.
• Talking to the cluster. Almost every tool builds a KafkaAdminClient (org.apache.kafka.clients.admin.Admin), issues admin RPCs, and prints results. A handful (perf tests) embed KafkaProducer / KafkaConsumer directly.
• Output format. Most tools default to a human-readable table; many accept --output-format json for scripting. JSON serialization uses Jackson via clients/.../common/utils/Json.java.
• Exit codes. Tools exit non-zero on validation failure or missing topics. Scripts wrap them; CI pipelines should rely on exit codes, not on parsing stdout.

Notable tools

• kafka-topics — TopicCommand.java. Create / delete / describe / alter topics. Talks to Admin.
• kafka-storage — StorageTool.java. Formats a log directory (writes meta.properties, sets up the metadata log). Required before first KRaft start.
• kafka-metadata-quorum — MetadataQuorumCommand.java. Inspects the controller quorum (lag, current leader, voter set).
• kafka-leader-election — replaces the legacy kafka-preferred-replica-election. Triggers preferred or unclean leader election via ElectLeadersRequest.
• kafka-reassign-partitions — ReassignPartitionsCommand.java. Generates and submits partition reassignment plans.
• kafka-dump-log — DumpLogSegments.java. Decodes a partition log file (or the metadata log) into human-readable records — invaluable for debugging.
• kafka-consumer-groups — ConsumerGroupCommand.java. Describe groups, list groups, reset offsets, delete groups.
• kafka-share-groups — share/ShareGroupCommand.java. Same idea for KIP-932 share groups.
• kafka-streams-application-reset — streams/StreamsResetter.java. Wipes a Streams app's internal topics for a fresh restart.
• kafka-acls — AclCommand.java. Manage ACLs in the metadata log.
• kafka-client-metrics — ClientMetricsCommand.java. Manage broker-driven client metrics subscriptions (KIP-714).
• kafka-features — FeatureCommand.java. Inspect / upgrade metadata.version and other feature levels.

Performance test tools

tools/src/main/java/org/apache/kafka/tools/perf/ holds Kafka's own throughput / latency benchmarks: ProducerPerformance, ConsumerPerformance, ShareConsumerPerformance, EndToEndLatency. They are real, but unsophisticated — for serious benchmarking, use jmh-benchmarks/ for microbenchmarks and Trogdor workloads (trogdor/) for cluster-level scenarios.

Verifiable producer / consumer

tools/src/main/java/org/apache/kafka/tools/VerifiableProducer.java, VerifiableConsumer.java, VerifiableShareConsumer.java are minimal clients that emit machine-readable JSON describing every action. They are used by ducktape system tests under tests/ to verify ordering / EOS / rebalance correctness.

Tools that migrated

A few CLIs that used to live in Scala have moved here in recent releases:

• kafka-acls, kafka-configs, kafka-features, kafka-leader-election, kafka-broker-api-versions, kafka-storage, kafka-delegation-tokens, kafka-client-metrics, kafka-reassign-partitions are now Java.
• kafka-mirror-maker.sh (MM1) was removed; MirrorMaker 2 lives in connect/mirror/.
• kafka-preferred-replica-election.sh was removed in favor of kafka-leader-election.sh.

Entry points for modification

• Add a new CLI: write the Java class under tools/src/main/java/org/apache/kafka/tools/ (or a sub-package), add a wrapper script under bin/ that exec's kafka-run-class.sh <FQN>. Add ducktape tests if behavior is non-trivial.
• Change tool output: be conservative — many users grep tool output. JSON output via --output-format json is the right way to add new fields.
• Public SPI changes: anything in tools-api/ is a public surface and requires a KIP.

Related pages

• Modules: clients — Admin, the substrate every tool runs on.
• How to contribute: Tooling — Gradle, lint, test infra.