Tooling

The build, code-quality, and test infrastructure that runs on every Kafka PR.

Gradle

The single source of truth is Gradle:

• Root build file: build.gradle (~3,300 lines).
• Module list: settings.gradle.
• Dependency catalog: gradle/dependencies.gradle.
• Wrapper: gradle/wrapper/. Always run via ./gradlew, not a system Gradle.
• Develocity build scans are enabled and uploaded to https://develocity.apache.org/ for triage.

The build is multi-module: each subproject has its own build.gradle snippet generated from the root. Most contributors only need a few task names:

Task	Purpose
jar	Build all module JARs
test / unitTest / integrationTest	Run tests at the matching tier
check	Lint + style + tests for all modules
processMessages / processTestMessages	Generate request/response classes from schemas
releaseTarGz	Build a release tarball
aggregatedJavadoc	Combined Javadoc for all modules
clean	Remove build/ for all modules
spotlessApply	Auto-fix import order
dependencyUpdates	Report outdated dependencies (uses the Ben Manes plugin)

RPC code generator

Module generator/ compiles JSON schemas under clients/src/main/resources/common/message/ into Java request/response classes during the processMessages task. Each schema declares the API key, supported versions, and the fields per version. The generator adds read/write/size/equality methods and is executed automatically by the build.

The list of all RPCs is in ApiMessageType.json and is mirrored by ApiKeys.java. New RPCs are introduced via a KIP.

Checkstyle

Configuration: checkstyle/. The rules are conservative — the most common violations are import order (handled by Spotless), 120-column lines, and unused imports.

./gradlew checkstyleMain checkstyleTest

Reports: <module>/build/reports/checkstyle/.

Spotless

Spotless enforces import order across Java sources.

./gradlew spotlessCheck    # report violations
./gradlew spotlessApply    # auto-fix

The Spotless config is in build.gradle; rules apply to all Java modules.

SpotBugs

./gradlew spotbugsMain spotbugsTest -x test

Reports: <module>/build/reports/spotbugs/. The exclude list is in build.gradle per module — keep it small. Use -PxmlSpotBugsReport=true for XML output.

Test coverage

./gradlew reportCoverage -PenableTestCoverage=true -Dorg.gradle.parallel=false

Java modules use JaCoCo (HTML at <module>/build/reports/jacoco/test/html/); the Scala core module uses Scoverage (core/build/reports/scoverageTest/).

Test retry plugin

The Gradle Test Retry plugin is wired up so that -PmaxTestRetries=N -PmaxTestRetryFailures=M will retry failing tests up to N times with at most M total failures before giving up. Used for known-flaky integration tests.

Build cache

buildCache {
    local { enabled = true }
    remote(develocity.buildCache) { enabled = false }
}

Local Gradle build cache is on by default; the remote (Develocity) cache is off in this repo. CI on GitHub Actions reuses the local cache because each runner is a fresh VM.

CI

The CI workflows live under .github/workflows/:

• ci.yml — main per-PR pipeline (build, lint, unit + integration tests).
• build.yml — invoked by ci.yml for the matrixed JDK builds.
• generate-reports.yml — daily Flaky Test Report.
• A handful of helpers for stale issues, release artifacts, etc.

Every push to a PR re-runs the matrix; results show up under the GitHub "Checks" tab.

Docs

The user-facing docs live under docs/. They are a combination of HTML, Velocity-templated text, and configuration tables generated from ConfigDef data. Documentation for new configurations is generated automatically; protocol/API docs need an HTML edit.

Docker

docker/ contains Apache Kafka's official Docker images and Compose files used both in production releases and for local testing. docker/README.md has the build instructions.

Vagrant

Vagrantfile and vagrant/ boot a multi-VM dev environment used by some system tests. Most contributors don't need this; ducktape's Docker driver (tests/docker/) is the easier path.

Trogdor

trogdor/ is the in-repo distributed test harness. It runs agents, accepts fault-injection workloads over REST, and reports results back to a coordinator. Used as the runtime for long-running ducktape system tests.

JMH

jmh-benchmarks/ is a self-contained module. ./gradlew jmh-benchmarks:shadowJar builds an executable JAR. Benchmarks under jmh-benchmarks/src/main/java/org/apache/kafka/jmh/ run via standard JMH command-line flags.

Committer tools

committer-tools/ contains scripts that ASF Kafka committers use to merge PRs and tag releases — merge_kafka_pr.py is the main one. Contributors don't need these unless they are committers.

Release scripts

release/ holds the orchestration for cutting a release; the entry point is release.py. Driven by Apache release managers, not contributors.