Development workflow

Day-to-day commands for hacking on Kafka. The full set of build switches lives in README.md; this page surfaces the ones a contributor uses every day.

Branching

• trunk is the default branch and the integration target for all PRs.
• Release branches (3.7, 3.8, 4.0, ...) only receive cherry-picked fixes.
• Topic branches in your fork should be short-lived and named after the JIRA, e.g. KAFKA-12345-fix-offset-reset.

Initial setup

git clone https://github.com/<your-fork>/kafka.git
cd kafka
git remote add upstream https://github.com/apache/kafka.git
git fetch upstream
git checkout -b KAFKA-XXXXX-short-description upstream/trunk

Build commands

./gradlew jar                                # build all JARs
./gradlew clients:jar                        # one module
./gradlew compileJava                        # only compile (faster than full jar)
./gradlew clean                              # blow away build outputs
./gradlew testJar                            # build test JARs (used by other modules)
./gradlew releaseTarGz                       # full release tarball under core/build/distributions

Generating RPC stubs after schema changes

Edit a JSON file under clients/src/main/resources/common/message/, then:

./gradlew processMessages processTestMessages

Schema documentation lives in clients/src/main/resources/common/message/README.md. Schema changes almost always need a KIP since they alter the wire protocol.

Iterating quickly

For most changes you can iterate with a single module's tests:

./gradlew clients:unitTest
./gradlew clients:test --tests org.apache.kafka.clients.MetadataTest
./gradlew clients:test --tests "*Metadata*" --info

--rerun re-runs even if Gradle thinks nothing changed; --info and --debug raise log verbosity.

Running a local broker against your build

The shell scripts under bin/ resolve classpath via kafka-run-class.sh, which picks up your local <module>/build/libs/. After a ./gradlew jar:

KAFKA_CLUSTER_ID="$(./bin/kafka-storage.sh random-uuid)"
./bin/kafka-storage.sh format --standalone -t "$KAFKA_CLUSTER_ID" -c config/server.properties
./bin/kafka-server-start.sh config/server.properties

Point a console producer / consumer at localhost:9092 to exercise the broker.

Code style auto-fix

Spotless enforces import order. To fix imports automatically:

./gradlew spotlessApply

For everything else (line length, naming), Checkstyle will tell you what to fix during ./gradlew check.

Submitting a PR

git push origin KAFKA-XXXXX-short-description

Open the PR against apache/kafka:trunk. The title must start with the JIRA key. The description should:

• Link to the JIRA.
• Link to the KIP (if any).
• Summarize what changed and why.
• Note any compatibility implications.
• List the tests added or updated.

CI runs the full Gradle build plus a sample of integration tests. A committer triages PRs after CI is green.

Cherry-picking to a release branch

Committers cherry-pick fixes to release branches:

git checkout 4.0
git cherry-pick -x <commit-on-trunk>
git push apache 4.0

Contributors usually do not need to do this themselves; ask in the PR or on dev@ if backport is required.

Working with the Streams quickstart archetype

The Streams archetype lives under streams/quickstart/ and uses Maven (not Gradle) for deployment:

cd streams/quickstart
mvn deploy

This is only needed for releases.

Useful Gradle properties

These are passed via -P<name>=<value>:

Property	Effect
maxParallelForks	Test JVMs to run in parallel (default = number of CPUs)
maxTestRetries	Retry failing tests up to N times
kafka.test.run.flaky=true	Also run tests marked @Flaky
kafka.test.run.new=true	Run tests tagged for the next release
enableTestCoverage=true	Bytecode-instrument for JaCoCo / Scoverage; used with reportCoverage
xmlSpotBugsReport=true	XML SpotBugs reports instead of HTML
skipSigning=true	Don't sign artifacts when running publishToMavenLocal