Patterns and conventions

Conventions a reader will trip over when first reading the Kafka codebase. Most are enforced by Checkstyle, SpotBugs, or Spotless; some are folklore that PR review will surface.

Two languages, one project

• Java is the primary language and is required for any new public API.
• Scala is concentrated in core/ (the broker request layer, replication, log management) and a thin streams/streams-scala/ wrapper. New Scala code is generally only added in core/; new modules are written in Java.
• The build sets release to Java 11 for clients and streams so that they remain consumable on Java 11 runtimes; everything else compiles for Java 17.
• Scala 2.13 is the only supported Scala version.

Public API rules

• Anything outside an internals/ package is part of the public surface. Changing or removing it requires a KIP.
• All public types must have Javadoc. Public methods must document parameters, return values, and exceptions.
• Public classes default to final unless they're explicitly designed to be subclassed.
• @InterfaceStability annotations (Stable, Evolving, Unstable) live in clients/src/main/java/org/apache/kafka/common/annotation/InterfaceStability.java and are the source of truth for binary-compatibility expectations.

internals packages

Almost every public package has a sibling internals/ package containing implementation classes. Anything in internals/ may be changed without a KIP, and clients must not import from it. Examples:

• clients/.../producer/internals/Sender.java — the producer's network thread; the public KafkaProducer orchestrates it.
• clients/.../consumer/internals/Fetcher.java, ConsumerNetworkThread.java, MembershipManager.java.
• streams/.../streams/internals/, streams/.../streams/processor/internals/.

When you find yourself reaching into internals from outside the module, that's usually a sign that the public surface needs to grow (a KIP) or that the call site belongs in the same module.

RPC schemas, not handcrafted serialization

Every request/response is described by a JSON schema in clients/src/main/resources/common/message/. The generator/ module compiles these into Java classes during processMessages. Treat this as the single source of truth:

• Adding a new RPC = new *.json schema + KIP. Don't write the request/response by hand.
• Adding a field to an existing RPC = bump the message version in the schema; schema must declare which versions support the field.
• The corresponding ApiKeys enum value lives in clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java; ApiMessageType.json is the master list.

Configuration

• Each subsystem owns a *Config class (KafkaConfig, ProducerConfig, ConsumerConfig, StreamsConfig, ConnectConfig, KRaftConfigs, etc.) that declares all options via ConfigDef (clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java).
• Each option must have a name, type, default, importance, and description string. Descriptions become user-facing docs.
• Adding/renaming/removing a configuration requires a KIP and a release-notes entry.
• For broker configs, dynamic (cluster-level / broker-level) overrides go through DynamicBrokerConfig (core/src/main/scala/kafka/server/DynamicBrokerConfig.scala).

Error handling

• Public exceptions live in clients/src/main/java/org/apache/kafka/common/errors/ and are typed by the Errors enum (clients/.../common/protocol/Errors.java), which maps numeric error codes to exception classes. Adding an Errors value requires a KIP.
• Retryable vs non-retryable is part of the type — RetriableException subclasses are retried automatically by clients.
• KafkaException is the unchecked supertype; use it sparingly, and prefer the most specific subclass.
• Don't swallow InterruptedException — re-set the interrupt flag and propagate.

Threading and concurrency

• The producer is built around two threads: the application thread that calls send() and the Sender thread that drives I/O. They communicate via a lock-free RecordAccumulator and a Deque per partition.
• The consumer's new (KIP-848) implementation uses a single ConsumerNetworkThread plus the application thread; the legacy implementation has a heartbeat thread additionally.
• The broker's KafkaApis is invoked on the request handler pool (KafkaRequestHandler); long-running work (e.g., delayed fetches, fetch sessions) is offloaded to purgatories and a background kafka-scheduler pool.
• Use the project's Time interface (clients/.../common/utils/Time.java) instead of System.currentTimeMillis() — it's mockable in tests via MockTime.
• Use the project's KafkaThread factory (clients/.../common/utils/KafkaThread.java) instead of raw new Thread(); it sets daemon and uncaught-exception-handler defaults.

Coding style

• Two-space indentation in Scala; four-space in Java (Checkstyle enforces).
• 120-column line length.
• Imports must be ordered exactly as Spotless wants — ./gradlew spotlessApply will fix them.
• Don't use var for class-level state; prefer final and explicit setters.
• Don't introduce wildcard imports.
• Don't suppress Checkstyle/SpotBugs warnings without an inline comment justifying it.

Testing conventions

See Testing. The shorthand:

• JUnit 5; Mockito for mocks; no PowerMock.
• MockTime, never Thread.sleep, in unit tests.
• @Tag("integration") for tests that boot a broker; everything else is unit.
• TestUtils.waitForCondition for asynchronous assertions; never busy-spin.

Logging

• Use SLF4J (org.slf4j.Logger); do not import Log4j directly.
• Pre-compute message strings only when the level is enabled or use the {} placeholder syntax — Kafka has been bitten by String.format in tight loops.
• For frequent debug output, gate with if (log.isDebugEnabled()) if formatting is non-trivial.
• Don't log at INFO from per-record code paths; reserve INFO for lifecycle events (startup, shutdown, partition state changes, rebalance results).

Backward compatibility

• All public APIs must remain source- and binary-compatible across minor releases.
• Wire protocol versioning is part of every RPC schema; never remove a version still supported by a current client.
• On-disk formats are versioned via RecordVersion (clients) and MetadataVersion (controller); upgrades go through explicit version transitions, never silent format changes.

Pull request hygiene

• One JIRA per PR.
• Title starts with the JIRA key.
• Squash commits in your branch before review (the merger will squash again at merge time, but reviewers benefit from a clean history).
• Don't merge unrelated refactoring into a feature PR; open a separate MINOR: PR for it.
• Update docs/ for any user-visible change.