Debugging

Pointers for tracking down failures, both in your local development cycle and when reading bug reports against a real cluster.

Logging

• The broker's runtime logger is Log4j 2. Configuration: config/log4j2.yaml.
• The CLI tools use a separate config: config/tools-log4j2.yaml.
• Connect uses config/connect-log4j2.yaml.
• Test logging is configured per-module under <module>/src/test/resources/log4j2.yaml. To raise the log level for a specific module's tests, edit that file (e.g., set level: INFO) and re-run with --rerun.

Logger names follow the package structure (org.apache.kafka.clients.consumer.internals.Fetcher, kafka.server.ReplicaManager, etc.), so you can scope verbosity tightly.

Common broker failure modes

Symptom	Where to look
Broker fails to start with format/cluster ID mismatch	kafka-storage.sh format was not run, or meta.properties is stale.
Producer/consumer fails with NOT_LEADER_OR_FOLLOWER	ReplicaManager is rejecting because of a recent leader change. Look at KafkaApis.handleProduceRequest / KafkaApis.handleFetchRequest.
Long tail in produce latency	ProducerStateManager snapshotting, log roll, or a slow follower in the ISR.
Consumer rebalance never completes	GroupCoordinator logs in group-coordinator/.../GroupMetadataManager. The new (KIP-848) protocol logs every state transition.
Topic create/delete hangs	Check the controller's QuorumController log; metadata records may not be replicating.
OffsetCommit fails with INVALID_GROUP_ID or UNKNOWN_MEMBER_ID	The consumer's group state was reset; usually a heartbeat timeout or an incompatible upgrade.
OutOfOrderSequenceException	Idempotent producer's PID was rotated; usually a transaction abort or a session expiration.

Useful broker introspection

• kafka-broker-api-versions.sh --bootstrap-server <host>:9092 — what RPCs the broker supports and at which versions.
• kafka-metadata-shell.sh --snapshot <log-dir>/__cluster_metadata-0/... — interactive view of the metadata log; lives in shell/.
• kafka-dump-log.sh --files <segment.log> --print-data-log — decodes a partition log segment.
• kafka-cluster.sh describe / kafka-metadata-quorum.sh — KRaft quorum status.
• JMX — every broker subsystem exposes JMX metrics; kafka-jmx.sh connects to a local broker. Names follow kafka.<group>:type=<subsystem>,name=<metric>.

Debugging from an IDE

For client-side issues, the easiest path is to write a unit test that reproduces the problem and run it under the IDE debugger. For broker-side issues:

1. Build with debug info (the default).
2. Start the broker with KAFKA_DEBUG=y ./bin/kafka-server-start.sh ... (this opens JDWP on port 5005; configurable via JAVA_DEBUG_PORT). See bin/kafka-run-class.sh.
3. Attach IntelliJ's "Remote JVM Debug" run configuration to localhost:5005.
4. Set breakpoints; the broker will block until the debugger hits them.

Reproducing flaky tests

N=500; I=0; while [ $I -lt $N ] && \
  ./gradlew clients:test --tests <FQN.method> --rerun --fail-fast; \
  do (( I=$I+1 )); echo "Run $I OK"; done

Run with -i (Gradle info) and crank the relevant log4j2.yaml to DEBUG. Most flakiness in Kafka is timing-related: tests that wait for a state transition with too short a timeout, or that race against the broker's request handler. Look for TestUtils.waitForCondition in the failing test — extending the timeout is rarely the right fix; usually the test is missing a synchronization point.

Reading thread dumps

A few thread names worth knowing:

Pattern	Component
kafka-network-thread-*	Processor threads in core/.../network/SocketServer
data-plane-kafka-request-handler-*	KafkaRequestHandler threads handling Produce/Fetch
control-plane-kafka-request-handler-*	Dedicated handler pool for controller-side requests
ReplicaFetcherThread-*	Followers fetching from leaders (core/.../server/ReplicaFetcherThread)
kafka-raft-io-thread	The Raft client's single-threaded IO loop (raft/.../KafkaRaftClient)
metadata-loader-event-handler	MetadataLoader applying records to local state
producer-network-thread | <client>	The Sender thread inside a KafkaProducer
kafka-coordinator-heartbeat-thread	Consumer heartbeat thread (classic protocol)
kafka-streams-...-thread-*	Stream threads in a Streams app

When things go really wrong

• Broker won't start, log dirs corrupted — kafka-log-dirs.sh --describe, then in the worst case kafka-storage.sh format --ignore-formatted after stopping the broker. Don't do this on a production node without understanding the consequences.
• Controller quorum is unhealthy — kafka-metadata-quorum.sh --bootstrap-server <broker> describe --status. Recovery procedures live in the operator docs under docs/ops.html.
• Hot-loop on test failure — capture a jstack of the test JVM (Gradle exposes build/test-results/...); compare to the thread-name table above.