Debugging

Pointers for diagnosing problems in a running CockroachDB process and in test failures.

Logs

Every CockroachDB binary uses the structured logger in pkg/util/log/. Logs default to cockroach-data/logs/. The logger supports multiple sinks (file, stderr, OTLP, fluent, network), per-channel routing, and redaction markers.

Useful entry points:

• pkg/util/log/file.go — file rotation and gzip.
• pkg/util/log/redact.go — <redactable> markers for sensitive data.
• pkg/util/log/channels.go — channels: DEV, OPS, HEALTH, STORAGE, SQL_INTERNAL_PERF, SQL_PERF, SQL_EXEC, SESSIONS, KV_DISTRIBUTION, SQL_SCHEMA, USER_ADMIN, PRIVILEGES.
• pkg/cli/log_flags.go — log-flag parsing for cockroach start.

To debug-merge logs from a multi-node roachprod cluster: ./bin/cockroach debug merge-logs ./logs/* (pkg/cli/debug_merge_logs.go).

Tracing

pkg/util/tracing/ wraps OpenTelemetry tracing. Spans are propagated through context via tracing.SpanFromContext and emitted to the active sink (Jaeger/Zipkin/OTLP). The DB Console exposes EXPLAIN ANALYZE (DEBUG) bundles that include a Jaeger-format trace of a SQL statement (pkg/sql/explain_bundle.go).

For ad-hoc tracing of a particular SQL statement:

SET tracing = on;
SELECT ...;
SET tracing = off;
SHOW TRACE FOR SESSION;

In Go tests, use tracing.NewTracerForTest(t) and tracing.WithRecording(...).

Debug pages on running nodes

Every node serves debug routes on its HTTP listener (default port 8080):

Path	Source	Description
/_status/vars	pkg/server/status.go	Prometheus-format metrics
/_status/raft	pkg/server/status.go	Per-replica Raft state
/_status/ranges/<rangeID>	pkg/server/status.go	Range debug
/debug/pprof/...	net/http/pprof	Standard Go profiles
/debug/requests	pkg/util/tracing/debug_requests.go	net/trace HTTP probes
/debug/logspy	pkg/util/log/logspy.go	Tail logs over HTTP
/debug/zip	pkg/cli/zip*.go	Same payload as cockroach debug zip
/debug/stores/<id>	pkg/server/storage_api/	Per-store info

The Datadog dashboard JSON for cluster-wide observability is in pkg/cli/cockroachdb_datadog_dashboard.json.

cockroach debug zip

cockroach debug zip --insecure --host=node1 ./mycluster.zip

Captures cluster-wide diagnostic information: logs, status pages, range descriptors, gossip state, settings, and many internal tables. The implementation is split across pkg/cli/zip.go, pkg/cli/zip_helpers.go, pkg/cli/zip_per_node.go, pkg/cli/zip_cluster_wide.go, and pkg/cli/zip_table_registry.go.

Local single-process debugging

Run with dlv after building debug-symboled:

./dev build short -- -c dbg
dlv exec ./cockroach -- start-single-node --insecure

For stuck tests: append -- -c dbg to ./dev test to disable symbol stripping.

Pebble engine inspection

cockroach debug pebble db scan --hex /path/to/store
cockroach debug pebble manifest dump /path/to/store/MANIFEST-...
cockroach debug pebble lsm /path/to/store

Source: pkg/cli/debug.go. These pass through to Pebble's pebble.Tool.

Common failure modes

• Stale BUILD.bazel — Gazelle hasn't been run. Symptom: missing-file errors or "compile failed" with the wrong source list. Fix: ./dev gen bazel.
• Generated proto out of sync — Symptom: type mismatch between hand-written code and proto getters. Fix: ./dev gen protobuf && ./dev gen bazel.
• Replica/lease leaks in tests — Tests using testcluster should call tc.Stopper().Stop(ctx); otherwise leaktest will flag goroutines and Bazel will mark the test flaky.
• Test depends on the wall clock — Use hlc.NewClockForTesting and manual.Set(...) instead of time.Now().
• Test relies on default cluster settings — Pin them explicitly in test setup; CI runs many test configs in parallel.

Mixed-version debugging

Bugs in upgrades often only reproduce in mixed-version clusters. Tools to know:

• pkg/upgrade/ — the migration framework.
• pkg/clusterversion/ — every gated capability.
• pkg/sql/logictest/tests/local-mixed-* — logictest configs that boot a mixed-version cluster.
• pkg/cmd/cockroach-short is what most local mixed-version reproductions are built around.

Useful internal queries

SELECT * FROM crdb_internal.ranges_no_leases WHERE range_id = 42;
SELECT * FROM crdb_internal.gossip_nodes;
SELECT * FROM crdb_internal.kv_node_status;
SELECT * FROM crdb_internal.cluster_settings;
SELECT * FROM crdb_internal.jobs WHERE status='running';

crdb_internal.go (pkg/sql/crdb_internal.go) is a virtual schema that exposes most live cluster state.