clickhouse/clickhouse

Debugging

The most useful debugging tools live inside ClickHouse itself: rich logs, a thousand system.* tables, and built-in profilers.

Logs

The server writes to /var/log/clickhouse-server/clickhouse-server.{log,err.log} by default. The format is configured in programs/server/config.xml. Log levels: trace, debug, information, warning, error, none. Settings:

<logger><level> — global level.
<text_log> — captures structured logs into a system.text_log table for SQL querying.
<query_log> — every query and its metrics in system.query_log.
<query_thread_log> — per-thread query metrics.
<part_log> — every MergeTree part operation.
<crash_log> — exceptions with stack traces.
<asynchronous_metric_log> / <metric_log> — periodic metric snapshots.

These tables are MergeTree-backed and queryable like any other:

SELECT type, query_duration_ms, read_rows, exception
FROM system.query_log
WHERE event_date = today() AND type != 'QueryStart'
ORDER BY event_time DESC
LIMIT 50;

The runtime metrics underlying them are emitted from src/Common/CurrentMetrics.cpp, ProfileEvents.cpp, and MemoryTracker.cpp.

Common error reports

Logical error — invariant violation. The release build does not crash; it raises an exception. Per AGENTS.md, refer to these as "exceptions" rather than "crashes" in commit messages and reports.
Allocator failures — MemoryTracker emits OOM messages with a per-query hierarchy and a global ceiling.
ZooKeeper / Keeper exceptions — replication can pause if the coordinator is unreachable. system.zookeeper_log is invaluable.

Stack traces

programs/main.cpp builds the binary with frame pointers and updates the phdr_cache so signal handlers can produce reliable traces. src/Common/StackTrace.cpp and src/Common/Exception.cpp symbolize them. In ASan/MSan/TSan builds the sanitizer's own trace is included.

To turn a non-symbolised trace into source locations:

addr2line -e build/programs/clickhouse 0x...   # offline

Or use the system.stack_trace table on a live server:

SELECT thread_id, query_id, arrayStringConcat(arrayMap(x -> demangle(addressToSymbol(x)), trace), '\n')
FROM system.stack_trace LIMIT 10;

Sanitizers

CI runs every PR through ASan, TSan, MSan, UBSan. To reproduce locally:

mkdir build_asan && cd build_asan
cmake -DSANITIZE=address ..
ninja clickhouse
build_asan/programs/clickhouse server --config-file=programs/server/config.xml

Per AGENTS.md, multiple build directories (build, build_debug, build_asan, etc.) live side-by-side. Tests can be run against a sanitizer build by pointing the driver at the right binary.

Fuzzers

To run a libFuzzer harness:

ninja parser_fuzzer        # (or any other fuzzer target)
build/src/.../parser_fuzzer corpus_dir/

Harnesses live under src/**/fuzzers/, programs/server/fuzzers/. A failing fuzzer typically writes a reproducer file; copy it into a stateless test as a regression.

Profiling

Built-in profilers:

Query profiler — set query_profiler_real_time_period_ns / query_profiler_cpu_time_period_ns per query. Samples land in system.trace_log.
EXPLAIN and EXPLAIN PIPELINE — show the logical plan and physical processor graph.
system.processors_profile_log — execution time per processor in a query.
OPTIMIZE TABLE ... FINAL / system.merges — diagnose merge backlogs.

Off-CPU profiling can use perf (compatible because of frame pointers and phdr_cache).

For micro-questions about types, alignment, or codegen, use the helper scripts under .claude/tools/ and utils/c++expr (see Tooling).

Common debugging recipes

Why is my query slow? EXPLAIN PIPELINE, system.query_log, system.processors_profile_log, system.trace_log.
Why is a part not merging? system.merges, system.part_log, system.replication_queue, SYSTEM SYNC REPLICA, system.merge_tree_settings for the relevant pool sizes.
Why is a replica falling behind? system.replication_queue, system.replicas, system.zookeeper_log.
Why is the server using so much memory? system.metrics, system.asynchronous_metrics, system.memory_usage (the *Profiler*Log tables on more recent versions), system.processes.
Why did BACKUP fail? system.backups, system.backup_log, the per-backup log file under <backup_dir>/.clickhouse_backup/.