Debugging

A debugging tour of the kernel from a developer's seat. The toolbox is large; pick the right tool for the failure mode.

Logs and printk

• printk(KERN_INFO "...") and the convenience macros pr_info(), pr_warn(), pr_err(), dev_info() are the day-to-day logging API.
• Output goes to the kernel log buffer and is read with dmesg or journalctl -k.
• Levels: KERN_EMERG (0) through KERN_DEBUG (7). Filtered by loglevel= boot parameter and console_loglevel.
• pr_debug() / dev_dbg() are compiled in only when DEBUG is defined or CONFIG_DYNAMIC_DEBUG is set.

Dynamic debug:

echo 'file mm/slub.c +p' > /sys/kernel/debug/dynamic_debug/control
echo 'module xfs +p' > /sys/kernel/debug/dynamic_debug/control

This turns on pr_debug calls at runtime without rebuilding.

ftrace

Function tracer in kernel/trace/. Mounted at /sys/kernel/tracing/.

cd /sys/kernel/tracing/
echo function > current_tracer
echo 'do_sys_open' > set_ftrace_filter
echo 1 > tracing_on
cat trace_pipe

Or use trace-cmd (in tools/) which provides a nicer CLI.

Other tracers: function_graph (call/return depth), wakeup, wakeup_rt, irqsoff, preemptoff, mmiotrace.

Tracepoints

Static instrumentation points declared with TRACE_EVENT() macros. See include/trace/events/. Examples: sched_switch, kmem_cache_alloc, block_rq_issue. Enable via /sys/kernel/tracing/events/.

kprobes / uprobes

Dynamic instrumentation points. kprobes attach to any kernel address; uprobes to user-space addresses. The modern way to use them is via BPF (bpftrace, bcc).

perf

User-space tool in tools/perf/. Wraps the perf_events kernel API.

perf top                    # CPU sampling
perf record -a sleep 10     # All-CPU sampling
perf report
perf stat -e cycles,instructions ./workload
perf trace -e syscalls

kgdb / kdb

For source-level debugging of the running kernel.

• kgdb — remote gdb attached over a serial line.
• kdb — built-in shell, summon with sysrq+g or panic.

Set up kgdboc=ttyS0,115200 on the kernel command line and use target remote /dev/ttyS0 from gdb on a host.

SysRq

Magic SysRq key (/proc/sysrq-trigger or Alt-SysRq-). Useful keys:

• t — show all task states / backtraces.
• w — show tasks in uninterruptible sleep.
• m — dump memory info.
• c — trigger a crash (for testing kdump).
• b — immediate reboot.

Sanitizer output

When KASAN, UBSAN, or KCSAN trips, you get a bug report with backtraces and (for KASAN) a memory map of the offending region. The WARN_ON() / BUG_ON() family produces similar output. All of these are dumped to the kernel log with a unique header line containing the subsystem and the failing condition.

A typical KASAN report includes:

• Type of error (use-after-free, out-of-bounds, etc.)
• Stack trace where the access happened
• Allocation and free stacks for the object
• Memory state in the surrounding bytes

crash / kdump

For post-mortem analysis of a panic'd kernel. Configure kexec to load a "crash kernel" that kicks in on panic and dumps the previous kernel's memory to disk. Analyze with crash or gdb. See Documentation/admin-guide/kdump/.

lockdep

CONFIG_LOCKDEP=y enables the lock dependency checker. It catches:

• Lock ordering violations (A then B vs. B then A in different paths)
• Recursive lock takes
• IRQ-unsafe lock taken in IRQ context
• Forgotten lock release on error path

Lockdep reports look like multi-page tree printouts. Read carefully; the format is documented in Documentation/locking/lockdep-design.rst.

Common failure modes and where to look

Symptom	Likely tool
Soft lockup / hang	sysrq-t, lockdep, ftrace
Memory corruption	KASAN, slub_debug, page_owner
Slow path / regression	perf, ftrace function_graph
Driver probe failure	dmesg, dynamic_debug, dev_dbg
BPF program rejected	bpftool prog load then look at the verifier log
OOM kill	dmesg, memcg counters, oom_score_adj
Suspend/resume failure	pm_test, dmesg, pm_print_times
Filesystem inconsistency	fsck.<fs>, btrfs check, xfs_repair

Documentation

• Documentation/admin-guide/bug-hunting.rst — the official intro.
• Documentation/dev-tools/ — KASAN, UBSAN, KCSAN, kunit, gdb scripts, etc.
• Documentation/trace/ — every tracer in detail.

Related pages

• Testing — how to run the test suites that produce these reports.
• Tooling — checkpatch, sparse, smatch.