Debugging

This page lists the techniques CPython core developers use day-to-day to chase crashes, leaks, and wrong-result bugs in the interpreter and its C modules.

Build debug

The first step is essentially always:

./configure --with-pydebug
make -j

A debug build:

• Defines Py_DEBUG, which enables assert statements throughout the C code.
• Defines Py_REF_DEBUG, which adds refcount sanity checks.
• Defines Py_TRACE_REFS (sometimes), which keeps a list of all live objects.
• Disables most compile-time optimisation so backtraces and source-level GDB make sense.
• Stores extra debug info on every PyObject (a 16-byte header that catches stomps).

Running ./python -X dev additionally enables development-time warnings (PYTHONDEVMODE=1).

When the interpreter crashes

Get a backtrace

gdb --args ./python -m test test_X
(gdb) run
(gdb) bt full

There's a Python-aware GDB extension at Tools/gdb/libpython.py. Either source it manually:

(gdb) source /path/to/cpython/Tools/gdb/libpython.py

…or use make gdb-test after building (it sets up auto-load-safe-path). Useful commands the extension adds:

• py-bt — print a Python-level traceback for the current frame.
• py-list — show the Python source around the current line.
• py-locals — show Python locals.
• py-up / py-down — navigate Python frames the same way up/down does C frames.

faulthandler

For crashes that don't happen in your own debugger session, enable faulthandler:

PYTHONFAULTHANDLER=1 ./python -m test test_X

It installs signal handlers that dump a Python traceback to stderr on SIGSEGV, SIGABRT, SIGBUS, SIGFPE, and SIGILL. This is on by default for python -X dev.

PYTHONUNBUFFERED=1

Always set this when you can't trust output ordering:

PYTHONUNBUFFERED=1 ./python ...

PYTHONMALLOC=malloc (with sanitizers)

pymalloc (the default small-object allocator in Objects/obmalloc.c) groups allocations into pools, which can hide bad pointers from ASan/Valgrind. To make every allocation pass through malloc():

PYTHONMALLOC=malloc ASAN_OPTIONS=detect_leaks=1 ./python ...

PYTHONMALLOC=debug adds CPython's own buffer-overrun detection without involving ASan.

When something leaks

Reference-counting bugs

CPython is dominantly refcounted; cycles are handled by the cycle GC (Garbage collector) but non-cycle leaks are usually missing Py_DECREFs. Tools:

• sys.gettotalrefcount() — debug-build only; total live refcount across all objects.
• ./python -m test -R 3:3:test_X — runs test_X 3 times to warm up, then 3 more to compare. Reports any object whose refcount changes between runs.
• gc.get_objects() — enumerate every container object the GC knows about.
• tracemalloc (Lib/tracemalloc.py, Modules/_tracemalloc.c) — Python-level allocation tracker. Enable with PYTHONTRACEMALLOC=1 or python -X tracemalloc.

Memory leaks (C-level)

Run with Valgrind or ASan and PYTHONMALLOC=malloc. CPython ships a Valgrind suppressions file at Misc/valgrind-python.supp.

PYTHONMALLOC=malloc valgrind --suppressions=Misc/valgrind-python.supp ./python -m test test_X

For ASan, the standard CI build is reproduced by .github/workflows/reusable-san.yml.

When a thread deadlocks or a test hangs

faulthandler.dump_traceback_later(timeout) will dump every thread's traceback after timeout seconds. Combined with regrtest's --timeout:

./python -m test --timeout 30 -v test_X

…you get a Python-level thread dump on hang. For the C side, gdb -p <pid> and thread apply all bt full is the workhorse.

The free-threaded build adds extra hazards (data races, missing _PyOnceFlag initialisation). Use TSan:

./configure --disable-gil --with-pydebug
TSAN_OPTIONS="suppressions=Tools/tsan/suppressions_free_threading.txt" make test

Tools/tsan/ holds the suppression files used in CI.

Wrong-result bugs at the bytecode level

When the interpreter computes the wrong answer, the cause is usually:

1. A specialized opcode in Python/specialize.c chose the wrong family — disable specialization with _testinternalcapi.disable_specialization() (debug build).
2. A uop optimization in Python/optimizer_bytecodes.c is wrong — disable with _testinternalcapi.set_optimizer(None).
3. A bug in Python/bytecodes.c — toggle the macro LLTRACE (debug build) to log every dispatch.

Useful tools:

• dis.dis(func, adaptive=True) — show the currently specialized bytecode rather than the canonical version.
• _opcode.get_specialization_stats() and the pystats build (see Python/pystats.c) — quantify specialization hits/misses.
• sys.monitoring — instrument bytecode events; see Python/instrumentation.c.

Reproducing CI failures

The most common pattern: a CI job failed but locally the test passes.

1. Check the exact configure flags — open the job's "configure" step. Many bugs are specific to --with-pydebug, --disable-gil, --enable-experimental-jit, or PGO.
2. Re-run with the same env vars: PYTHONHASHSEED=0, PYTHONIOENCODING=utf-8, etc.
3. Match the worker count: regrtest randomizes test order between workers; -j8 --randomize -randseed=12345 reproduces a specific shuffling.
4. Check resource gating: -uall is enabled in the buildbot job but not in the default make test.
5. For sanitizer runs, use Tools/tsan/, Tools/ubsan/, or run inside the GH Actions container locally.

Special debug envs and flags

Env var / flag	Effect
PYTHONDEVMODE=1 / python -X dev	Strict warnings, faulthandler, extra checks.
PYTHONMALLOC=debug	obmalloc adds canary bytes around every block.
PYTHONMALLOC=malloc	bypass obmalloc.
PYTHONFAULTHANDLER=1	dump tracebacks on fatal signals.
PYTHONTRACEMALLOC=N	record N frames per allocation.
PYTHONDUMPREFS=1	dump all live objects on shutdown (debug build, Py_TRACE_REFS).
PYTHONVERBOSE=1	Trace import machinery.
PYTHONPRINTSWITCHINTERVAL	Show GIL handoffs.
PYTHONPROFILEIMPORTTIME=1	Print per-module import timings.
python -X importtime	Same as above (newer flag).
python -X frozen_modules=off	Disable frozen-module imports — useful when you suspect the frozen importlib.

The full list is in Doc/using/cmdline.rst and printed by ./python -h.