Debugging

A short toolkit for the most common diagnosis tasks.

Eager / autograd

Symptom	Try
Wrong gradient	torch.autograd.gradcheck(fn, (x,)); check for in-place ops
"modified by an inplace operation"	A saved tensor was mutated; find the culprit with torch.autograd.detect_anomaly
NaN / Inf in backward	with torch.autograd.detect_anomaly():
Want to inspect autograd graph	make_dot(loss) from torchviz, or walk .grad_fn recursively
Slow autograd	torch.profiler.profile(with_stack=True)

detect_anomaly is expensive but invaluable: it stores Python tracebacks at each Node creation and re-raises them when backward errors.

CUDA

Symptom	Try
OOM	torch.cuda.memory._record_memory_history() then visualize at memory_viz
cuBLAS error 13, cuDNN error N	Set CUBLAS_WORKSPACE_CONFIG=:16:8, torch.backends.cudnn.benchmark = True
Wrong stream sync	CUDA_LAUNCH_BLOCKING=1 to serialize for clearer errors
Mysterious unknown error	CUDA_LAUNCH_BLOCKING=1 + cuda-memcheck to find the original kernel
Want a deterministic run	torch.use_deterministic_algorithms(True) (slower, may raise on unsupported ops)
Suspect allocator fragmentation	PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

Compile (torch.compile)

Symptom	Try
Slow first call	Expected; compile takes time. Check the cache is enabled
Constant recompiles	TORCH_LOGS=recompiles shows the reason; consider dynamic=True
Graph break in a hot path	TORCH_LOGS=graph_breaks; refactor the offending Python
Wrong output	Compare with backend="aot_eager", then eager, to bisect the culprit
Compile crash	torch._dynamo.repro.after_aot produces a self-contained reproducer
Unsupported op	Check the lowering in torch/_inductor/lowering.py; add a decomposition

Distributed

Symptom	Try
Hang	TORCH_NCCL_TRACE_BUFFER_SIZE=2000000, TORCH_NCCL_DUMP_ON_TIMEOUT=1
Mismatched collective	TORCH_NCCL_DESYNC_DEBUG=1
Slow all-reduce	NCCL profiler, or record_function around each step
OOM only on one rank	TORCH_DISTRIBUTED_DEBUG=DETAIL

C++ / GDB

For native crashes:

gdb --args python my_repro.py
(gdb) catch throw
(gdb) run

The repo ships .gdbinit and .lldbinit with PyTorch-specific pretty printers for at::Tensor, c10::IValue, and friends.

Logging

TORCH_LOGS=... (comma-separated) enables structured logs for the compile stack and a few other systems:

TORCH_LOGS="dynamo,recompiles,graph_breaks"
TORCH_LOGS="aot_graphs,aot_joint_graph"
TORCH_LOGS="inductor,output_code"
TORCH_LOGS="+dtensor"        # leading + means DEBUG level
TORCH_LOGS="+all"            # everything (very verbose)
TORCH_LOGS="-dynamo"         # leading - silences a logger

TORCH_COMPILE_DEBUG=1 dumps every IR / scheduler / kernel artefact to torch_compile_debug/.

Production tracing

Per CLAUDE.md's "Logging and Structured Tracing" section, code that wants production-friendly diagnostics should use torch._logging.trace_structured:

from torch._logging import trace_structured

trace_structured(
    "artifact",
    metadata_fn=lambda: {"name": "my_debug_artifact", "encoding": "string"},
    payload_fn=lambda: payload,
)

These artefacts are written to the structured-trace file (controlled by TORCH_TRACE) and decoded offline by tlparse.

Where to look

Tool	Purpose
torch.autograd.detect_anomaly	Anomaly-mode autograd
torch.cuda.memory._record_memory_history	Memory profiler
torch._dynamo.repro.after_aot / after_dynamo	Compile bug repros
torch.distributed.flight_recorder	NCCL flight recorder
tlparse	Structured trace parser
.gdbinit, .lldbinit	Native debugger pretty printers