Pitfalls

A compendium of things that have surprised contributors. None of them are bugs; all of them are consequences of design decisions that aren't obvious until you trip over them.

Eager / autograd

Saved tensors are checked for in-place modification

If you do x.add_(y) after x was used in a forward that's being differentiated, autograd may raise:

RuntimeError: one of the variables needed for gradient computation has been
modified by an inplace operation

The version counter on the storage is bumped on every in-place op; saved tensors record their version at save time and compare on backward. This is a feature.

requires_grad=True is sticky on outputs

y = x.detach() returns a tensor with requires_grad=False. But y = x + 0 returns a tensor with requires_grad=True (because x did). To explicitly opt out, use .detach().

torch.no_grad() doesn't disable autograd-key registration

Inside with torch.no_grad():, the autograd key is excluded from the dispatcher's keyset, so backward graph isn't recorded. But the ops themselves still go through the autograd key's fallthrough — the perf cost isn't zero.

model.eval() does not disable gradients

It only flips training=False (turns off dropout, switches BN to use running stats). Pair with torch.no_grad() for inference.

Tensor shapes / strides

view vs reshape

view requires contiguity-compatible shapes; if not compatible, it raises. reshape is "view if possible, copy otherwise". For a function that should never silently copy, use view.

as_strided doesn't validate

as_strided(out_of_bounds_strides) will happily produce a tensor that, when read, segfaults. PyTorch trusts you. Use sparingly.

Memory-format ≠ shape

A 4D tensor can be memory_format=Contiguous (NCHW) or memory_format=ChannelsLast (NHWC) with the same sizes. .contiguous() without an argument makes it Contiguous-format. .contiguous(memory_format=torch.channels_last) makes it ChannelsLast.

CUDA

CUDA is async; errors are async

A CUDA error: invalid configuration argument may surface several ops after the actual culprit. Set CUDA_LAUNCH_BLOCKING=1 to serialize and locate the original error.

tensor.cuda() is async

The H2D copy may not have completed by the time tensor.cuda() returns. Subsequent CUDA ops on the same stream see the data; CPU code that expects to inspect it must torch.cuda.synchronize().

torch.cuda.empty_cache() doesn't help training memory

It frees blocks back to the CUDA driver but does not reduce peak memory. Useful only for "I want to know how much I'm actually using right now" debugging.

Streams aren't free across processes

CUDA IPC works (used by DataLoader workers and torch.multiprocessing), but stream synchronization across processes requires explicit events and adds latency. Most users should let the framework handle this rather than building custom multi-process CUDA pipelines.

DataLoader

num_workers=0 is the default

That's a single-process loader. For non-trivial datasets, set num_workers >= 4.

Forking with CUDA contexts is broken

If you've already called torch.cuda.init() in the parent, fork() will give workers an unusable CUDA context. Use multiprocessing_context="spawn" or initialize CUDA only in workers.

persistent_workers=True cuts startup overhead

By default, workers are killed at the end of each epoch and respawned. Setting this to True keeps them alive — much faster for small datasets / many epochs.

torch.compile

First call is slow

Compilation costs seconds-to-tens-of-seconds. Subsequent calls hit the cache. Don't time the first call.

Recompiles are silent by default

TORCH_LOGS=recompiles reveals which guards are firing. Common culprits: shape changes, dtype changes, calling with requires_grad flipping.

Graph breaks fall back to eager

Dynamo will compile what it can, fall back to eager for the rest, and resume. The result is correct but the perf is whatever the worst-case eager+compile mix gives. TORCH_LOGS=graph_breaks shows them.

torch.compile(model.forward) ≠ torch.compile(model)

The first compiles the bound method; the second wraps the module so its __call__ is compiled. Subtle but the second is usually what you want.

mode="reduce-overhead" requires fixed input shapes

It enables CUDAGraphs, which capture a fixed sequence of GPU work for a fixed input shape. Variable shapes break it. Use dynamic=True for variable-shape compile (no CUDAGraphs).

Distributed

NCCL collectives must run in matching order on every rank

If rank 0 does all_reduce(a); all_reduce(b) and rank 1 does all_reduce(b); all_reduce(a), NCCL hangs. Set TORCH_NCCL_DESYNC_DEBUG=1 to detect this.

Default process_group is set per-process

torch.distributed.init_process_group(...) sets the default group. Operations like all_reduce(t) use it. To use a different group: all_reduce(t, group=my_group).

init_process_group waits

The default rendezvous is blocking. Mismatched world sizes cause hangs. Use --rdzv-backend=c10d --rdzv-endpoint=... with elastic for more graceful failure modes.

Custom ops

Not registering an Autograd kernel ⇒ no gradient

@torch.library.custom_op without an autograd formula won't differentiate through your op. Either register a backward or set mutates_args=() and let autograd derive one (with limitations).

Fake / meta kernels are required for torch.compile

Without a fake kernel, torch.compile can't infer output shape during tracing and will graph-break.

Tags matter

Ops without pointwise / data_dependent_output / inplace_view tags may not get the optimization treatment they deserve in Inductor.

Where to read next

• Design decisions — the why behind these.
• Debugging — diagnostic tools.