Glossary

PyTorch is large enough that newcomers regularly hit unfamiliar terms. This glossary covers project-specific vocabulary; for general deep-learning terms (gradient, batch normalization, etc.) any introductory ML resource will do.

The repo also ships its own short glossary at GLOSSARY.md covering ATen, kernels, and TorchScript; the entries below are a superset.

Core abstractions

Tensor — n-dimensional array with attached dtype, device, layout, and optional requires_grad. The Python torch.Tensor is a thin wrapper over at::Tensor (C++) which is itself a smart pointer to c10::TensorImpl.

TensorImpl — the heap-allocated backing object for a tensor (c10/core/TensorImpl.h). Holds shape, strides, storage offset, dtype, device, dispatch keyset, and a pointer to a Storage.

Storage — a typed contiguous chunk of memory shared by one or more tensors. Implemented in c10/core/Storage.h and c10/core/StorageImpl.h.

Allocator — an interface for allocating memory on a device (c10/core/Allocator.h). Backends register their own (e.g., CUDACachingAllocator).

Device — (type, index) pair (c10/core/Device.h). Types include cpu, cuda, mps, xpu, mtia, meta, vulkan, lazy, privateuseone.

Layout — strided / sparse_coo / sparse_csr / sparse_csc / sparse_bsr / sparse_bsc / mkldnn / jagged.

ScalarType / dtype — the element type. PyTorch ships ~25 dtypes including float32, bfloat16, float8_e4m3fn, int8, quint4x2, etc. Defined in c10/core/ScalarType.h.

Operators and kernels

ATen — short for "A Tensor Library". The C++ tensor and op library (aten/). Almost every Python torch.* op lands in ATen.

Operator — a unit of work, e.g., aten::matmul. The set of operators is declared in aten/src/ATen/native/native_functions.yaml.

Schema — an operator's typed signature. Stored as a string like aten::add(Tensor self, Tensor other) -> Tensor. Parsed by c10::FunctionSchema.

Native operation — an operator that ships with ATen, defined in native_functions.yaml.

Custom operator — an operator registered at runtime via TORCH_LIBRARY / torch.library. Often used by extensions and by torch.compile's decomposition system.

Compound operator (a.k.a. composite op, non-leaf op) — an op implemented in terms of other ops. Usually device-agnostic with no explicit derivative; autograd derives via the chain rule on the constituent ops.

Leaf operator — a "primitive" op with explicit dispatch entries and an explicit derivative.

Kernel — concrete implementation of an op for a particular dispatch key. A leaf op has many kernels (CPU, CUDA, MPS, …); a compound op typically has a single composite kernel.

TensorIterator — aten/src/ATen/TensorIterator.h. The main loop driver for elementwise/reduction ops; computes broadcasting, dtype promotion, and iteration order.

Dispatcher

Dispatcher — the central op routing table (aten/src/ATen/core/dispatch/Dispatcher.h). Maps (operator, dispatch keyset) to a kernel.

DispatchKey — enum (c10/core/DispatchKey.h) labelling concerns: CPU, CUDA, Autograd, Autocast, FuncTorchBatched, Functionalize, Conjugate, Negative, ZeroTensor, PythonTLSSnapshot, Meta, Sparse*, Mkldnn*, Quantized*, Lazy, XLA, XPU, MPS, MTIA, PrivateUse1..3, …

KeySet — bitmap of dispatch keys carried by a tensor and ambient TLS. The dispatcher iterates highest-priority key first.

Functionalization — a dispatch-key-based pass that converts in-place and view ops into pure functional ones, used heavily by torch.compile and AOT autograd. See aten/src/ATen/FunctionalTensorWrapper.h.

Fallthrough kernel — a no-op kernel for a key that says "skip me, redispatch to the next key".

Autograd

grad_fn — the Node the autograd engine will call during backward to produce gradients for a tensor's parents.

Tape / autograd graph — the dynamic DAG of Nodes built up as ops execute. Nodes are produced by op-specific autograd kernels (e.g., MatmulBackward0).

Saved tensors — tensors stashed during forward for later use during backward. Memory-management hot spot; see torch/csrc/autograd/saved_variable.cpp.

Functional vs. inplace — autograd treats inplace ops by bumping a per-storage version counter; if a saved tensor is later mutated, backward will raise.

Compilation stack

torch.compile — user-facing decorator/function that triggers Dynamo + AOT autograd + Inductor.

Dynamo — Python-bytecode-level graph capture (torch/_dynamo/). Outputs FX graphs.

FX — symbolic trace + intermediate representation (torch/fx/). Functional graphs of call_function / call_module nodes.

AOT Autograd — pre-traces forward + backward into a joint graph (torch/_functorch/_aot_autograd/).

Inductor — compiler backend that lowers FX to Triton (GPU) and C++ (CPU) (torch/_inductor/).

Triton — GPU kernel DSL by OpenAI; PyTorch's primary GPU codegen target.

HOP (Higher-Order Operator) — special operators that take subgraphs as arguments (torch/_higher_order_ops/); used to model cond, while_loop, FlexAttention, etc.

FakeTensor — a tensor with shape/dtype/device but no real data (torch/_subclasses/fake_tensor.py). Used by Dynamo and AOT autograd to symbolically execute graphs.

Symbolic shapes (sym int / sym float) — lazily-evaluated integer/float expressions used to track dynamic shapes through tracing (torch/fx/experimental/symbolic_shapes.py).

Decomposition — replacement of an op by an FX subgraph that implements it in terms of simpler ops (torch/_decomp/).

torch.export — the supported way to capture a model into a stable, serializable graph (torch/export/).

TorchScript / JIT

JIT — Just-In-Time compilation. In PyTorch context typically means the legacy TorchScript compiler (torch/csrc/jit/).

TorchScript — a typed subset of Python that the JIT can compile to a serializable IR.

Tracing — torch.jit.trace: record a function's ops on example inputs.

Scripting — torch.jit.script: parse the function source and compile it as TorchScript.

Distributed

c10d — the C++ collective communication library (torch/csrc/distributed/c10d/). Backends: NCCL, Gloo, UCC, MPI.

Process Group / PG — a group of ranks that can collectively communicate. Created by dist.init_process_group and friends.

DTensor — distributed tensor that knows its sharding/replication across a device mesh (torch/distributed/tensor/). Successor to ShardedTensor.

Device Mesh — n-dimensional grid of ranks (torch/distributed/device_mesh.py).

FSDP / FSDP2 — Fully Sharded Data Parallel (torch/distributed/fsdp/ for v1, torch/distributed/_composable/fsdp/ for v2). Shards parameters across ranks and gathers them just-in-time.

DDP — DistributedDataParallel (torch/nn/parallel/distributed.py). Older replicated-parameter scheme.

TP / SP — Tensor parallelism / sequence parallelism, implemented on top of DTensor.

TorchElastic / torchrun — multi-node launcher and rendezvous (torch/distributed/run.py, torch/distributed/elastic/).

Pipeline parallelism — torch/distributed/pipelining/.

Quantization

FX Graph Mode Quantization — graph-rewriting based quantization via FX (torch/ao/quantization/quantize_fx.py).

Eager Mode Quantization — module-replacement based quantization (torch/ao/quantization/quantize.py).

PT2E (PyTorch 2 Export Quantization) — export-and-rewrite quantization on top of torch.export.

Observer — module that records statistics (min/max, histograms) used to choose quantization parameters.

Fake quant — module that simulates quantization in float during training (QAT).

Misc

spin — the developer-facing CLI for lint, test, format. See .spin/cmds.toml.

ghstack — Meta's stacked-PR workflow tool. See CLAUDE.md "ghstack Workflow".

tlparse — log parser for production debugging of structured traces emitted by torch._logging.trace_structured.

torchgen — the codegen package. Reads native_functions.yaml, derivatives.yaml, etc. and emits C++/Python code under torch/csrc/autograd/generated/ and aten/src/ATen/Operators*.

OpInfo — a structured description of an op (sample inputs, supported dtypes/devices, gradient support) used by the test infrastructure (torch/testing/_internal/common_methods_invocations.py).

@parametrize — PyTorch's test parameter expansion decorator.

instantiate_device_type_tests — generates per-device test classes (e.g., TestFooCPU, TestFooCUDA, TestFooMPS) from a base.