PyTorch overview

PyTorch is a Python-first deep learning framework that combines a NumPy-like tensor library with strong GPU acceleration and a tape-based autograd system. It powers a large fraction of academic and industry deep learning work, including most modern LLM and vision research stacks. The repository contains the core tensor library, the eager runtime, the torch.compile graph capture and inductor backend, distributed training primitives, mobile and ONNX export pipelines, and the Python bindings that tie it all together.

What is PyTorch

PyTorch is structured as a layered system around a single concept: the Tensor. The library exposes:

• Tensor computation — n-dimensional arrays on CPU, CUDA, ROCm, MPS, XPU, and other accelerators with broadcasting, indexing, type promotion, and autograd.
• torch.nn — a deep learning module library built on top of autograd.
• torch.compile (Dynamo + Inductor) — a Python bytecode-level graph capture compiler with a Triton-based code generator.
• torch.distributed — collectives, FSDP, DTensor, and pipeline parallelism for multi-GPU and multi-node training.
• Export and serving — torch.export, ONNX export, JIT/TorchScript, and the torch::deploy/AOTInductor runtimes.

High-level layout

graph TD
    Py[Python frontend<br/>torch/*.py] -->|pybind11| Csrc[C++ Python bindings<br/>torch/csrc/]
    Csrc -->|calls| ATen[ATen tensor library<br/>aten/src/ATen/]
    Csrc -->|autograd engine| AG[Autograd<br/>torch/csrc/autograd/]
    ATen -->|dispatcher| Kernels[Backend kernels<br/>CPU / CUDA / MPS / XPU / ROCm]
    ATen --> C10[c10 core<br/>c10/]
    Py -.-> Compile[torch.compile<br/>_dynamo / _inductor / _functorch]
    Compile -->|Triton, C++| Kernels
    Py -.-> Dist[torch.distributed<br/>c10d, FSDP, DTensor]
    Dist -->|NCCL/Gloo/UCC| Net[Process group backends]

The core tensor library is aten/src/ATen/. The reference-counted, header-only types (Storage, TensorImpl, dispatch keys, scalar types) live in c10/. The Python bindings, autograd engine, JIT, and compiler entry points live in torch/csrc/. The Python-facing API lives in torch/.

Where to start

• First-time reader: this page → Architecture → Glossary.
• Want to build it: Getting started.
• Want to contribute: How to contribute.
• Want to understand a specific subsystem: jump to Systems (ATen, autograd, dispatcher, dynamo, inductor, distributed, …) or Features.
• Looking for the canonical APIs: Reference.

Quick orientation table

Question	Page
How does eager execution work?	Systems / ATen + dispatcher
How is autograd implemented?	Systems / autograd
What is torch.compile?	Features / torch.compile
How are CUDA kernels dispatched?	Systems / dispatcher
How do I add a new operator?	How to contribute / patterns and conventions
How does FSDP work?	Systems / distributed
What's the relationship between ATen and c10?	Systems / c10

Project facts

• Language mix (excluding third_party/): ~570K lines of Python, ~800K lines of C++, ~390K lines of C++ headers, ~110K lines of CUDA, plus Objective-C++ for MPS and Metal.
• Top-level source directories: torch/, aten/, c10/, torchgen/, functorch/, tools/, test/, benchmarks/, caffe2/ (legacy), android/, binaries/, scripts/.
• Build system: CMake (CMakeLists.txt) driven by setup.py/pip install -e . --no-build-isolation, with Bazel/Buck files for internal Meta builds.
• Code generation: The torchgen/ package generates ATen op stubs, autograd kernels, and Python bindings from aten/src/ATen/native/native_functions.yaml and tools/autograd/derivatives.yaml.
• CI: GitHub Actions under .github/workflows/, with ciflow/* tags driving on-demand jobs (see Deployment).

For a numerical snapshot of the codebase see By the numbers. For history and major rewrites see Lore.