Tensor

What it is

The central type. A torch.Tensor (Python) is a thin wrapper over at::Tensor (C++), which is itself a c10::intrusive_ptr<c10::TensorImpl>. Almost every other concept in PyTorch ties back to this object.

Memory layout

Tensor (intrusive_ptr<TensorImpl>)
  └── TensorImpl
        ├── sizes_       SmallVector<int64_t>
        ├── strides_     SmallVector<int64_t>
        ├── storage_offset_  int64_t
        ├── dtype_       caffe2::TypeMeta (≈ ScalarType)
        ├── device_      Device
        ├── key_set_     DispatchKeySet
        ├── version_counter_  intrusive_ptr<VersionCounter>
        ├── autograd_meta_    unique_ptr<AutogradMeta> (or null)
        └── storage_     Storage (pointer to storage)
                            └── StorageImpl
                                  ├── allocator_  Allocator*
                                  ├── data_ptr_   DataPtr (refcounted)
                                  └── nbytes_     size_t

Two tensors can share the same Storage (e.g., a view and its base, or two slices of the same underlying buffer). They have their own sizes/strides/storage_offset but read into the same memory.

Sizes, strides, contiguity

size(i) is the length of dim i; stride(i) is the number of elements (not bytes) to skip to advance one along dim i. The element at index (i_0, i_1, …) lives at storage[storage_offset + Σ i_k * stride_k].

A tensor is contiguous in memory format M if its strides match the canonical iteration order for M:

• MemoryFormat::Contiguous — last dim varies fastest (row-major).
• MemoryFormat::ChannelsLast (NHWC for 4D, NDHWC for 5D) — channels dim varies fastest.

tensor.contiguous() returns a contiguous copy (no-op if already so). tensor.is_contiguous(memory_format=...) checks.

Dtypes

Defined in c10/core/ScalarType.h. The supported list as of late 2025:

float32 (default), float64, float16, bfloat16,
float8_e4m3fn, float8_e4m3fnuz, float8_e5m2, float8_e5m2fnuz, float8_e8m0fnu,
int8, int16, int32, int64, uint8, uint16, uint32, uint64,
bool,
complex32, complex64, complex128,
qint8, quint8, quint4x2, quint2x4, qint32,
bits1x8, bits2x4, bits4x2, bits8, bits16

Not every backend supports every dtype. A qint8-on-CUDA op may not exist; the dispatcher will report a missing kernel.

Devices and key sets

tensor.device is (type, index); tensor.layout is strided/sparse**/mkldnn/jagged. The key_set* carries the set of dispatch keys this tensor cares about — backend (CUDA), layout (Sparse*), tensor flags (Conjugate, Negative, ZeroTensor), wrapper subclasses (Python, FuncTorch*).

This is the reason dispatcher lookups are O(1): the keyset bitmap lets the dispatcher pick the highest-priority key with __builtin_clz.

Autograd metadata

If requires_grad was ever set, TensorImpl::autograd_meta_ is a unique_ptr<AutogradMeta> with:

• grad_ — the accumulated gradient tensor.
• grad_fn_ — the backward node that produced this tensor.
• version_counter_ — bumped on every in-place op; saved-tensor version checks consult this.
• hooks_ — per-tensor backward hooks.
• fw_grad_ — forward-mode tangent.

Tensors with requires_grad=False and no parent grad_fn carry no AutogradMeta (it's lazily allocated).

Views

A view shares storage with its base. view, slice, select, transpose, permute, unsqueeze, squeeze, expand, as_strided, narrow, unfold, and many more produce views. The version_counter_ is inherited from the base — that's how autograd detects in-place mutations to a view that was saved as a base (or vice versa).

A small but important wrinkle: not every "looks like a view" op produces a view. flatten may copy if the result wouldn't be representable as a strided view of the input. PyTorch's policy is "view if possible, copy otherwise"; the test for "possible" is in aten/src/ATen/MemoryOverlap.cpp and the size/stride machinery.

Reference counting

Tensor is movable and copyable. Copy is cheap — it bumps the intrusive_ptr<TensorImpl> refcount. Mutations to one alias the other (because the underlying TensorImpl is shared). The Python Tensor is a torch._C._TensorBase wrapping the same C++ smart pointer.

Where to look

File	Purpose
c10/core/TensorImpl.h	The TensorImpl declaration
c10/core/TensorImpl.cpp	Implementation
aten/src/ATen/templates/TensorBody.h	Public C++ at::Tensor API
c10/core/Storage.h, StorageImpl.h	Storage
c10/core/ScalarType.h	Dtype enum
c10/core/MemoryFormat.h	Memory format
c10/core/DispatchKeySet.h	Keyset
torch/csrc/autograd/variable.h	AutogradMeta
aten/src/ATen/MemoryOverlap.cpp	View-vs-copy machinery

Where to read next

• Primitives / Storage and allocators — backing memory.
• Primitives / Dispatch keys — what's in the keyset.
• Systems / c10 — the layer where TensorImpl lives.