Sparse and special tensors

What it is

Most tensors in PyTorch are strided — a contiguous (or strided) memory range with shape and strides. PyTorch also supports several non-strided tensor varieties:

• Sparse COO / CSR / CSC / BSR / BSC — for matrices/tensors that are mostly zero.
• Nested (jagged) tensors — for batches of variable-length sequences.
• Masked tensors — value + boolean mask, with mask-aware ops.
• Quantized tensors — quint8/qint8/etc. with per-tensor or per-channel scale/zero-point.

This page is the user-level orientation for the four; for tensor-subclass machinery in general see Systems / Tensor subclasses.

Sparse tensors

Layouts:

• COO (torch.sparse_coo_tensor) — (indices, values) with shape [ndim, nnz] and [nnz]. Most flexible; relatively slow on GPU.
• CSR (torch.sparse_csr_tensor) — Compressed Sparse Row; (crow_indices, col_indices, values). Fast for spmm and most linalg.
• CSC (torch.sparse_csc_tensor) — Compressed Sparse Column.
• BSR / BSC (torch.sparse_bsr_tensor, …) — Blocked CSR/CSC; for matrices with dense block structure.

import torch
i = torch.tensor([[0, 1, 2], [2, 0, 1]])
v = torch.tensor([3.0, 4.0, 5.0])
s = torch.sparse_coo_tensor(i, v, (3, 3))
y = torch.sparse.mm(s, x)  # spmm

The C++ tensor impls are aten/src/ATen/SparseTensorImpl.{h,cpp} and aten/src/ATen/SparseCsrTensorImpl.{h,cpp}. Kernels live under aten/src/ATen/native/sparse/ and dispatch on the Sparse* / SparseCsr* keys.

A separate hardware-aware path is 2:4 semi-structured sparse — every block of 4 contiguous elements has at most 2 non-zero. Hopper+ tensor cores support this natively. Helpers in torch.sparse.semi_structured.

Nested (jagged) tensors

A sequence of tensors with the same number of dimensions but different sizes along one or more "ragged" dims. Internally stored as one contiguous buffer + an offsets array.

import torch
nt = torch.nested.nested_tensor([torch.randn(s, 8) for s in [3, 5, 7]])
attn = torch.nn.functional.scaled_dot_product_attention(nt, nt, nt)

Implementation:

• C++: aten/src/ATen/NestedTensorImpl.{h,cpp} and ops under aten/src/ATen/native/nested/.
• Python: torch/nested/. The "jagged" layout (torch.jagged) is the modern incarnation that integrates with torch.compile and DTensor.

Used heavily for transformer attention with variable-length sequences (no padding waste).

Masked tensors

A MaskedTensor is a (values, mask) pair where ops propagate the mask:

from torch.masked import masked_tensor
a = masked_tensor(torch.tensor([1., 2., 3.]), torch.tensor([True, False, True]))
b = a + 1  # values [2, 3, 4] but mask [True, False, True]

Implementation in torch/masked/. Experimental; useful for "ignore these entries during computation" semantics.

Quantized tensors

Quantized dtypes are a separate axis from layout: a tensor can be qint8 and strided, with associated scale/zero_point. The per-channel variants (per_channel_affine) carry a vector of scales and zero points along one dimension.

import torch
xq = torch.quantize_per_tensor(x, scale=0.1, zero_point=10, dtype=torch.qint8)

Quantized ops live at aten/src/ATen/native/quantized/ and dispatch on the Quantized* keys. See Features / Quantization.

Where to look

Path	Contents
torch/sparse/	Public sparse API
aten/src/ATen/native/sparse/	Sparse op kernels
torch/nested/	Public nested API
aten/src/ATen/native/nested/	Nested op kernels
torch/masked/	Masked tensor
torch/ao/quantization/	Quantization (separate page)

Where to read next

• Systems / Tensor subclasses — the mechanism behind these.
• Features / Quantization — quantized tensors.