Storage and allocators

What it is

Storage is the thin layer between a Tensor and the raw bytes that hold its data. A storage owns:

• A DataPtr — a refcounted pointer to a buffer plus a deleter.
• A size in bytes.
• An Allocator* — who allocated it and who will free it.

Multiple tensors can share one storage (views, slices), and one storage can outlive the tensors that wrap it.

Layered view

Tensor → TensorImpl → Storage → StorageImpl → DataPtr → bytes
                                              ↑
                                              └── allocated/freed by Allocator

Allocators

c10::Allocator is the abstract interface (c10/core/Allocator.h). Each device has its own implementation:

Backend	Allocator	Notes
CPU	c10::DefaultCPUAllocator (c10/core/CPUAllocator.cpp)	malloc-based, with optional MemoryPlanner integration
CUDA	c10::cuda::CUDACachingAllocator (c10/cuda/)	Caching allocator with size-bucketed free lists
ROCm/HIP	c10::hip::HIPCachingAllocator (c10/hip/)	Same shape as the CUDA allocator
MPS	at::mps::MPSAllocator (aten/src/ATen/mps/MPSAllocator.mm)	MTLBuffer-based caching allocator
XPU	c10::xpu::XPUCachingAllocator (c10/xpu/)	SYCL-backed

The CPU allocator is small. The accelerator allocators are large because they implement the caching, multi-stream coordination, and OOM diagnostics covered in CUDA backend.

Caching allocator semantics

CUDA's allocator (and its mirrors on other backends) does not call the device-runtime free at the moment a tensor is freed. Instead it returns the block to a per-device, per-stream free list bucketed by size class. Subsequent allocations of the same size class hit the free list in O(1).

This matters because:

• cudaFree synchronizes the device. Calling it on every tensor free would destroy throughput.
• Concurrency. Two streams can hold pointers into the same allocation pool without stepping on each other.
• Fragmentation. The allocator merges adjacent free blocks back into larger chunks.

The PYTORCH_CUDA_ALLOC_CONF environment variable controls every knob. Common settings:

Knob	What it does
max_split_size_mb	Don't split large blocks below this size
garbage_collection_threshold	Aggressive GC threshold to reduce fragmentation
expandable_segments	Use cuMemMap-backed segments that can grow
roundup_power2_divisions	How many size classes per power-of-2 bucket
pinned_use_cuda_host_register	Pin host memory via cudaHostRegister instead of cudaMallocHost

Pinned memory

Pinned (page-locked) host memory enables async H2D copies. PyTorch's torch.cuda.PinnedMemoryAllocator and the DataLoader's pin_memory=True go through c10::cuda::CUDAHostAllocator / c10::cuda::CUDACachingHostAllocator (c10/cuda/CUDAHostAllocator.cpp). Pinned allocations are precious — over-allocating starves the OS — so the cached host allocator is conservative.

Shared and mmap-backed storage

aten/src/ATen/MapAllocator.cpp implements memory-mapped storage. Used by:

• torch.from_file(..., shared=True) — shared between processes.
• torch.load(..., mmap=True) — mmap a checkpoint instead of reading it.
• torch.multiprocessing shared CPU tensors — uses MapAllocator under the hood.

Custom allocators

Two extension points:

1. c10::cuda::CUDACachingAllocator::allocator() — replace the global CUDA allocator at startup (advanced; unstable).
2. Custom backend — register your Allocator for PrivateUse1 along with your dispatcher kernels.

Data flow on free

When the last intrusive_ptr<StorageImpl> referring to an allocation is dropped:

1. StorageImpl::~StorageImpl runs.
2. The DataPtr's deleter is invoked.
3. The deleter (set by the allocator at allocation time) calls back into the allocator to recycle the block.

For the CUDA allocator the recycling is purely a list-update; the underlying cudaMalloc'd segment is preserved.

Where to look

File	Purpose
c10/core/Storage.h, StorageImpl.h	Storage and StorageImpl
c10/core/Allocator.h	Allocator interface
c10/core/CPUAllocator.cpp	Default CPU allocator
c10/cuda/CUDACachingAllocator.cpp	CUDA caching allocator
c10/cuda/CUDAHostAllocator.cpp	Pinned host allocator
aten/src/ATen/MapAllocator.cpp	mmap-backed storage
c10/cuda/CUDAAllocatorConfig.cpp	PYTORCH_CUDA_ALLOC_CONF parsing

Where to read next

• Primitives / Tensor — what sits on top of Storage.
• Systems / CUDA backend — the allocator in context.
• Features / Profiling and tracing — memory snapshots.