Platforms and hardware backends

Active contributors: Micah Williamson, rasmith, Charlie Fu, Kunshang Ji, Lucas Wilkinson.

Purpose

vLLM runs on many accelerators. vllm/platforms/ is the abstraction layer that lets the same engine pick the right defaults (executor backend, attention backend, dtype, communicator, allocator) per device class. A Platform is also where the model loader, kernel autotuner, and CUDA-graph capture ask "is this op supported here?".

Directory layout

vllm/platforms/
├── __init__.py    # current_platform setup, platform detection
├── interface.py   # Platform abstract base (~33 KB)
├── cuda.py        # NVIDIA GPUs
├── rocm.py        # AMD GPUs (~35 KB)
├── cpu.py         # Generic x86/ARM CPU
├── zen_cpu.py     # AMD Zen-series CPU specialization
├── xpu.py         # Intel XPU (~16 KB)
└── tpu.py         # TPU placeholder (real impl ships as a plugin)

Out-of-tree plugins implement additional platforms (Google TPU, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, etc.).

Key abstractions

Abstraction	File	Role
Platform	vllm/platforms/interface.py	Abstract base; methods for capability queries
current_platform	vllm/platforms/__init__.py	The active platform singleton
is_cuda(), is_rocm(), is_xpu(), is_cpu()	vllm/platforms/__init__.py	Convenience checks
Platform.get_device_capability()	platform	(major, minor) compute capability
Platform.supported_attn_backends()	platform	List of attention backend enums supported here
Platform.recommended_executor_backend()	platform	mp, ray, uni default per platform

What a platform decides

graph TD
    PD[Platform detection<br/>vllm/platforms/__init__.py]
    Defaults[Default config:<br/>executor backend, attention backend,<br/>dtype, communicator, allocator]
    Caps[Capability checks:<br/>FP8 KV cache? CUDA graphs? sliding window?]
    Plug[Plugin discovery via entry points]
    Cfg[VllmConfig finalization]

    PD --> Defaults --> Cfg
    PD --> Caps --> Cfg
    Plug --> PD

Detection order:

1. Check VLLM_PLATFORM env (explicit override).
2. Discover platform plugins via the vllm.platform_plugins entry point group.
3. Probe torch.cuda.is_available(), torch.xpu.is_available(), ROCm, etc.
4. Default to UnspecifiedPlatform (mostly used by vllm bench and other CPU-only flows; see vllm/entrypoints/cli/main.py).

CUDA platform (vllm/platforms/cuda.py)

• Detects compute capability and device name.
• Picks the FlashAttention vs FlashInfer vs Triton priority list.
• Sets the default cuda_graph_mode (usually FULL_AND_PIECEWISE).
• Wires up the cumem_allocator.
• Supplies CUDA-specific worker (vllm/v1/worker/gpu_worker.py).

ROCm platform (vllm/platforms/rocm.py)

• Detects MI300X / MI250 / RDNA tiers.
• Filters attention backends (ROCM_ATTN, ROCM_AITER_FA, ROCM_AITER_MLA*).
• Switches to AITER ops (vllm/_aiter_ops.py) where available.
• Picks NCCL or RCCL via the ROCm communicator.

CPU platform (vllm/platforms/cpu.py, zen_cpu.py)

• Selects the CPU model runner / worker.
• Tunes thread pinning via vllm/utils/numa_utils.py and numa_wrapper.sh.
• Picks the CPU attention backend (cpu_attn.py) and CPU MoE (cpu_fused_moe.py).
• For Zen CPUs, enables AVX-512 / AMX paths.

XPU platform (vllm/platforms/xpu.py)

• Targets Intel Data Center GPU Max (Ponte Vecchio) and discrete Arc GPUs.
• Uses oneCCL for collectives and IPEX for ops not yet in upstream PyTorch.

Plugins

Out-of-tree platform plugins register through Python entry points:

# Example pyproject.toml of a hypothetical plugin
[project.entry-points."vllm.platform_plugins"]
my_platform = "my_platform.platform:MyPlatform"

vllm/plugins/__init__.py discovers them on engine startup. The plugin's Platform subclass gets a chance to register its own attention backends, model architectures, executor backend, communicator, and quantization formats.

Key source files

File	Purpose
vllm/platforms/__init__.py	Detection, current_platform singleton
vllm/platforms/interface.py	Abstract Platform
vllm/platforms/cuda.py	NVIDIA defaults
vllm/platforms/rocm.py	AMD defaults
vllm/platforms/cpu.py	CPU defaults
vllm/platforms/xpu.py	Intel XPU defaults
vllm/plugins/__init__.py	General-plugin loader (also handles platform plugins)

Entry points for modification

• New device tier: tweak the Platform subclass to add the new compute capability and enable the right backends.
• New plugin: ship a separate package, register vllm.platform_plugins entry point, subclass Platform.
• Default tuning: most defaults (block size, gpu memory utilization, CUDA graph mode) are computed per-platform — adjust there rather than per-model.