Quantization

Active contributors: Michael Goin, Lucas Wilkinson, rasmith, Tyler Michael Smith.

Purpose

Modern inference is rarely FP16/BF16 weights only. vLLM ships ~20 quantization backends so that a wide range of weight-only, weight-and-activation, and KV-cache quantization schemes work end-to-end with the right kernels on the right hardware.

Where it lives

vllm/model_executor/layers/quantization/
├── __init__.py             # QuantizationMethods registry
├── base_config.py          # QuantizationConfig base class
├── kv_cache.py             # KV cache quantization (FP8 / int8)
├── schema.py               # JSON schema for compressed-tensors
├── awq.py / awq_marlin.py / awq_triton.py
├── gptq.py / gptq_marlin.py
├── fp8.py / input_quant_fp8.py / fbgemm_fp8.py
├── modelopt.py             # NVFP4 / FP8 (NVIDIA ModelOpt)
├── mxfp4.py                # OCP MXFP4
├── compressed_tensors/     # NeuralMagic compressed-tensors
├── bitsandbytes.py         # 4-bit / 8-bit (bnb)
├── gguf.py                 # GGUF (llama.cpp)
├── humming.py              # vLLM-native fused humming format
├── inc.py                  # Intel Neural Compressor
├── quark/                  # AMD QUARK
├── torchao.py              # PyTorch torchao
├── turboquant/             # NVIDIA TurboQuant
├── moe_wna16.py            # Weight-only int4 for MoE
├── cpu_wna16.py            # CPU equivalent
├── fp_quant.py / experts_int8.py / qutlass_utils.py
├── online/                 # Online (RTN) quantization
└── utils/                  # Shared shape and dtype helpers

Key abstractions

Abstraction	File	Role
QuantizationMethods	vllm/model_executor/layers/quantization/__init__.py	Registry: name → config class
QuantizationConfig	base_config.py	Per-method config + get_linear_method
LinearMethodBase / FusedMoEMethodBase	per-backend	Layer-side API the method implements
vLLMParameter	vllm/model_executor/parameter.py	Sharding-aware storage for quantized weights
kv_cache.py::QuantizedKVCacheBackend	kv_cache.py	FP8 / int8 KV cache support
OnlineQuantizationConfigArgs	vllm/config/quantization.py	RTN online-quantization settings

How a quantization method gets used

graph TD
    Hf[HF model config<br/>quantization_config or arg]
    Reg[QuantizationMethods registry]
    QC[QuantizationConfig instance]
    LM[get_linear_method / get_moe_method / get_kv_cache_method]
    Layer[Linear / FusedMoE / Attention layers]
    Kernel[Backend kernel<br/>Marlin / CUTLASS / Triton / AITER / CPU]

    Hf --> Reg --> QC --> LM --> Layer --> Kernel

When a model is loaded:

1. --quantization (or the quantization_config block in config.json) selects the registry entry.
2. Layers (linear, MoE, attention) call quant_config.get_linear_method(...) (or get_moe_method, get_kv_cache_method) to obtain a quant-aware layer wrapper.
3. The wrapper holds quantized vLLMParameters, dequant scales/zeros, and dispatches the right kernel in forward.

Backends at a glance

Method	Format	Kernels
awq	INT4 weight-only	awq.py / awq_marlin.py (Marlin) / awq_triton.py
gptq	INT4 weight-only	gptq.py / gptq_marlin.py (Marlin)
fp8	FP8 weight + activation	CUTLASS / per-token dynamic / fbgemm
compressed-tensors	Generic (INT4/8, FP8, NVFP4)	compressed_tensors/ — multiple internal kernels
modelopt	NVFP4 / FP8	modelopt.py (NVIDIA ModelOpt)
mxfp4	OCP MXFP4	mxfp4.py
bitsandbytes	NF4 / FP4 / INT8	bitsandbytes.py (bnb)
gguf	GGUF (llama.cpp)	gguf.py
inc	Intel Neural Compressor	inc.py
quark	AMD QUARK	quark/
torchao	PyTorch torchao	torchao.py
turboquant	NVIDIA TurboQuant	turboquant/
humming	vLLM-native fused MoE format	humming.py (~37 KB)
moe_wna16 / cpu_wna16	Weight-only int4 for MoE	moe_wna16.py / cpu_wna16.py
experts_int8	INT8 experts	experts_int8.py
online (RTN)	Round-to-nearest	online/
KV cache quantization	FP8 / INT8 KV	kv_cache.py + per-backend support in attention backends

Online quantization

Some methods (fp8, compressed-tensors, online/) can quantize weights at load time instead of requiring pre-quantized files. OnlineQuantizationConfigArgs (vllm/config/quantization.py) exposes the knobs (group size, calibration, etc.). Online quantization is convenient for experiments but slower at load.

Kernels

The native kernels live in csrc/quantization/ — Marlin, CUTLASS-based GEMMs, FP8 group-scaled GEMMs, INT8 GEMMs, NVFP4 / MXFP4 paths, plus AWQ/GPTQ specialized variants. They are exposed via vllm/_custom_ops.py.

Picking a method

Rough rules of thumb:

• NVIDIA Hopper / Blackwell — fp8 (FBGEMM/CUTLASS), modelopt (NVFP4/FP8), or compressed-tensors.
• NVIDIA Ampere/Ada — gptq_marlin / awq_marlin (INT4 weight-only) for quick wins; FP8 paths when supported.
• AMD MI300X — quark or compressed-tensors with AITER kernels.
• CPU — cpu_wna16, bitsandbytes, or gguf for server CPUs; inc on Intel.
• MoE-heavy models — humming, moe_wna16, experts_int8, or compressed-tensors with MoE support.

Key source files

File	Purpose
vllm/model_executor/layers/quantization/__init__.py	Registry
vllm/model_executor/layers/quantization/base_config.py	Base QuantizationConfig
vllm/model_executor/layers/quantization/kv_cache.py	KV cache quantization
vllm/model_executor/layers/quantization/compressed_tensors/	Largest sub-tree; multi-format support
vllm/model_executor/layers/quantization/modelopt.py	NVIDIA ModelOpt (~80 KB)
vllm/model_executor/layers/fused_moe/	MoE-side quant kernels
csrc/quantization/	Native quant kernels
vllm/_custom_ops.py	Python surface

Entry points for modification

• New format: subclass QuantizationConfig in a new module, register via QuantizationMethods, implement get_linear_method (and friends) returning a layer-side method class. Add tests under tests/quantization/.
• New kernel for an existing format: add the kernel, expose via _custom_ops.py, and select inside the existing LinearMethod's dispatch.
• KV cache quant: extend kv_cache.py and ensure each attention backend you target lists the dtype in its supported_kv_cache_dtypes.

For the layer wrappers that consume quantized weights, see Model executor. For attention backends that consume quantized KV, see Attention backends.