Quantization

Purpose

transformers integrates with 20+ quantization backends so users can load weights at int8 / int4 / fp4 / mxfp4 / fp8 precision and recover most of the throughput and memory benefits without writing kernel code. The HfQuantizer abstraction (introduced in PR #26610, January 2024) decouples quantization from modeling_utils.py and lets each backend implement its own load-time and runtime behaviour.

Key abstractions

Class / function	File	Role
HfQuantizer (base)	src/transformers/quantizers/base.py	Lifecycle hooks called by from_pretrained
Quantizer subclasses	src/transformers/quantizers/quantizer_*.py	One per backend
AutoQuantizationConfig, QuantizationConfigMixin	src/transformers/quantizers/auto.py, src/transformers/utils/quantization_config.py (89K LOC)	Config dataclasses
Integration helpers	src/transformers/integrations/<backend>.py	Linear layer replacements, kernel calls

Supported backends

Backend	Quantizer file	Integration helper	Bits
bitsandbytes	quantizer_bnb_8bit.py, quantizer_bnb_4bit.py	integrations/bitsandbytes.py	int8, int4 (NF4, FP4)
GPTQ (auto-gptq)	quantizer_gptq.py	(via optimum)	int4
AWQ (autoawq)	quantizer_awq.py	integrations/awq.py	int4
AQLM	quantizer_aqlm.py	integrations/aqlm.py	extreme low-bit
AutoRound	quantizer_auto_round.py	(via auto-round)	int4
EETQ	quantizer_eetq.py	integrations/eetq.py	int8
HQQ	quantizer_hqq.py	integrations/hqq.py	int1–int8
Quanto	quantizer_quanto.py	integrations/quanto.py	int2/int4/int8
TorchAO	quantizer_torchao.py	integrations/torchao.py	many
mxfp4	quantizer_mxfp4.py	integrations/mxfp4.py (28K LOC)	mxfp4
FBGEMM-FP8	quantizer_fbgemm_fp8.py	integrations/fbgemm_fp8.py	fp8
Fine-grained FP8	quantizer_finegrained_fp8.py	integrations/finegrained_fp8.py (39K LOC)	fp8
VPTQ	quantizer_vptq.py	integrations/vptq.py	extreme low-bit
SinQ	quantizer_sinq.py	integrations/sinq.py	int4
SpQR	quantizer_spqr.py	integrations/spqr.py	low-bit + sparse
Higgs	quantizer_higgs.py	integrations/higgs.py (31K LOC)	low-bit
Compressed Tensors	quantizer_compressed_tensors.py	(Neural Magic)	various
BitNet	quantizer_bitnet.py	integrations/bitnet.py	1-bit
Quark	quantizer_quark.py	integrations/quark.py	various
FPQuant	quantizer_fp_quant.py	integrations/fp_quant.py	fp4
fouroversix	quantizer_fouroversix.py	integrations/fouroversix.py	bespoke
Metal	quantizer_metal.py	integrations/metal_quantization.py	Apple Silicon

Loading a quantized model

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-1.5B", quantization_config=bnb, device_map="auto")

from_pretrained reads quantization_config (or finds one on disk if the checkpoint is pre-quantized), instantiates the right HfQuantizer, and the quantizer's lifecycle hooks replace nn.Linear layers with backend-specific quantized variants during weight loading. The hooks are:

• validate_environment — check that the dependency is installed.
• update_torch_dtype — force compute dtype if needed.
• _process_model_before_weight_loading — replace layers.
• update_missing_keys, update_unexpected_keys — adjust loading reports.
• _process_model_after_weight_loading — finalize.
• is_serializable, is_trainable — capability flags.

Saving and reloading

Quantizers that are serializable (is_serializable=True) write their state to the checkpoint so a later from_pretrained reproduces the same quantization without specifying a config.

Training quantized models

Most backends are inference-only. PEFT (LoRA) is the standard way to fine-tune a quantized base model:

from peft import LoraConfig, get_peft_model
model = get_peft_model(model, LoraConfig(...))
trainer.train()

bitsandbytes, hqq, quanto, and torchao advertise is_trainable=True for QLoRA-style workflows. See PEFT integration.

KV cache quantization

Separate from weight quantization, the Cache page describes QuantizedCache for compressing the KV state during generation.

Testing

tests/quantization/<backend>/ contains per-backend tests gated by @require_<backend>. Common tests cover:

• Round-trip serialization.
• Generation parity within a tolerance.
• Memory reduction.
• LoRA training (where supported).

Integration points

• Modeling — from_pretrained calls into the HfQuantizer lifecycle.
• Trainer — quantized models are usually combined with PEFT for training.
• Continuous batching — quantized weights work transparently with paged attention.

Entry points for modification

• Add a backend → drop a quantizer_<name>.py in quantizers/, an integration helper in integrations/, a config class in utils/quantization_config.py, register it in quantizers/auto.py, add tests under tests/quantization/<name>/.
• Tune defaults → modify the config dataclass in utils/quantization_config.py.
• Debug a load → set transformers.logging.set_verbosity_debug() and inspect the loading report from utils/loading_report.py.