Quantization

What it is

Quantization runs a model in int8 (or smaller) instead of float32 for faster inference and smaller memory footprint. PyTorch supports three workflows:

• Eager mode — manually insert QuantStub/DeQuantStub, prepare and convert.
• FX graph mode — graph-rewrite based, takes a config dict.
• PT2 Export (PT2E) — built on torch.export; the recommended path going forward.

For implementation see Systems / Quantization. This page is the user-level orientation.

Workflow comparison

Workflow	Trace mechanism	Backends	Deployment	Status
Eager	Manual stubs	FBGEMM, QNNPACK	TorchScript / Mobile	Mature, maintained
FX	torch.fx.symbolic_trace	FBGEMM, QNNPACK	TorchScript / Mobile	Mature
PT2E	torch.export	XNNPACK, X86Inductor, custom	ExecuTorch / AOTInductor	Recommended for new work

Eager mode

import torch
from torch.ao.quantization import get_default_qconfig, prepare, convert
from torch.ao.quantization import QuantStub, DeQuantStub

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.quant = QuantStub()
        self.linear = torch.nn.Linear(8, 8)
        self.dequant = DeQuantStub()

    def forward(self, x):
        return self.dequant(self.linear(self.quant(x)))

model = M().eval()
model.qconfig = get_default_qconfig("fbgemm")
prepared = prepare(model, inplace=False)

# calibration loop ...
for x in calibration_data:
    prepared(x)

quantized = convert(prepared, inplace=False)

FX mode

from torch.ao.quantization.quantize_fx import prepare_fx, convert_fx
from torch.ao.quantization import get_default_qconfig_mapping

qconfig_mapping = get_default_qconfig_mapping("fbgemm")
prepared = prepare_fx(model, qconfig_mapping, example_inputs=(x,))
# calibrate ...
quantized = convert_fx(prepared)

The qconfig_mapping lets you target specific module types or names with different qconfigs.

PT2E mode

import torch
from torch.export import export
from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
from torch.ao.quantization.quantizer.xnnpack_quantizer import (
    XNNPACKQuantizer,
    get_symmetric_quantization_config,
)

ep = export(model, args=(x,))
quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
prepared = prepare_pt2e(ep.module(), quantizer)
# calibrate ...
quantized = convert_pt2e(prepared)

The Quantizer chooses what to quantize; built-in quantizers cover XNNPACK, X86Inductor, and a few internal Meta backends.

QAT (quantization-aware training)

QAT inserts fake-quant modules during training so the model learns to be robust to quantization. The Eager and FX flows have a prepare_qat/prepare_qat_fx variant; PT2E has prepare_pt2e_qat.

Sparsity and pruning

torch/ao/sparsity/ and torch/ao/pruning/ are the sister packages for unstructured/structured sparsity and pruning. They share the same observer/qconfig vocabulary.

What's quantized vs. dequantized

In a typical convnet, every conv/matmul/linear/elementwise lives in int8; activations between fused conv-relu blocks stay int8; the final classifier may stay in float; loss-sensitive ops stay in float. The qconfig + backend config decide.

Where to look

Path	Contents
torch/ao/quantization/	All quantization code
torch/ao/quantization/observer.py	Observers
torch/ao/quantization/fake_quantize.py	Fake-quant modules
torch/ao/quantization/quantize_fx.py	FX flow
torch/ao/quantization/pt2e/	PT2E flow
aten/src/ATen/native/quantized/	Quantized op kernels
torch/ao/nn/	Quantized nn.Modules

Where to read next

• Systems / Quantization — implementation details.
• Features / Mobile and edge — XNNPACK + ExecuTorch deployment.
• Features / torch.export and AOTInductor — PT2E's foundation.