Speculative decoding

Active contributors: Cody Yu, Lily Liu, Wentao Ye, Cyrus Leung.

Purpose

Drafting cheap candidate tokens and verifying them against the target model accelerates decoding when acceptance rates are high. vLLM's V1 spec-decode infrastructure is method-agnostic: the same scheduler/sampler infrastructure supports n-gram, suffix, EAGLE, MTP, Medusa, and standalone draft-model proposers.

Where it lives

vllm/v1/spec_decode/
├── llm_base_proposer.py    # Generic LLM-based proposer driver (~82 KB)
├── eagle.py                # EAGLE 2 / EAGLE 3
├── medusa.py               # Medusa heads
├── dflash.py               # DFlash drafting
├── ngram_proposer.py / ngram_proposer_gpu.py
├── suffix_decoding.py
├── draft_model.py          # Standalone draft model
├── extract_hidden_states.py
├── metrics.py              # SpecDecodingStats
├── metadata.py             # Per-step SpecDecodeMetadata
└── utils.py

vllm/v1/sample/rejection_sampler.py  # Verification step (~35 KB)
vllm/config/speculative.py           # SpeculativeConfig

Per-model spec decode pieces live in vllm/model_executor/models/:

• llama_eagle.py, llama_eagle3.py, llama4_eagle.py, deepseek_eagle.py, deepseek_eagle3.py, mistral_eagle.py, minicpm_eagle.py, mistral_large_3_eagle.py
• medusa.py
• mlp_speculator.py
• *_mtp.py files: deepseek_mtp.py, deepseek_v4_mtp.py, glm4_moe_mtp.py, glm4_moe_lite_mtp.py, ernie_mtp.py, mimo_mtp.py, mimo_v2_mtp.py, nemotron_h_mtp.py, qwen3_5_mtp.py, qwen3_next_mtp.py, step3p5_mtp.py, openpangu_mtp.py, hy_v3_mtp.py, glm_ocr_mtp.py, exaone4_5_mtp.py, exaone_moe_mtp.py, longcat_flash_mtp.py

Methods

Method	Pros	Cons
n-gram	Free at training time; great when prompt has repetitive structure	Acceptance falls off for novel content
suffix	Suffix-automaton over the whole context; broader matches than fixed-length n-gram	Memory-heavy on long contexts
draft model	Highest acceptance for natural text; flexible (any small LM)	Adds a second model to host
EAGLE / EAGLE3	Single-step head trained on hidden states; high acceptance	Per-architecture training; needs head weights
Medusa	Multi-head extra logits; trained alongside target	Per-architecture; smaller acceptance than EAGLE
MTP (multi-token prediction)	Built into the model; ~free at inference	Requires MTP-trained checkpoints
DFlash	Diff-flash specialized variant	Niche; specific model families

Configuration

SpeculativeConfig (vllm/config/speculative.py) is the user-facing dial. Pass it via --speculative-config '{...}':

# n-gram, prompt-based
vllm serve <model> --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_max":4}'

# Standalone draft model
vllm serve Qwen/Qwen2-7B-Instruct \
  --speculative-config '{"method":"draft_model","model":"Qwen/Qwen2-0.5B-Instruct","num_speculative_tokens":5}'

# EAGLE
vllm serve <model> --speculative-config '{"method":"eagle","model":"<eagle-head-weights>","num_speculative_tokens":4}'

# MTP (auto-detected for MTP-capable checkpoints)
vllm serve <model> --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

How a step runs

graph TD
    Sch[Scheduler.schedule]
    P[Proposer.propose:<br/>n-gram match / draft forward / EAGLE head]
    EX[Executor.execute_model<br/>with draft tokens]
    SmpV[RejectionSampler:<br/>verify each draft against<br/>target distribution]
    Acc[Accepted tokens]
    Rej[Rejected → fall back to argmax/sample]
    Out[ModelRunnerOutput]

    Sch --> P --> EX --> SmpV
    SmpV --> Acc --> Out
    SmpV --> Rej --> Out

The verification step is implemented in vllm/v1/sample/rejection_sampler.py. It re-uses the standard sampler for the "speculation failed" fallback so logits processors and structured-output masks still apply.

Tree attention

Some proposers (notably EAGLE) generate trees of candidate tokens that must be verified at once. That requires an attention backend with tree support — TREE_ATTN (vllm/v1/attention/backends/tree_attn.py) is automatically selected when needed.

Metrics

SpecDecodingStats (vllm/v1/spec_decode/metrics.py) reports:

• Drafted tokens / accepted tokens / acceptance rate
• Per-position acceptance (helps tune num_speculative_tokens)
• Wall time spent on draft vs verify

These flow into the standard stats pipeline (StatLoggerManager).

Key source files

File	Purpose
vllm/config/speculative.py	SpeculativeConfig (~1,400 lines)
vllm/v1/spec_decode/llm_base_proposer.py	Driver class for LLM-based proposers
vllm/v1/spec_decode/eagle.py	EAGLE 2 / 3
vllm/v1/spec_decode/ngram_proposer.py	n-gram (CPU)
vllm/v1/spec_decode/ngram_proposer_gpu.py	n-gram (GPU)
vllm/v1/spec_decode/medusa.py	Medusa
vllm/v1/spec_decode/dflash.py	DFlash
vllm/v1/spec_decode/suffix_decoding.py	Suffix automaton
vllm/v1/sample/rejection_sampler.py	Verification
vllm/v1/attention/backends/tree_attn.py	Tree attention

Entry points for modification

• New proposer: implement a class with propose(...) returning draft token ids per request. Add a branch in SpeculativeConfig.method and wire it in the GPU runner.
• New EAGLE-like head: drop a model file in vllm/model_executor/models/<arch>_eagle.py and register it; the proposer driver handles the rest.
• Tweak verification: subclass RejectionSampler and select via SpeculativeConfig.

For the model side, see Model executor. For attention support, see Attention backends.