Multi-modal

Active contributors: Roger Wang, Cyrus Leung, Isotr0py, DarkLight1337.

Purpose

vllm/multimodal/ provides the request preprocessing, encoder caching, and budgeting infrastructure for vision-language, audio-language, and video-language models. The actual encoders live with the model implementations in vllm/model_executor/models/ (e.g., clip.py, pixtral.py, vision.py).

Directory layout

vllm/multimodal/
├── __init__.py
├── inputs.py          # Multi-modal request types (MMInput, MultiModalFeatureSpec)
├── parse.py           # Parsing of OpenAI-style image/audio/video parts
├── registry.py        # MULTIMODAL_REGISTRY: per-model preprocessing
├── processing/        # Generic processing pipeline pieces
├── media/             # PIL/torchvision/torchaudio loaders
├── image.py           # Image utilities
├── audio.py           # Audio resampling/normalization
├── video.py           # Video frame extraction (~37 KB)
├── cache.py           # MMCache: dedup repeated multimodal inputs
├── encoder_budget.py  # MultiModalBudget — per-step encoder admission
├── evs.py             # Embedding versioning state
├── hasher.py          # Content hash for MM cache
└── utils.py

Key abstractions

Abstraction	File	Role
MULTIMODAL_REGISTRY	vllm/multimodal/registry.py	Per-model preprocessing + token-count helpers
MultiModalFeatureSpec	vllm/multimodal/inputs.py	Describes one MM input slot in a request
MultiModalBudget	vllm/multimodal/encoder_budget.py	Caps how many MM tokens enter the encoder per step
MMCache	vllm/multimodal/cache.py	Reuses encoder outputs across requests
EncoderCacheManager	vllm/v1/core/encoder_cache_manager.py	Schedules encoder cache slots
EncoderDecoderCacheManager	vllm/v1/core/encoder_cache_manager.py	Encoder-decoder split
EncoderCUDAGraph	vllm/v1/worker/encoder_cudagraph.py	CUDA-graph capture for encoder forwards

Pipeline

graph TD
    HTTP[OpenAI request<br/>messages with images/audio/video]
    Parse[parse.py: extract MM parts]
    Reg[MULTIMODAL_REGISTRY:<br/>per-model preprocessor]
    Hash[hasher.py: content hash]
    Cache[MMCache lookup]
    Spec[MultiModalFeatureSpec list]
    EngReq[EngineCoreRequest.mm_features]
    Sched[Scheduler reserves encoder slot via MultiModalBudget]
    Worker[Worker runs encoder + cross-attention via EncoderCacheManager]

    HTTP --> Parse --> Reg --> Hash --> Cache
    Cache -->|hit| Spec
    Cache -->|miss| Reg --> Spec
    Spec --> EngReq --> Sched --> Worker

Encoder budgeting

MultiModalBudget lets the scheduler cap how many encoder tokens (image patches, audio frames, video frames) it admits per step. Without budgeting, a single request with a 1080p video could starve other requests of the encoder. The budget is configurable via MultiModalConfig.mm_processor_kwargs and platform defaults.

MM cache

MMCache keys encoder outputs by (model_hash, input_hash) and stores them in CPU RAM. When the same image (e.g., system prompt avatar) appears in many requests, only the first request runs the encoder; the rest reuse the cached features. The encoder cache hit rate is exposed in metrics.

Per-model preprocessing

Each MM model registers a preprocessor via MULTIMODAL_REGISTRY.register_processor(...) in its model file. The preprocessor handles:

• Image / audio / video resizing, normalization, patching
• Prompt tokenization with image/audio token placeholders
• Computing the number of model-side tokens each MM input contributes (used by the budget)

Examples to look at:

• vllm/model_executor/models/llava.py — classic vision-language
• vllm/model_executor/models/qwen2_vl.py and qwen3_vl.py — Qwen-VL family
• vllm/model_executor/models/whisper.py, whisper_causal.py — speech encoder
• vllm/model_executor/models/voxtral_realtime.py — streaming audio
• vllm/model_executor/models/qwen2_5_omni_thinker.py, qwen3_omni_moe_thinker.py — omni-modal MoE
• vllm/model_executor/models/glm4_1v.py, glm4v.py — GLM vision variants
• vllm/model_executor/models/molmo.py, molmo2.py — Molmo
• vllm/model_executor/models/paddleocr_vl.py, deepseek_ocr.py, glm_ocr.py, nemotron_parse.py — OCR/document understanding

Speech-to-text frontend

The realtime / streaming speech endpoints (/v1/realtime, /v1/audio/transcriptions) are served by:

• vllm/entrypoints/openai/speech_to_text/ — REST handlers
• vllm/entrypoints/openai/realtime/ — WebSocket handlers
• vllm/config/speech_to_text.py — config dataclass

Audio-decoding utilities live in vllm/multimodal/audio.py and vllm/multimodal/media/.

Key source files

File	Purpose
vllm/multimodal/registry.py	The MULTIMODAL_REGISTRY
vllm/multimodal/inputs.py	MM input dataclasses
vllm/multimodal/parse.py	OpenAI-style request parser
vllm/multimodal/encoder_budget.py	MultiModalBudget
vllm/multimodal/cache.py	MMCache
vllm/multimodal/video.py	Video frame loader
vllm/v1/core/encoder_cache_manager.py	Encoder cache scheduling
vllm/v1/worker/encoder_cudagraph.py	Encoder CUDA-graph capture
vllm/distributed/ec_transfer/	Encoder-cache transfer for disaggregated MM serving

Entry points for modification

• Add a new MM model: write the model in vllm/model_executor/models/, register a preprocessor with MULTIMODAL_REGISTRY, mark it SupportsMultiModal, and add tests in tests/multimodal/ and tests/models/.
• Add a new modality: extend MultiModalFeatureSpec, add a parser branch in parse.py, and route through the registry.
• Tune encoder admission: subclass MultiModalBudget or tweak its plug-in point in the scheduler.
• Add an encoder transport: implement an ECConnector (vllm/distributed/ec_transfer/ec_connector/) and register via the factory.

For how MM features end up in a forward pass, see Executors and workers. For how scheduling enforces the encoder budget, see Scheduler.