Model executor and the model zoo

Active contributors: Isotr0py, Roger Wang, Cyrus Leung, Harry Mellor, Woosuk Kwon.

Purpose

vllm/model_executor/ contains everything the worker needs to actually compute a model: the layer library, the model implementations, the registry that maps Hugging Face architecture strings to those implementations, the weight loaders, and the per-platform offloader. It is by far the largest area of the repo (~290 model files alone).

Directory layout

vllm/model_executor/
├── __init__.py
├── custom_op.py            # CustomOp base — dispatch between native/triton/eager
├── parameter.py            # vLLMParameter (sharding-aware tensors)
├── utils.py
├── kernels/                # Python wrappers around csrc kernels (TBE, etc.)
├── warmup/                 # Pre-warm CUTLASS / Triton autotuners
├── offloader/              # Layer-level offloading helpers
├── model_loader/           # Weight loaders
│   ├── __init__.py         # get_model_loader factory
│   ├── base_loader.py      # BaseModelLoader
│   ├── default_loader.py   # safetensors / pytorch bin / fastsafetensors
│   ├── tensorizer.py / tensorizer_loader.py
│   ├── runai_streamer_loader.py
│   ├── sharded_state_loader.py
│   ├── bitsandbytes_loader.py
│   ├── gguf_loader.py
│   ├── dummy_loader.py
│   ├── ep_weight_filter.py # Drops experts not owned by this rank
│   ├── reload/             # Live weight reload
│   ├── utils.py
│   └── weight_utils.py     # Shard reading, format detection (61 KB)
├── layers/                 # Reusable layer library
│   ├── linear.py           # ColumnParallelLinear, RowParallelLinear, fused variants (~1,500 lines)
│   ├── attention/          # Model-side Attention layer wrapper
│   ├── attention_layer_base.py
│   ├── activation.py / layernorm.py / mla.py / mhc.py / conv.py
│   ├── batch_invariant.py  # Numerics-stable batch-size handling
│   ├── deepseek_compressor.py / deepseek_v4_attention.py
│   ├── kda.py / lightning_attn.py / sparse_attn_indexer.py
│   ├── logits_processor.py
│   ├── pooler/             # Pooling heads (CLS, mean, last, attention)
│   ├── resampler.py        # Perceiver-style resampler
│   ├── rotary_embedding/
│   ├── vocab_parallel_embedding.py
│   ├── fla/                # FLA (linear attention) primitives
│   ├── mamba/              # SSM kernels and ops
│   ├── fused_moe/          # ~30 files of MoE kernels
│   └── quantization/       # ~20+ quantization formats
└── models/                 # 293 model implementations + registry
    ├── registry.py         # Architecture string → class
    ├── interfaces.py       # SupportsLoRA, SupportsPP, SupportsMultiModal, SupportsV0Only, etc.
    ├── interfaces_base.py
    ├── adapters.py
    ├── utils.py
    ├── vision.py
    ├── module_mapping.py
    ├── transformers/       # Bridges to upstream HF modeling code
    └── *.py                # one or more files per architecture (llama, qwen3_vl, deepseek_v4, etc.)

Key abstractions

Abstraction	File	Role
ModelRegistry	vllm/model_executor/models/registry.py	Architecture-string → implementation class
SupportsLoRA, SupportsPP, SupportsMultiModal, SupportsV0Only, SupportsTranscription, SupportsScoring, etc.	interfaces.py	Capability mixins models declare to opt into features
Attention layer	vllm/model_executor/layers/attention/	Bridges model layers to the active attention backend
ColumnParallelLinear, RowParallelLinear, MergedColumnParallelLinear, QKVParallelLinear	linear.py	TP-aware linear layers used everywhere
FusedMoE	vllm/model_executor/layers/fused_moe/layer.py	Drop-in MoE layer with kernel selection inside
RotaryEmbedding	vllm/model_executor/layers/rotary_embedding/	Family of RoPE variants (default, NTK, dynamic, YaRN)
VocabParallelEmbedding	vllm/model_executor/layers/vocab_parallel_embedding.py	TP-sharded vocab embedding + LM head
LogitsProcessor	vllm/model_executor/layers/logits_processor.py	Logits transform layer (TP, sampling temp, etc.)
Pooler	vllm/model_executor/layers/pooler/	Embedding/score/classify heads
BaseModelLoader	vllm/model_executor/model_loader/base_loader.py	Loader interface (download → state dict → module)
vLLMParameter	vllm/model_executor/parameter.py	Tensor + sharding metadata
CustomOp	vllm/model_executor/custom_op.py	Multi-backend op dispatcher

How a model is loaded

sequenceDiagram
    participant EC as EngineCore
    participant Ex as Executor
    participant W as Worker
    participant ML as Model loader
    participant Reg as ModelRegistry
    participant M as Model class

    EC->>Ex: collective_rpc("init_device") + init_kv_cache_specs
    Ex->>W: collective_rpc("load_model")
    W->>Reg: get(architectures[0])
    Reg-->>W: model class (e.g., LlamaForCausalLM)
    W->>M: instantiate(vllm_config)
    W->>ML: get_model_loader(load_config).load_weights(model, model_config)
    ML->>ML: download (HF hub / S3 / local)
    ML->>ML: parse safetensors / GGUF / bnb
    ML->>M: load weights into vLLMParameters (sharded)
    M-->>W: ready

The registry walks the model's architectures list (from the HF config) and picks the first matching entry. If the architecture is unknown, it falls back to the transformers integration in vllm/model_executor/models/transformers/, which wraps an upstream modeling class.

Model registration

Each implementation file typically defines:

• A Model class (e.g., LlamaModel) — the transformer body.
• A ForCausalLM / ForConditionalGeneration / etc. wrapper that adds the LM head and the forward signature vLLM expects.
• One or more capability mixins from interfaces.py (SupportsLoRA, SupportsPP, SupportsMultiModal).
• A weight-loading hook (load_weights) that maps HF parameter names to vLLM-internal names.

Some architectures have multiple files for variants (deepseek_v2.py, deepseek_v4.py, deepseek_eagle.py, deepseek_eagle3.py, deepseek_mtp.py, deepseek_v4_mtp.py, deepseek_ocr.py).

The transformers shim

For architectures that don't (yet) have a hand-written vLLM implementation, vllm/model_executor/models/transformers/ provides a generic adapter that uses upstream transformers modeling code with vLLM's KV cache and parallelism plumbing. This is what makes the "200+ supported models" claim real — many of them rely on the shim until a hand-written port lands.

Weight loaders

Each loader handles a specific source format / strategy:

Loader	Source
default_loader.py	safetensors and .bin from local disk or HF hub
bitsandbytes_loader.py	bnb 4-bit / 8-bit
gguf_loader.py	GGUF (llama.cpp format)
tensorizer_loader.py	CoreWeave Tensorizer single-file format
runai_streamer_loader.py	Run:ai Streamer (S3/object storage)
sharded_state_loader.py	Pre-sharded state dicts
dummy_loader.py	All-zero weights for kernel benchmarking
reload/	Hot weight reload (used by RL frameworks)

Selection happens in model_loader/__init__.py::get_model_loader(LoadConfig). The active loader is derived from LoadConfig.load_format (auto, pt, safetensors, gguf, bitsandbytes, tensorizer, runai_streamer, sharded_state, dummy).

Quantization

vllm/model_executor/layers/quantization/ houses ~20 quantization backends. Each defines a QuantizationConfig, layer wrappers, and the kernels they call. See Quantization (feature).

Fused MoE

vllm/model_executor/layers/fused_moe/ is the MoE gym. Highlights:

• layer.py — the user-facing FusedMoE module.
• fused_moe.py — Triton-based fused-MoE forward (~2,000 lines).
• flashinfer_cutlass_moe.py, fused_marlin_moe.py, fused_humming_moe.py, fused_batched_moe.py, cpu_fused_moe.py, rocm_aiter_fused_moe.py — alternative backends.
• modular_kernel.py — the framework that lets a FusedMoE layer pick between backends per call.
• prepare_finalize/, experts/, runner/, router/ — sub-systems for token routing and expert dispatch.
• lora_experts_mixin.py / lora_context.py — MoE LoRA support.
• routed_experts_capturer.py — captures per-step expert routing for telemetry / batch invariance.

EPLB (vllm/distributed/eplb/) reroutes experts across ranks under load. Elastic-EP (vllm/distributed/elastic_ep/) lets the topology change at runtime.

Key source files

File	Purpose
vllm/model_executor/models/registry.py	Architecture-string registry (~55 KB)
vllm/model_executor/models/interfaces.py	Capability mixins
vllm/model_executor/layers/linear.py	TP-aware linear primitives
vllm/model_executor/layers/fused_moe/layer.py	FusedMoE
vllm/model_executor/layers/quantization/__init__.py	Quantization registry
vllm/model_executor/layers/rotary_embedding/__init__.py	RoPE variants
vllm/model_executor/model_loader/weight_utils.py	Format detection, shard reading
vllm/model_executor/custom_op.py	CustomOp dispatcher
vllm/model_executor/parameter.py	vLLMParameter

Entry points for modification

• Add a model: drop a vllm/model_executor/models/<arch>.py, register in registry.py, declare the right capability mixins, and add a smoke test under tests/models/.
• Add a layer: prefer vllm/model_executor/layers/<thing>.py plus a CustomOp if it has a kernel.
• Add a weight loader: subclass BaseModelLoader, register in model_loader/__init__.py.
• Add a quantization format: see quantization (feature).

For the kernels these layers call, see csrc/ — and the Python wrappers at vllm/_custom_ops.py. For how the model is actually invoked, see Executors and workers.