vllm-project/vllm
Glossary
vLLM-specific terminology. For background concepts (transformers, attention, KV cache, etc.) see external resources.
| Term | Meaning |
|---|---|
| PagedAttention | The original vLLM contribution: managing the attention KV cache as fixed-size pages so that requests share physical memory and fragmentation is avoided. |
| V1 / V1 engine | The current engine, in vllm/v1/. Replaced the V0 engine in 2024–2025. Multi-process by default, async-first, with a redesigned scheduler and KV cache manager. |
| EngineCore | The process that runs the scheduler loop and dispatches forward passes to workers (vllm/v1/engine/core.py). One per data-parallel replica. |
| AsyncLLM | The asyncio-friendly engine client used by HTTP servers (vllm/v1/engine/async_llm.py). |
| LLMEngine | A synchronous wrapper preserved for the offline LLM API (vllm/v1/engine/llm_engine.py). |
| Executor | Abstraction over the worker fleet (vllm/v1/executor/abstract.py). Backends: MultiprocExecutor (mp), RayDistributedExecutor (ray), RayExecutorV2, UniProcExecutor (uni), ExecutorWithExternalLauncher. |
| Worker | A process that holds part of the model and runs the forward pass (vllm/v1/worker/gpu_worker.py, cpu_worker.py, xpu_worker.py). |
| GPUModelRunner | The class inside a worker that orchestrates the forward pass, CUDA graphs, sampling, KV connector hooks, and spec decode (vllm/v1/worker/gpu_model_runner.py). |
| SchedulerOutput | The struct produced by the scheduler each step describing which requests run, how many tokens each gets, and which KV blocks to use (vllm/v1/core/sched/output.py). |
| KVCacheManager | Allocates and frees KV cache blocks per request, integrating with the prefix cache and KV connectors (vllm/v1/core/kv_cache_manager.py). |
| KVCacheBlock | A fixed-size physical block of KV memory. The block size is configurable via CacheConfig.block_size. |
| Block hash / block hasher | Hash of a token range used to deduplicate identical prefixes across requests for prefix caching (vllm/v1/core/kv_cache_utils.py). |
| Encoder cache | Separate cache for encoder outputs in encoder-decoder and multi-modal models (vllm/v1/core/encoder_cache_manager.py). |
| KV connector | Pluggable transport that pulls/pushes KV blocks between engines (e.g., between a prefill and a decode instance) — vllm/distributed/kv_transfer/kv_connector/. |
| EC connector | Encoder-cache equivalent of a KV connector, used for disaggregated multimodal encoder/decoder setups (vllm/distributed/ec_transfer/). |
| Disaggregated prefill | A deployment pattern where prefill and decode run on different machines and exchange KV blocks via a connector. Image: vllm/distributed/kv_transfer/disagg_prefill_workflow.jpg. |
| Speculative decoding | Drafting tokens with a cheaper model (n-gram, draft model, EAGLE, DFlash, Medusa, MTP) and verifying with the target. Implemented under vllm/v1/spec_decode/. |
| Structured output | Constraining sampling to match a JSON schema, regex, choice list, or grammar. Backends: xgrammar, guidance, outlines, lm-format-enforcer — vllm/v1/structured_output/. |
| Pooling | Non-generative inference: embeddings, classification, scoring, reward modeling. Lives under vllm/entrypoints/pooling/, vllm/v1/pool/, vllm/model_executor/layers/pooler/. |
| MLA (Multi-head Latent Attention) | DeepSeek-style attention where K and V are derived from a low-rank latent. Backends in vllm/v1/attention/backends/mla/. |
| MoE / Fused MoE | Mixture-of-Experts. Fused MoE kernels live in vllm/model_executor/layers/fused_moe/. EPLB = Expert-Parallel Load Balancer. |
| EPLB | Expert-Parallel Load Balancer — re-routes experts across ranks under load (vllm/distributed/eplb/, EPLBConfig). |
| CUDA graph mode | One of NONE, PIECEWISE, FULL_AND_PIECEWISE, FULL. Controls how forward passes are captured into CUDA graphs (vllm/config/compilation.py CUDAGraphMode). |
| Compilation mode | One of NO_COMPILATION, STOCK_TORCH_COMPILE, VLLM_COMPILE. Controls torch.compile usage (vllm/config/compilation.py CompilationMode). |
| Custom op | A torch.Library registration that lets a Python module call into a CUDA/Triton/C++ kernel (vllm/_custom_ops.py, vllm/model_executor/custom_op.py). |
| Attention backend | An implementation of attention selected at runtime: FLASH_ATTN, FLASHINFER, TRITON_ATTN, TREE_ATTN, FLEX_ATTN, ROCM_ATTN, ROCM_AITER_*, CPU_ATTN, MAMBA*, FLASHMLA*, etc. (vllm/v1/attention/backends/registry.py). |
| Tokenizer-mode | auto, slow, mistral, custom — controls the tokenizer implementation; see vllm/tokenizers/. |
| Runner | A high-level model role: generate, pooling, transcription, etc. Defined in vllm/tasks.py. |
| Reasoning parser | Plug-in that strips reasoning tags (e.g., DeepSeek <think>) from the streaming output (vllm/reasoning/). |
| Tool parser | Plug-in that recognizes tool/function calls in model output (vllm/tool_parsers/). |
vllm plugin |
A separate package that registers attention backends, model architectures, platforms, or general hooks via Python entry points. Discovered in vllm/plugins/. |
| Sleep / wake | Putting the executor into a low-memory state and bringing it back. Used to free GPU memory for other workloads. See Executor.sleep / wake_up. |
| Headless mode | Running EngineCore with no API server (--headless). Useful when an external orchestrator owns the HTTP layer. |
| External launcher | Distributed setup where vLLM does not spawn workers itself; the user launches them with torchrun or similar and connects them via ExecutorWithExternalLauncher. |
| Forward context | The thread-local state used during a forward pass (active LoRAs, KV connector metadata, etc.) — vllm/forward_context.py. |
| Kernel config | The KernelConfig dataclass that toggles per-kernel feature flags such as deep-gemm, flashinfer cutlass MoE, etc. — vllm/config/kernel.py. |
Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.