vLLM

vLLM is a high-throughput, memory-efficient inference and serving engine for large language models. It pioneered PagedAttention, a virtual-memory–style allocator for the attention KV cache, and has grown into one of the most active open-source AI projects, with over 2,000 contributors and 200+ supported model architectures.

What this project is

The repository contains a single Python package (vllm) plus a substantial C++/CUDA kernel library (csrc/). It exposes three primary front-ends:

• An offline batch API (LLM in vllm/entrypoints/llm.py) for embedding inference into Python programs.
• An OpenAI-compatible HTTP server (vllm.entrypoints.openai.api_server) launched via the vllm serve CLI, with parallel implementations for Anthropic Messages, gRPC, and SageMaker.
• A CLI (vllm) defined in vllm/entrypoints/cli/main.py that dispatches serve, bench, chat, complete, run-batch, collect-env, and launch subcommands.

Underneath, all of these front-ends route requests into the V1 engine (vllm/v1/), which is the modern, asynchronous, multi-process scheduler/executor introduced in October 2024. The legacy V0 engine has been removed; everything goes through V1.

What it is good at

• Throughput: continuous batching, chunked prefill, prefix caching, and PagedAttention let a single GPU sustain many concurrent requests.
• Hardware breadth: NVIDIA GPUs are the primary target, but the same Python stack runs on AMD ROCm, Intel XPU, x86 CPU, ARM CPU, PowerPC, TPU (via plugin), Gaudi (via plugin), and other accelerators.
• Model breadth: ~290 model implementations live in vllm/model_executor/models/, covering decoder-only LLMs, MoE, hybrid SSM/attention, multi-modal (vision, audio, video), embedding, and reward models.
• Production features: tensor/pipeline/data/expert/context parallelism, LoRA, quantization (FP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO), structured outputs (xgrammar, guidance, outlines, lm-format-enforcer), tool calling, reasoning parsers, KV cache offloading, disaggregated prefill/decode, and speculative decoding (n-gram, suffix, EAGLE, DFlash, Medusa, MTP).

Quick links

• Architecture — process model, request lifecycle, layered diagram
• Getting started — install, run a model, hit the API
• Glossary — vLLM-specific vocabulary
• How to contribute — workflow, tests, conventions
• V1 engine internals — the inner loop
• Scheduler — how requests get batched
• KV cache & paged attention — block allocation, prefix caching
• Attention backends — FlashAttention, FlashInfer, MLA, Triton, ROCm AITER
• Model executor — layers, model registry, weight loading
• API server — OpenAI / Anthropic / gRPC / SageMaker
• By the numbers — repository statistics
• Lore — how vLLM evolved

Repository roots at a glance

Directory	Purpose
vllm/	The Python package: engine, scheduler, model implementations, layers, executors, entry points
vllm/v1/	The V1 engine — scheduler, KV cache manager, async LLM, executor backends, GPU model runner
vllm/model_executor/	Model architectures (models/), reusable layers (layers/), weight loaders (model_loader/)
csrc/	C++/CUDA kernels: attention, cache, quantization (CUTLASS, Marlin, MoE), custom all-reduce, layernorm
tests/	~1,170 pytest files covering models, engine, kernels, serving, distributed, quantization
benchmarks/	Throughput, latency, and serving benchmark harnesses
docs/	The user-facing site published at https://docs.vllm.ai
examples/	Runnable examples for offline inference, online serving, LoRA, multi-modal, distributed
requirements/	Pinned and unpinned dependency lists, split per platform (CUDA, ROCm, CPU, TPU, XPU)
tools/	Build and developer tooling (e.g., wheel size checks, model registry validators)
cmake/, CMakeLists.txt, setup.py	Build system that compiles csrc/ into the _C extension (and platform-specific extensions like _rocm_C)
.buildkite/	The primary CI pipeline (Buildkite). GitHub Actions in .github/workflows/ handle pre-commit and lighter checks