Architecture

vLLM is structured as a layered pipeline. A request enters at the front-end (HTTP server or in-process LLM), is converted into engine-core requests, gets queued in the scheduler, is allocated KV cache blocks, executed by one or more workers (each running a model on a device), sampled into tokens, and finally streamed back to the client.

Process model

A typical vllm serve deployment uses three classes of processes:

graph TB
    subgraph Client
        C[HTTP / gRPC client]
    end

    subgraph "API server process(es)"
        AS[FastAPI app<br/>vllm/entrypoints/openai/api_server.py]
        AC[AsyncLLM client<br/>vllm/v1/engine/async_llm.py]
    end

    subgraph "EngineCore process(es)"
        EC[EngineCore loop<br/>vllm/v1/engine/core.py]
        SCH[Scheduler<br/>vllm/v1/core/sched/scheduler.py]
        KV[KVCacheManager<br/>vllm/v1/core/kv_cache_manager.py]
    end

    subgraph "Worker process(es)"
        W0[Worker rank 0<br/>vllm/v1/worker/gpu_worker.py]
        W1[Worker rank 1]
        Wn[Worker rank N]
    end

    C -->|HTTP| AS
    AS --> AC
    AC <-->|ZMQ msgpack| EC
    EC --> SCH
    SCH --> KV
    EC <-->|RPC| W0
    EC <-->|RPC| W1
    EC <-->|RPC| Wn
    W0 -->|NCCL/Gloo| W1
    W1 -->|NCCL/Gloo| Wn

• The API server translates HTTP/gRPC into EngineCoreRequest messages and streams tokens back. It can be replicated (--api-server-count N) for high-fanout deployments.
• The EngineCore runs the scheduler and dispatches steps. There is one EngineCore per data-parallel replica; with --data-parallel-size > 1 you get multiple.
• Workers hold model weights and run the forward pass. Tensor/pipeline parallelism shards weights across workers; the executor (MultiprocExecutor, RayDistributedExecutor, UniProcExecutor, or ExternalLauncher) coordinates them via collective RPC.

The wire format between the API server and EngineCore is msgpack over ZMQ. The serializers live in vllm/v1/serial_utils.py and the message types in vllm/v1/engine/__init__.py (EngineCoreRequest, EngineCoreOutput, EngineCoreOutputs).

Layered view

graph TD
    A[Frontends:<br/>OpenAI server · LLM offline · gRPC · Anthropic · SageMaker · MCP]
    B[Engine clients:<br/>AsyncLLM · LLMEngine · EngineCoreClient]
    C[EngineCore:<br/>scheduler loop · request lifecycle · KV manager · structured output]
    D[Executor:<br/>MultiprocExecutor · RayExecutor · UniProc · external_launcher]
    E[Workers + GPUModelRunner:<br/>forward · CUDA graphs · sampling · spec decode]
    F[Layers:<br/>linear · attention · MoE · layernorm · rotary · vocab embedding]
    G[Kernels:<br/>FlashAttention · FlashInfer · Triton · CUTLASS · Marlin · custom all-reduce]
    H[Platforms:<br/>CUDA · ROCm · CPU · XPU · TPU plugin · Gaudi plugin · etc.]

    A --> B --> C --> D --> E --> F --> G
    G --> H
    F --> H

Each layer corresponds to a directory or set of directories in the repo:

Layer	Code path
Frontends	vllm/entrypoints/
Engine clients	vllm/v1/engine/{async_llm,llm_engine,core_client}.py
EngineCore	vllm/v1/engine/core.py, vllm/v1/core/
Executors	vllm/v1/executor/
Workers / runners	vllm/v1/worker/
Layers	vllm/model_executor/layers/
Kernels (Python)	vllm/_custom_ops.py, vllm/_aiter_ops.py, vllm/_xpu_ops.py
Kernels (C++/CUDA)	csrc/
Platforms	vllm/platforms/

Request lifecycle

sequenceDiagram
    autonumber
    participant Cli as Client
    participant API as OpenAI server
    participant Async as AsyncLLM
    participant Core as EngineCore loop
    participant Sched as Scheduler
    participant KV as KVCacheManager
    participant Exec as Executor + Worker
    participant Smp as Sampler / Logits

    Cli->>API: POST /v1/chat/completions
    API->>Async: generate(prompt, sampling_params)
    Async->>Async: InputProcessor: tokenize, build EngineCoreRequest
    Async->>Core: send EngineCoreRequest via ZMQ
    Core->>Sched: add_request
    loop scheduler step (each engine iteration)
        Sched->>KV: allocate / append blocks (paged attention)
        Sched->>Exec: SchedulerOutput (running set, new tokens)
        Exec->>Exec: GPUModelRunner.execute_model (forward)
        Exec->>Smp: sample tokens (greedy/top-p/etc.)
        Smp-->>Core: ModelRunnerOutput
        Core->>Sched: update_from_output
        Core->>Async: EngineCoreOutputs
    end
    Async->>Async: OutputProcessor: detokenize, accumulate
    Async-->>API: stream RequestOutput chunks
    API-->>Cli: SSE / JSON response

The "step" in the loop above is the unit of scheduling. In one step the scheduler picks a budget of running requests, the KV cache manager assigns physical blocks (reusing prefix-cached blocks when possible), the executor runs a single forward pass, the sampler picks the next token(s), and the result is fed back to the scheduler. Speculative decoding, structured-output grammars, and chunked prefill all happen inside the step.

Configuration object

All configuration is centralized in VllmConfig (vllm/config/vllm.py). It is a dataclass that aggregates ~20 sub-configs:

Sub-config	File	Purpose
ModelConfig	vllm/config/model.py	Model identifier, dtype, max length, tokenizer, runner kind
CacheConfig	vllm/config/cache.py	Block size, GPU/CPU memory ratio, prefix caching switches
ParallelConfig	vllm/config/parallel.py	TP/PP/DP/EP sizes, executor backend, EPLB, batch invariance
SchedulerConfig	vllm/config/scheduler.py	Max running seqs, batched tokens, async scheduling
LoRAConfig	vllm/config/lora.py	LoRA ranks, max LoRAs
SpeculativeConfig	vllm/config/speculative.py	Spec decode method (n-gram, EAGLE, DFlash, MTP, draft model)
StructuredOutputsConfig	vllm/config/structured_outputs.py	Backend (xgrammar, guidance, outlines, lm-format-enforcer)
KVTransferConfig / ECTransferConfig	vllm/config/kv_transfer.py, ec_transfer.py	Disaggregated prefill, KV/EC connectors
CompilationConfig	vllm/config/compilation.py	torch.compile mode, CUDA graph mode, custom passes
ObservabilityConfig	vllm/config/observability.py	OTLP tracing, KV-cache metrics
MultiModalConfig	vllm/config/multimodal.py	MM token limits, encoder budget, video/audio settings

VllmConfig is built from EngineArgs (vllm/engine/arg_utils.py, ~2,500 lines of argument parsing) and is read by every other component via set_current_vllm_config / get_current_vllm_config.

Where to look first

• The OpenAI server entry point: vllm/entrypoints/openai/api_server.py
• The offline LLM API: vllm/entrypoints/llm.py
• The async engine loop: vllm/v1/engine/async_llm.py
• The EngineCore process: vllm/v1/engine/core.py
• The scheduler: vllm/v1/core/sched/scheduler.py
• The KV cache manager: vllm/v1/core/kv_cache_manager.py
• The GPU worker: vllm/v1/worker/gpu_worker.py
• The GPU model runner: vllm/v1/worker/gpu_model_runner.py (~7,000 lines — the heart of the forward pass)

For deeper dives, see Engine core, Scheduler, KV cache, Executors and workers, and Attention backends.