Engine core (V1)

Active contributors: Nick Hill, Robert Shaw, Cyrus Leung, Woosuk Kwon.

Purpose

The EngineCore process is vLLM's inference loop. It owns the scheduler, the KV cache manager, the structured-output manager, and the executor. Front-ends never call into the model directly; they always go through an EngineCore. There is one EngineCore per data-parallel replica.

Directory layout

vllm/v1/engine/
├── __init__.py            # EngineCoreRequest / Output / Outputs structs (msgspec)
├── core.py                # EngineCore class + EngineCoreProc loop (2,145 lines)
├── core_client.py         # Client side: AsyncMPClient, MPClient, etc. (2,300 lines)
├── async_llm.py           # AsyncLLM (1,065 lines)
├── llm_engine.py          # LLMEngine (legacy sync wrapper, 424 lines)
├── coordinator.py         # DP coordinator
├── input_processor.py     # Tokenization + EngineCoreRequest builder (650 lines)
├── output_processor.py    # Detokenization + RequestOutput builder (1,000 lines)
├── detokenizer.py         # Per-request detokenizer
├── logprobs.py
├── parallel_sampling.py   # Parent-request bookkeeping for n>1 sampling
├── tensor_ipc.py          # SHM-based tensor transfer for spec decode draft hidden states
├── exceptions.py          # EngineDeadError, EngineGenerateError
└── utils.py               # CoreEngineProcManager, launch_core_engines, ZMQ helpers (1,500 lines)

Key abstractions

Abstraction	File	Role
EngineCore	vllm/v1/engine/core.py	The inner loop: holds the scheduler + executor
EngineCoreProc	vllm/v1/engine/core.py	Process wrapper: runs EngineCore.step() forever, talks ZMQ
EngineCoreRequest	vllm/v1/engine/__init__.py	Request msg from front-end to EngineCore (msgspec)
EngineCoreOutput	vllm/v1/engine/__init__.py	Per-request output chunk
EngineCoreOutputs	vllm/v1/engine/__init__.py	Per-step output bundle (one per scheduler step)
EngineCoreClient	vllm/v1/engine/core_client.py	Client used by AsyncLLM/LLMEngine to talk to one or more EngineCoreProcs
AsyncLLM	vllm/v1/engine/async_llm.py	Asyncio-friendly engine client; what HTTP servers use
LLMEngine	vllm/v1/engine/llm_engine.py	Legacy synchronous façade kept for from vllm import LLMEngine
InputProcessor	vllm/v1/engine/input_processor.py	Builds EngineCoreRequest from EngineInput
OutputProcessor	vllm/v1/engine/output_processor.py	Builds RequestOutput from EngineCoreOutput
CoreEngineProcManager	vllm/v1/engine/utils.py	Spawns EngineCore subprocesses with handshake

How a step runs

graph TD
    Q[ZMQ input queue<br/>EngineCoreRequest, ABORT, UTILITY, START_DP_WAVE]
    A[EngineCoreProc input thread]
    L[EngineCore loop]
    SC[scheduler.schedule]
    SR[KV transfer hooks]
    EX[executor.execute_model]
    SP[executor.sample_tokens]
    UP[scheduler.update_from_output]
    OUT[ZMQ output queue<br/>EngineCoreOutputs]

    Q --> A --> L
    L --> SC --> SR --> EX --> SP --> UP --> L
    UP --> OUT

The loop in EngineCore.run_busy_loop/step (depending on async-scheduling) does roughly:

1. Drain incoming requests from the input thread (calls EngineCore.add_request, abort_request, start_dp_wave).
2. Schedule with Scheduler.schedule() — produces a SchedulerOutput.
3. Forward + sample via the Executor (execute_model then sample_tokens) — produces a ModelRunnerOutput.
4. Update state via Scheduler.update_from_output — emits per-request EngineCoreOutputs.
5. Publish an EngineCoreOutputs bundle on the output ZMQ socket.

When async scheduling is enabled (SchedulerConfig.async_scheduling), step 3 is non-blocking and the next schedule() overlaps with the current forward. Implementation lives in vllm/v1/core/sched/async_scheduler.py.

Process model

graph LR
    subgraph "Front-end process(es)"
        AL[AsyncLLM]
        OP[OutputProcessor]
    end

    subgraph "EngineCore process"
        IT[Input thread]
        EC[EngineCore loop]
        SC[Scheduler]
        EX[Executor handle]
    end

    subgraph "Worker process(es)"
        WB[WorkerBase]
        MR[GPUModelRunner]
        Mod[Model]
    end

    AL -->|EngineCoreRequest<br/>ZMQ| IT --> EC
    EC -->|collective_rpc| WB
    WB --> MR --> Mod
    MR --> WB
    WB -->|RPC return| EC
    EC -->|EngineCoreOutputs<br/>ZMQ| OP --> AL

Communication between AsyncLLM and EngineCore is msgpack over ZMQ. The serializer is vllm/v1/serial_utils.py::MsgpackEncoder/Decoder, customized to handle torch tensors (via tensor_ipc.py), msgspec structs, and pydantic dataclasses.

Data parallelism

When parallel_config.data_parallel_size > 1, launch_core_engines (in utils.py) starts multiple EngineCore processes. They are coordinated through vllm/v1/engine/coordinator.py (DPCoordinator) which routes requests round-robin or via the external load balancer modes:

• Internal LB (default) — coordinator distributes requests to engines.
• External LB (--data-parallel-external-lb) — the caller picks the engine.
• Hybrid LB (--data-parallel-hybrid-lb) — combines DP across boxes plus a shared LB.

DP also introduces "waves": current_wave on EngineCoreRequest is used to ensure a request that arrives during a wave handoff lands in the right epoch.

Pause / resume / sleep

EngineCore supports several lifecycle operations triggered as UTILITY messages:

• Pause/resume generation (pause_generation / resume_generation) with modes abort, wait, keep — see PauseMode in vllm/v1/engine/__init__.py.
• Sleep / wake the executor to free GPU memory (Executor.sleep / wake_up in vllm/v1/executor/abstract.py).
• Reconfigure distributed — used by elastic-EP scaling (ReconfigureDistributedRequest).

Multi-engine bookkeeping

When more than one EngineCore is alive (data parallel), the front-end may need to know which engine produced a given output. EngineCoreOutputs.engine_index carries that info, and the finished_req_ids_dict map (gated by include_finished_set) lets the client GC its bookkeeping efficiently.

Integration points

• Inputs from vllm/v1/engine/input_processor.py (front-end side).
• Outputs to vllm/v1/engine/output_processor.py (front-end side).
• Scheduling delegated to Scheduler.
• KV management delegated to KVCacheManager.
• Forward pass delegated to the Executor.
• Metrics via vllm/v1/metrics/loggers.py (StatLoggerManager) and Prometheus (vllm/v1/metrics/prometheus.py).
• Tracing via vllm/tracing.py (OTLP).

Entry points for modification

• To add a new EngineCore RPC, extend EngineCoreRequestType in vllm/v1/engine/__init__.py, route it in EngineCoreProc._handle_input, and add a handler on EngineCore.
• To change how requests reach the scheduler, modify EngineCore.add_request.
• To add a new output channel (besides the default ZMQ one), wrap EngineCoreClient.
• To intercept inputs or outputs without forking the engine, add a StatLogger plug-in (loaded via load_stat_logger_plugin_factories) — they get every step's IterationStats.