Executors and workers

Active contributors: Nick Hill, Cyrus Leung, youkaichao, Robert Shaw.

Purpose

EngineCore does not run forward passes itself. It hands a SchedulerOutput to an Executor, which fans out to one or more Worker processes. Each worker holds part of the model and is driven by a ModelRunner. This separation is what lets vLLM target tensor/pipeline/data parallelism, Ray-managed clusters, and external launchers from a single engine API.

Directory layout

vllm/v1/executor/
├── abstract.py            # Executor base class (gets_class dispatch)
├── multiproc_executor.py  # MultiprocExecutor (1,235 lines) — the default
├── ray_executor.py        # RayDistributedExecutor (legacy)
├── ray_executor_v2.py     # RayExecutorV2 (default Ray backend going forward)
├── ray_utils.py           # Ray initialization, env propagation
├── ray_env_utils.py
└── uniproc_executor.py    # UniProcExecutor + ExecutorWithExternalLauncher

vllm/v1/worker/
├── worker_base.py         # WorkerBase (sleep/wake, KV init, RPC dispatch)
├── gpu_worker.py          # GPU worker (1,400 lines)
├── gpu_model_runner.py    # The forward orchestrator (~7,070 lines)
├── gpu_input_batch.py     # InputBatch builder for GPUModelRunner
├── gpu_ubatch_wrapper.py  # Sub-batch grouping for piecewise CUDA graphs
├── cpu_worker.py / cpu_model_runner.py
├── xpu_worker.py / xpu_model_runner.py
├── tpu_input_batch.py     # TPU model runner inputs
├── encoder_cudagraph.py   # Encoder-side CUDA graph manager
├── lora_model_runner_mixin.py
├── kv_connector_model_runner_mixin.py
├── ec_connector_model_runner_mixin.py
├── ubatching.py / ubatch_utils.py
├── workspace.py           # Shared scratch tensors
├── block_table.py         # Per-batch block table tensor builder
├── mamba_utils.py
├── cp_utils.py            # Context-parallel utilities
├── dp_utils.py            # Data-parallel utilities
└── utils.py

Key abstractions

Abstraction	File	Role
Executor	vllm/v1/executor/abstract.py	Abstract base; chooses concrete class via Executor.get_class
MultiprocExecutor	vllm/v1/executor/multiproc_executor.py	mp backend — spawns Python subprocesses, talks via pipes
RayDistributedExecutor	vllm/v1/executor/ray_executor.py	ray backend (legacy)
RayExecutorV2	vllm/v1/executor/ray_executor_v2.py	ray backend (when VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1)
UniProcExecutor	vllm/v1/executor/uniproc_executor.py	uni backend — single process, no IPC
ExecutorWithExternalLauncher	vllm/v1/executor/uniproc_executor.py	external_launcher — workers launched by torchrun, etc.
WorkerBase	vllm/v1/worker/worker_base.py	Common worker lifecycle (init, sleep, wake, compile, KV init)
GPUWorker	vllm/v1/worker/gpu_worker.py	GPU specialization
GPUModelRunner	vllm/v1/worker/gpu_model_runner.py	Per-step forward orchestrator
InputBatch	vllm/v1/worker/gpu_input_batch.py	Builds the actual tensors fed into the model
KVOutputAggregator	vllm/distributed/kv_transfer/kv_connector/utils.py	Aggregates per-rank KV connector outputs

Executor selection

# vllm/v1/executor/abstract.py
@staticmethod
def get_class(vllm_config):
    backend = vllm_config.parallel_config.distributed_executor_backend
    if backend == "ray":
        return RayExecutorV2 if envs.VLLM_USE_RAY_V2_EXECUTOR_BACKEND else RayDistributedExecutor
    if backend == "mp": return MultiprocExecutor
    if backend == "uni": return UniProcExecutor
    if backend == "external_launcher": return ExecutorWithExternalLauncher
    if isinstance(backend, str): return resolve_obj_by_qualname(backend)
    if isinstance(backend, type) and issubclass(backend, Executor): return backend

The default is mp for single-host TP, ray for multi-host, uni when world_size == 1, and external_launcher when the user owns process spawning.

Forward pass

sequenceDiagram
    participant Sch as Scheduler
    participant E as Executor
    participant W as WorkerBase (per rank)
    participant MR as GPUModelRunner
    participant M as Model
    participant Smp as Sampler

    Sch->>E: execute_model(SchedulerOutput)
    E->>W: collective_rpc("execute_model", ...)
    W->>MR: execute_model(scheduler_output)
    MR->>MR: build InputBatch (block tables, slot mapping, MM)
    MR->>MR: enter forward_context (LoRA, MM, KV connector meta)
    MR->>M: model.forward(input_ids, kv_caches, attn_metadata, ...)
    M-->>MR: hidden states / logits
    MR->>Smp: sample(logits, sampling_metadata)
    Smp-->>MR: sampled token ids, logprobs
    MR-->>W: ModelRunnerOutput
    W-->>E: RPC response
    E-->>Sch: ModelRunnerOutput

Highlights from GPUModelRunner.execute_model (vllm/v1/worker/gpu_model_runner.py):

• InputBatch construction: pads to the right ubatch size, builds block tables, slot mappings, and attention metadata for the active backend.
• CUDA graph dispatch: depending on CompilationConfig.cuda_graph_mode (PIECEWISE, FULL, FULL_AND_PIECEWISE, NONE), the runner replays a captured graph or runs eager.
• Forward context: vllm/forward_context.py makes per-step state visible to layers (active LoRAs, MM features, KV connector roles).
• Sampling: vllm/v1/sample/sampler.py runs greedy / top-k / top-p / penalties / etc.; RejectionSampler runs spec-decode verification (rejection_sampler.py).
• KV connector hooks: kv_connector_model_runner_mixin.py calls connector.start_load_kv before forward and wait_for_save after.
• Speculative decoding: when configured, the runner produces draft tokens (n-gram, EAGLE, MTP, etc.) for the next step.

CUDA graph mode

vllm/config/compilation.py defines:

• CUDAGraphMode.NONE — eager forward
• CUDAGraphMode.PIECEWISE — replays graphs for static-shape sub-batches; falls back to eager for the rest
• CUDAGraphMode.FULL — captures the entire forward
• CUDAGraphMode.FULL_AND_PIECEWISE — full graph for hot shapes plus piecewise fallback

The dispatcher (vllm/v1/cudagraph_dispatcher.py) selects which graph to replay each step. Encoder-side capture is in vllm/v1/worker/encoder_cudagraph.py.

Sleep / wake

Workers can be put to sleep to free GPU memory:

• WorkerBase.sleep(level) evicts model weights and/or KV cache to host memory, depending on level.
• WorkerBase.wake_up(tags) brings them back. Tags identify which buffers to restore (weights, kv_cache).

This is exposed at the executor level (Executor.sleep / wake_up) and used for warm-spare workflows where a long-running process needs to free GPU memory between batches.

RPC plumbing

Executor.collective_rpc(method, args, kwargs) is the universal way to call into all workers. Implementation differs:

• MultiprocExecutor — Python pipes between the EngineCore process and worker subprocesses. Methods are named or sent as cloudpickle-serialized callables.
• RayExecutor — Ray actor calls; uses vllm/v1/metrics/ray_wrappers.py for stats forwarding.
• UniProc / ExternalLauncher — direct in-process calls.

collective_rpc supports non_block=True to return Futures, used by async scheduling and multi-step pipelines.

Per-platform workers

Platform	Worker	Runner
CUDA	vllm/v1/worker/gpu_worker.py	gpu_model_runner.py
ROCm	(CUDA worker; ROCm-specific paths in runner)	(CUDA runner with ROCm dispatch)
CPU	cpu_worker.py	cpu_model_runner.py
XPU	xpu_worker.py	xpu_model_runner.py
TPU	(plugin, out-of-tree)	uses tpu_input_batch.py from the in-tree pieces

Entry points for modification

• New executor backend: subclass Executor, implement _init_executor, collective_rpc, check_health. Register via the distributed_executor_backend string or by passing a class.
• New worker capability: add a method on WorkerBase and call it via collective_rpc("your_method", ...) from the executor.
• Per-step state in the runner: extend InputBatch (gpu_input_batch.py) and consume in GPUModelRunner.execute_model.
• Custom CUDA graph behavior: tweak vllm/v1/cudagraph_dispatcher.py and vllm/compilation/cuda_graph.py.

For how layers and kernels are arranged, see Model executor. For how attention metadata is wired into the forward pass, see Attention backends.