Scheduler

Active contributors: Cyrus Leung, Cody Yu, Andreas Karatzas, Nick Hill.

Purpose

The scheduler decides, every engine step, which requests advance, how many tokens each request runs (prefill or decode), and which KV cache blocks back them. It implements continuous batching, chunked prefill, prefix-cache reuse, structured-output budget enforcement, KV-connector handoff, and (when enabled) async scheduling.

Directory layout

vllm/v1/core/sched/
├── __init__.py
├── interface.py        # SchedulerInterface (abstract) + PauseState
├── scheduler.py        # The Scheduler implementation (2,308 lines)
├── async_scheduler.py  # AsyncScheduler subclass (overlaps schedule with execute)
├── output.py           # SchedulerOutput, NewRequestData, CachedRequestData, GrammarOutput
├── request_queue.py    # FIFO / priority request queues
└── utils.py            # check_stop, remove_all helpers

Key abstractions

Abstraction	File	Role
SchedulerInterface	vllm/v1/core/sched/interface.py	Abstract base
Scheduler	vllm/v1/core/sched/scheduler.py	The concrete scheduler
AsyncScheduler	vllm/v1/core/sched/async_scheduler.py	Variant that schedules step N+1 while step N executes
SchedulerOutput	vllm/v1/core/sched/output.py	What the scheduler emits each step (running set, blocks, MM hashes, grammar)
RequestQueue	vllm/v1/core/sched/request_queue.py	FIFO and priority queues for waiting requests
Request	vllm/v1/request.py	The scheduler's view of a request
RequestStatus	vllm/v1/request.py	Enum: WAITING, RUNNING, PREEMPTED, FINISHED, etc.

How a step is scheduled

graph TD
    Wait[Waiting queue<br/>FIFO or priority]
    Run[Running list]
    Sched[Scheduler.schedule]
    Budget[Budget:<br/>max_num_running_reqs<br/>max_num_scheduled_tokens<br/>token_budget]
    KV[KVCacheManager.allocate]
    PC[Prefix cache lookup<br/>BlockHash chain]
    GR[StructuredOutputManager.grammar_step]
    Pre[Preempt running if needed]
    Out[SchedulerOutput<br/>NewRequestData / CachedRequestData / FinishedRequest]

    Wait -->|admit if budget allows| Sched
    Run --> Sched
    Sched --> Budget
    Sched --> KV --> PC
    Sched --> GR
    Sched --> Pre
    Sched --> Out

Each call to Scheduler.schedule():

1. Walks the running list in priority order. For each request, it computes how many additional tokens to compute this step (decoding step = 1 + draft tokens; prefill chunk size for prefill stage), respecting the global token budget.
2. Tries to admit waiting requests while there's slack: it consults the KV cache manager to allocate blocks (with prefix-cache reuse) and grammar manager to make sure structured-output state is ready.
3. Preempts running requests whose KV blocks must be evicted to admit higher-priority work — preempted requests go back to the head of the waiting queue.
4. Builds a SchedulerOutput: lists of NewRequestData (just-admitted), CachedRequestData (continuing), and finished_req_ids (terminal in this step), plus the per-request token budgets and KV block layouts.
5. Records EngineCoreEvents (QUEUED, SCHEDULED, PREEMPTED) used by stats loggers.

After the executor returns a ModelRunnerOutput, Scheduler.update_from_output:

1. Applies the new tokens to each running Request.
2. Detects stop strings, max-tokens, repetition, and other terminal conditions via check_stop.
3. Releases KV blocks for finished requests (KVCacheManager.free).
4. Returns EngineCoreOutputs to the EngineCore loop.

Scheduling policies

SchedulerConfig.policy chooses the request queue:

• fcfs — first-come-first-serve.
• priority — priority queue keyed on Request.priority (set per-request).

SchedulerConfig.scheduling_policy (FCFS, etc.) lives in vllm/v1/core/sched/request_queue.py::SchedulingPolicy.

Chunked prefill

Long prompts are broken into chunks so that decode requests in the same batch are not starved. The chunk size is dynamic: the scheduler computes tokens_remaining = token_budget - decode_tokens and assigns it to the next prefill request. Chunked prefill is the default; it can be disabled via config but only at significant throughput cost.

Prefix caching

When a request is admitted, the scheduler asks the KV cache manager for its prefix-cached blocks: any leading hash-matched block sequence is reused, and only the tail is computed. Implementation:

• Hashing in vllm/v1/core/kv_cache_utils.py (BlockHash, get_request_block_hasher).
• Reuse in KVCacheManager.allocate via KVCacheCoordinator.
• Per-step prefix-cache stats on SchedulerStats.prefix_cache_stats.

See KV cache and prefix caching.

KV connectors

Disaggregated prefill/decode is integrated at the scheduler level. The scheduler:

1. Asks the KV connector (e.g., NIXL, Redis, Mooncake) whether KV for a request is already on a remote prefill instance.
2. If so, schedules a "load" step that pulls the KV blocks before the first decode forward pass.
3. If outbound (we are the prefill side), schedules a "save" step that pushes KV after each layer.

KVConnectorBase_V1 lives in vllm/distributed/kv_transfer/kv_connector/v1/base.py; the scheduler's hooks are in Scheduler.schedule and update_from_output. Per-step connector metadata is KVConnectorMetadata.

Async scheduling

SchedulerConfig.async_scheduling = True swaps in AsyncScheduler (vllm/v1/core/sched/async_scheduler.py), which calls executor.execute_model(..., non_block=True) and immediately starts scheduling the next step. The previous step's Future[ModelRunnerOutput] is awaited at the start of update_from_output. This roughly doubles scheduler concurrency on ≥H100-class GPUs.

Encoder cache

For multimodal models the scheduler cooperates with EncoderCacheManager / EncoderDecoderCacheManager (vllm/v1/core/encoder_cache_manager.py) to reserve encoder cache slots before scheduling cross-attention work.

Key source files

File	Purpose
vllm/v1/core/sched/scheduler.py	The main scheduler logic
vllm/v1/core/sched/output.py	SchedulerOutput dataclass family
vllm/v1/core/sched/request_queue.py	FIFO / priority queues
vllm/v1/core/sched/async_scheduler.py	Async scheduling subclass
vllm/v1/core/sched/utils.py	check_stop, remove_all
vllm/v1/core/sched/interface.py	Abstract base (used by tests and alt schedulers)
vllm/v1/request.py	Request, RequestStatus, StreamingUpdate
vllm/v1/metrics/perf.py	PerfStats, ModelMetrics consumed by the scheduler

Entry points for modification

• New scheduling policy: extend SchedulingPolicy in request_queue.py and add a queue.
• New stop condition: implement check_stop semantics in vllm/v1/core/sched/utils.py. Add a corresponding FinishReason.
• New per-step state for the model: extend SchedulerOutput/CachedRequestData and consume it in vllm/v1/worker/gpu_model_runner.py.
• New connector hook: implement KVConnectorBase_V1; the scheduler will call your start_load_kv / wait_for_save automatically.