Disaggregated prefill / KV transfer

Active contributors: Robert Shaw, Wensheng Tang, Tyler Michael Smith, Wentao Ye.

Purpose

Prefill (compute KV for the prompt) and decode (one token per step) have very different bottlenecks: prefill is compute-bound and benefits from parallelism, decode is memory-bandwidth-bound and benefits from large-batch sharing. Disaggregating them onto separate GPUs/machines decouples those resources. vLLM does this by transferring KV cache blocks between engines through a KV connector.

Architecture

graph LR
    Cli[Client]
    LB[Router / load balancer]
    subgraph "Prefill cluster"
        P0[Engine P0]
        P1[Engine P1]
    end
    subgraph "Decode cluster"
        D0[Engine D0]
        D1[Engine D1]
    end
    KVC[KV connector<br/>NIXL / Mooncake / LMCache / Redis]

    Cli --> LB
    LB -->|prefill phase| P0
    LB -->|prefill phase| P1
    P0 -->|KV blocks| KVC
    P1 -->|KV blocks| KVC
    KVC -->|fetch| D0
    KVC -->|fetch| D1
    LB -->|decode phase| D0
    LB -->|decode phase| D1
    D0 -->|tokens| Cli
    D1 -->|tokens| Cli

Each side runs the same vLLM binary with different roles configured via KVTransferConfig:

• The prefill engine runs the prompt and writes per-block KV into the connector.
• The decode engine reads those blocks from the connector before emitting the first decode token.

For a workflow image (the original V0 design but conceptually the same flow), see vllm/distributed/kv_transfer/disagg_prefill_workflow.jpg.

Connector roles

KVConnectorBase_V1 (vllm/distributed/kv_transfer/kv_connector/v1/base.py) is implemented by every transport. There are two roles:

• SCHEDULER — runs on the EngineCore process. Decides what to pull/push and when.
• WORKER — runs on each worker. Performs the actual block reads/writes.

Hooks the scheduler uses (vllm/v1/core/sched/scheduler.py):

• connector.get_num_new_matched_tokens(request) — how many leading tokens are already on the connector?
• connector.start_load_kv(request, blocks) — initiate fetch.
• connector.wait_for_save(request, blocks) — fence after a save.
• connector.update_state_after_alloc(request, blocks) — track what's local now.

Hooks the worker uses (vllm/v1/worker/kv_connector_model_runner_mixin.py):

• connector.start_load_kv / wait_for_load
• connector.start_save_kv / wait_for_save

The aggregator KVOutputAggregator (vllm/distributed/kv_transfer/kv_connector/utils.py) collects per-rank outputs from Executor.execute_model so the scheduler sees a unified view.

Built-in connectors

vllm/distributed/kv_transfer/kv_connector/v1/:

Connector	Notes
null	No-op (default)
nixl	RDMA / IB transport via NVIDIA NIXL
mooncake	Mooncake transfer engine
lmcache	LMCache external KV store
redis	Object-store based

The factory (vllm/distributed/kv_transfer/kv_connector/factory.py) instantiates the right pair based on KVTransferConfig.kv_connector.

Encoder cache transfer

For multimodal disaggregation (e.g., running the vision encoder on a different machine), vllm/distributed/ec_transfer/ provides the analogous ECConnector interface. Configuration is in ECTransferConfig (vllm/config/ec_transfer.py), and the worker mixin is vllm/v1/worker/ec_connector_model_runner_mixin.py.

KV events

vllm/distributed/kv_events.py defines an EventPublisher that broadcasts per-block events (EVICTED, STORED, LOADED) to subscribers. This is used by external KV stores to keep their indexes in sync with the engine's view, and by tools that want to observe the cache without owning a connector.

Configuration

Minimal example for a NIXL pair:

# On the prefill machine
vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"nixl","kv_role":"kv_producer", ...}'

# On the decode machine
vllm serve <model> \
  --kv-transfer-config '{"kv_connector":"nixl","kv_role":"kv_consumer", ...}'

KVTransferConfig (vllm/config/kv_transfer.py) carries the role (kv_producer / kv_consumer / kv_both), connector-specific URIs, and timeout settings.

Key source files

File	Purpose
vllm/config/kv_transfer.py	KVTransferConfig
vllm/distributed/kv_transfer/kv_connector/v1/base.py	KVConnectorBase_V1
vllm/distributed/kv_transfer/kv_connector/factory.py	Factory
vllm/distributed/kv_transfer/kv_connector/v1/metrics.py	KVConnectorStats
vllm/distributed/kv_transfer/kv_connector/utils.py	KVOutputAggregator
vllm/v1/core/sched/scheduler.py	Scheduler hooks
vllm/v1/worker/kv_connector_model_runner_mixin.py	Worker hooks
vllm/distributed/ec_transfer/	Encoder-cache equivalent
vllm/distributed/kv_events.py	KV events publisher

Entry points for modification

• New transport: implement KVConnectorBase_V1 (and the matching WORKER half), register via KVConnectorFactory. Add tests under tests/distributed/.
• Custom routing: write a load balancer that asks the connector which engine has the closest prefix match; the connector exposes get_num_new_matched_tokens for this purpose.
• Encoder transfer: implement ECConnectorBase and register via ECConnectorFactory.

For how scheduling integrates with KV transfer, see Scheduler. For the cache mechanics, see KV cache.