KV cache and prefix caching

Active contributors: Cyrus Leung, Cody Yu, Andreas Karatzas, Woosuk Kwon.

Purpose

vLLM's signature contribution is treating the attention KV cache as a paged, content-addressed pool. A request's KV is a list of fixed-size physical blocks, not a contiguous tensor. This eliminates fragmentation, lets unrelated requests share leading tokens (prefix caching), and supports preemption and migration without copies.

Directory layout

vllm/v1/core/
├── block_pool.py                 # BlockPool: free list of KVCacheBlocks
├── kv_cache_manager.py           # KVCacheManager (561 lines) — main allocator
├── kv_cache_coordinator.py       # KVCacheCoordinator — multi-group dispatch
├── single_type_kv_cache_manager.py # Specialized per kv_cache_group manager
├── kv_cache_utils.py             # BlockHash, hashing, sizing helpers (huge)
├── kv_cache_metrics.py           # KVCacheMetricsCollector
├── encoder_cache_manager.py      # Encoder cache for cross-attention
└── ...

vllm/v1/kv_cache_interface.py     # KVCacheConfig, KVCacheSpec, AttentionSpec
vllm/v1/kv_offload/               # CPU and disk offload
vllm/v1/simple_kv_offload/        # Lightweight offload manager
vllm/distributed/kv_transfer/     # Cross-engine KV transport (NIXL, Redis, Mooncake)

Key abstractions

Abstraction	File	Role
KVCacheBlock	vllm/v1/core/kv_cache_utils.py	Physical block descriptor (block_id, ref_count, hash)
KVCacheBlocks	vllm/v1/core/kv_cache_manager.py	Per-request block list, partitioned by kv_cache_group
KVCacheManager	vllm/v1/core/kv_cache_manager.py	Allocates and frees blocks; integrates prefix cache
KVCacheCoordinator	vllm/v1/core/kv_cache_coordinator.py	Routes between multiple kv_cache_groups (e.g., per-layer specs)
BlockPool	vllm/v1/core/block_pool.py	The free list
BlockHash	vllm/v1/core/kv_cache_utils.py	Content hash of a block-sized token chunk
KVCacheConfig / KVCacheSpec	vllm/v1/kv_cache_interface.py	Layout: number of blocks, block size, dtype, layout
AttentionSpec	vllm/v1/kv_cache_interface.py	Per-layer attention metadata (head dim, num heads, dtype)
EncoderCacheManager	vllm/v1/core/encoder_cache_manager.py	Caches encoder hidden states for cross-attention

How allocation works

graph TD
    R[Request prompt tokens]
    H[Block hasher<br/>kv_cache_utils.get_request_block_hasher]
    Hashes[List of BlockHashes]
    Coord[KVCacheCoordinator]
    PC[Prefix cache map<br/>BlockHash -> KVCacheBlock]
    Pool[BlockPool free list]
    BL[Allocated KVCacheBlocks]

    R --> H --> Hashes --> Coord
    Coord -->|hit| PC --> BL
    Coord -->|miss| Pool --> BL

1. The scheduler calls KVCacheManager.allocate(request, num_tokens).
2. The block hasher splits the prefix into block-sized chunks and hashes each.
3. For every hash, the coordinator first checks the prefix cache (a map of BlockHash → KVCacheBlock). On a hit it reuses the block (and bumps the refcount); on a miss it pops a free block from the pool.
4. The result is KVCacheBlocks, which is just a tuple of per-group block lists.

Frees go through KVCacheManager.free(request) which decrements refcounts and returns blocks to the pool when they reach zero.

KV cache groups

Modern vLLM supports heterogeneous attention layers in a single model (e.g., MLA + standard MHA, or hybrid Mamba + attention). Each layer is assigned to a kv_cache_group, and the manager keeps a separate block pool per group. The coordinator maps a request's logical block list to the right physical pools.

KVCacheConfig (built by generate_scheduler_kv_cache_config in vllm/v1/core/kv_cache_utils.py) describes the groups: their block sizes, dtypes, and counts. Sizing happens during Executor.initialize_from_config based on Executor.determine_available_memory() reported by each worker.

Prefix caching

Default-on in V1. Mechanism:

• Block-aligned hashes (BlockHash) are computed from token sequences via SHA-256 (get_hash_fn_by_name) optionally salted by cache_salt for tenant isolation.
• The KVCacheManager keeps a BlockHash → block map. Hits bump refcount; the block is shared across requests until they all finish.
• Eviction is LRU on the free side: when a block leaves the free list, it stays in the prefix-cache map until reclaimed for a new allocation.

This is what makes "system prompt + many user turns" workloads fly — every request reuses the system prompt blocks for free.

KV offloading

For long contexts that don't fit in HBM, blocks can be evicted to host RAM or NVMe:

• vllm/v1/kv_offload/ (full implementation) and vllm/v1/simple_kv_offload/ (lightweight) provide pluggable offload backends.
• OffloadConfig, PrefetchOffloadConfig, UVAOffloadConfig (vllm/config/offload.py) configure the policy.
• The cumem allocator (csrc/cumem_allocator.cpp) gives the engine fine-grained control over CUDA virtual memory so that eviction can happen at block granularity without re-allocating tensors.

KV transfer (disaggregated prefill)

When prefill and decode are split across machines, KV blocks must move between engines. vllm/distributed/kv_transfer/ provides:

• KVConnectorBase_V1 — the connector interface (factory, role: SCHEDULER/WORKER, base, metrics).
• NIXL connector — high-performance cross-host transport.
• Other connectors under vllm/distributed/kv_transfer/kv_connector/v1/ (LMCache, Mooncake, Redis, etc.).

The scheduler hooks (in Scheduler.schedule / update_from_output) call connector.get_num_new_matched_tokens and connector.start_load_kv / wait_for_save to choreograph the transfer. The original V0 disaggregated prefill workflow image lives at vllm/distributed/kv_transfer/disagg_prefill_workflow.jpg.

Encoder cache

For encoder-decoder and multimodal models, encoder hidden states are cached separately:

• EncoderCacheManager — single-modal encoders.
• EncoderDecoderCacheManager — encoder-decoder split.
• MultiModalBudget (vllm/multimodal/encoder_budget.py) — caps how much encoder work is admitted per step.

EC blocks are also transferable between engines via vllm/distributed/ec_transfer/ (mirrors the KV transfer interface).

Metrics

Per-step prefix-cache stats are surfaced through PrefixCacheStats (vllm/v1/metrics/stats.py) and Prometheus gauges (vllm/v1/metrics/prometheus.py). KVCacheMetricsCollector (vllm/v1/core/kv_cache_metrics.py) samples queue depth and block utilization.

Entry points for modification

• New connector: implement KVConnectorBase_V1, register in KVConnectorFactory, expose via KVTransferConfig.kv_connector.
• New offload backend: implement an OffloadBackend (vllm/config/offload.py) and wire it through vllm/v1/kv_offload/.
• Different hash function: extend get_hash_fn_by_name and configure via CacheConfig.prefix_cache_hash_algo.
• New attention layout (e.g., MLA latent): add an AttentionSpec variant in vllm/v1/kv_cache_interface.py and a single-type manager in single_type_kv_cache_manager.py.

For how scheduling and KV cache co-evolve, see Scheduler. For how attention layers consume KV blocks, see Attention backends.