Prefix caching

Active contributors: Cyrus Leung, Cody Yu, Andreas Karatzas, Woosuk Kwon.

Purpose

Identical leading token sequences across requests should not pay for the same KV cache twice. Prefix caching turns the KV cache into a content-addressed pool keyed by block-aligned hashes; matching prefixes share blocks transparently.

How it works

1. The scheduler asks KVCacheManager.allocate(request, num_tokens).
2. The block hasher (vllm/v1/core/kv_cache_utils.py) splits the prefix into block-sized chunks (default block size 16 tokens) and computes a hash chain — each BlockHash includes the previous block's hash so identical prefixes produce identical chains.
3. For each hash, the KVCacheCoordinator checks the prefix cache map (BlockHash → KVCacheBlock). On a hit it bumps the block's refcount; on a miss it pulls a fresh block from the pool.
4. After the request finishes, KVCacheManager.free(request) decrements refcounts. Blocks with refcount > 0 (those still backing other requests, or still in the prefix-cache map) stay; the rest go back on the free list.
5. When the free list is empty, the eviction policy (LRU on the prefix-cache side) reclaims the least-recently-used cached block.

Configuration

Defaults are on. Knobs live on CacheConfig (vllm/config/cache.py):

Field	Effect
enable_prefix_caching	Master switch (default True)
block_size	Tokens per KV block (default 16; some backends prefer 32 or 64)
prefix_cache_hash_algo	sha256 (default), builtin (Python hash), xxhash
cache_dtype	KV dtype; some quant modes require matching prefix-cache support
prefix_cache_salt (per-request cache_salt)	Tenant isolation: requests with different salts never share blocks

Invalidation

The hash chain depends on:

• The exact token sequence of the prefix
• The active LoRA (LoRAed prefixes get a different cache lane)
• The cache_salt (if set on the request)
• The model + layer config (different KV layouts produce different specs)

Different chains never collide; a single bit flip in a prompt produces a new chain and forces a fresh prefill.

Metrics

Per-step prefix-cache stats are emitted on SchedulerStats.prefix_cache_stats:

• Hits / misses / new tokens
• Bytes saved
• LRU eviction counts

Prometheus gauges expose the running totals; the logging stat logger prints them at a configurable cadence.

Interaction with other features

• Disaggregated prefill — when a KV connector is configured, the scheduler asks the connector for matched tokens before the local prefix cache. A remote prefill instance can pre-warm the cache for a request.
• KV offloading — evicted blocks can move to host RAM or NVMe instead of being freed; subsequent prefix hits restore them.
• Speculative decoding — drafts share the prefix cache because they branch off the verified prefix.
• Multi-modal — the encoder cache is a parallel prefix-style cache for encoder outputs (vllm/multimodal/cache.py).

Key source files

File	Purpose
vllm/v1/core/kv_cache_manager.py	The allocator
vllm/v1/core/kv_cache_coordinator.py	Multi-group dispatch
vllm/v1/core/block_pool.py	Free list
vllm/v1/core/kv_cache_utils.py	Hashing, sizing, BlockHash
vllm/config/cache.py	CacheConfig
vllm/v1/metrics/stats.py	PrefixCacheStats

Entry points for modification

• Different hash function: extend get_hash_fn_by_name and select via CacheConfig.prefix_cache_hash_algo.
• Eviction policy: subclass BlockPool to implement a different policy.
• Per-tenant isolation: send a cache_salt on each EngineCoreRequest (the OpenAI server forwards an HTTP header into it).

For the broader KV cache story, see KV cache. For how the scheduler reasons about it, see Scheduler.