pingora-cache

Active contributors: ewang, mgumport, ssiloti

Purpose

The HTTP caching layer for proxies. Implements RFC 9111-style caching with a state machine driven from pingora-proxy. Provides a pluggable storage backend (Storage trait), a MemCache reference implementation, eviction policies, request coalescing locks, and Vary-based cache variance.

The README warns: "Pingora proxy integration with caching should be considered experimental, and as such APIs related to caching are currently highly volatile."

Directory layout

pingora-cache/src/
├── lib.rs               HttpCache state machine (~1,800 lines)
├── cache_control.rs     Parse/render Cache-Control directives
├── filters.rs           Cacheability filters (response status, headers)
├── key.rs               CacheKey type + variance hashing
├── lock.rs              CacheKeyLock — coalesces concurrent misses
├── max_file_size.rs     Per-asset size cap tracking
├── memory.rs            MemCache: in-memory Storage backend
├── meta.rs              CacheMeta — what we record about a cached asset
├── predictor.rs         Heuristic cacheability predictor
├── put.rs               Cache PUT (write-side) state machine
├── storage.rs           Storage trait, HitHandler, MissHandler, PurgeType
├── trace.rs             Optional cache tracing spans (gated by `trace` feature)
├── variance.rs          VarianceBuilder
├── hashtable.rs         Thread-safe hashtable for in-memory store
└── eviction/            Eviction policies (LRU, fifo, simple)

Key abstractions

Type	File	What it is
HttpCache	pingora-cache/src/lib.rs	The per-request cache state machine
CachePhase	pingora-cache/src/lib.rs	The state: Disabled/Uninit/Bypass/CacheKey/Lookup/Hit/Miss/Stale/Revalidating/Expired
NoCacheReason	pingora-cache/src/lib.rs	Reason a cache lookup was skipped
CacheKey, CompactCacheKey, HashBinary	pingora-cache/src/key.rs	Cache key types
CacheMeta, CacheMetaDefaults	pingora-cache/src/meta.rs	Persisted metadata for one cached asset
Storage	pingora-cache/src/storage.rs	Backend trait — implementors store and retrieve assets
HitHandler, MissHandler	pingora-cache/src/storage.rs	Read- and write-side handles to a cached asset
PurgeType	pingora-cache/src/storage.rs	Hard purge, hint purge, etc.
MemCache	pingora-cache/src/memory.rs	Reference Storage impl backed by an in-memory hashtable
CacheKeyLockImpl, LockStatus, WritePermit	pingora-cache/src/lock.rs	Coalesces concurrent fills
VarianceBuilder	pingora-cache/src/variance.rs	Builds the variance hash from Vary headers
MaxFileSizeTracker	pingora-cache/src/max_file_size.rs	Streams writes through a size cap
Predictor	pingora-cache/src/predictor.rs	Heuristic for whether to admit

How it works

stateDiagram-v2
    [*] --> Uninit
    Uninit --> Disabled : disable()
    Uninit --> CacheKey : enable() / set_cache_key
    CacheKey --> Lookup : storage.lookup
    Lookup --> Hit : asset found, fresh
    Lookup --> Stale : asset found, expired
    Lookup --> Miss : nothing
    Stale --> Revalidating : revalidate
    Revalidating --> Hit : 304
    Revalidating --> Miss : 200
    Miss --> Expired : finished writing
    Hit --> Expired : after read
    Expired --> [*]

The state machine is driven from pingora-proxy/src/proxy_cache.rs based on the user's overrides on ProxyHttp. The proxy calls HttpCache::cache_lookup, then either serves from HitHandler, fills via MissHandler, or coordinates revalidation.

Cache lock

When two requests for the same key miss simultaneously, the second is held on a CacheKeyLock while the first fills. LockStatus (added in 0.7.0) lets user code see whether they're waiting on a lock and decide whether to bypass. Stale-while-revalidate uses a similar coordination so that the in-flight revalidation isn't blocked by callers that would prefer the stale copy.

Variance

Vary: Accept-Encoding, ... means different request headers map to different cached variants. VarianceBuilder (variance.rs) hashes the relevant request header values into a per-variant key. The add_owned_name_value method (added in 0.7.0) lets callers add ad-hoc variance dimensions.

Eviction

The eviction/ subdirectory has policies that decide what to drop when storage fills:

• simple_lru.rs — naive LRU
• fifo.rs — FIFO

Eviction integrates via the EvictionManager trait. The peek_lru method (added in 0.8.0) lets callers inspect the LRU head without popping it.

Predictor

predictor.rs is a heuristic cacheability predictor — short-lived assets get demoted, frequently-revisited assets get promoted. It helps avoid putting one-off requests into the cache.

MemCache

Reference Storage impl in memory.rs. Used in tests and for in-memory cache tier in production. Backed by hashtable.rs, which is itself a thin wrapper over flurry (concurrent hashmap).

put.rs

Cache PUT — explicit insertion of an asset, used by some Cloudflare flows. Implements its own state machine separate from the proxy-driven cache fill.

Integration points

• Driven by pingora-proxy/src/proxy_cache.rs.
• Re-exported under pingora::cache::* when cache feature is on.
• The Storage trait is the integration point for plugging in your own backend (e.g. disk-backed, distributed). The CHANGELOG notes that storage APIs are still volatile.

Entry points for modification

• New eviction policy → add to eviction/, implement EvictionManager.
• New storage backend → implement Storage, HitHandler, MissHandler from storage.rs. Use MemCache in memory.rs as a reference.
• Tweaking cache-control parsing → cache_control.rs.
• Adding a cache phase → mostly a pingora-proxy change, but you may need new CachePhase variants here.

Key source files

File	Purpose
pingora-cache/src/lib.rs	HttpCache, CachePhase, top-level state machine
pingora-cache/src/storage.rs	Storage trait, HitHandler, MissHandler, PurgeType
pingora-cache/src/lock.rs	Cache lock + LockStatus
pingora-cache/src/key.rs	CacheKey, CompactCacheKey
pingora-cache/src/meta.rs	CacheMeta
pingora-cache/src/memory.rs	MemCache reference backend
pingora-cache/src/eviction/	Eviction policies
pingora-cache/src/variance.rs	Vary handling
pingora-cache/src/cache_control.rs	Cache-Control parser
pingora-cache/src/predictor.rs	Cacheability predictor