pingora-memory-cache

Active contributors: yuchen

Purpose

Async, in-memory cache with read-through semantics and a built-in cache lock to prevent stampedes. Distinct from pingora-cache (which is the HTTP cache for proxies). This crate is for caching arbitrary computation results — DNS resolutions, expensive lookups, anything where multiple concurrent callers want the same value.

Directory layout

pingora-memory-cache/src/
├── lib.rs               MemoryCache top-level API
└── read_through.rs      Read-through pattern with lock coalescing

Key abstractions

Type	File	What it is
MemoryCache<K, V>	pingora-memory-cache/src/lib.rs	The cache, generic over key and value
Lookup	pingora-memory-cache/src/lib.rs	Result enum for the cache lookup
RTCache<K, V> (read-through)	pingora-memory-cache/src/read_through.rs	Cache + closure that computes missing values

How it works

The cache wraps tinyufo for admission/eviction (the actual TinyLFU-flavor algorithm lives in the tinyufo crate). Values are owned Arc<V>s so multiple callers can share a value without copying.

Read-through mode (RTCache):

graph TD
    a[Caller asks for key K]
    b{In cache?}
    c[Return value]
    d{Lock<br/>acquired?}
    e[Compute<br/>via user closure]
    f[Insert into cache]
    g[Wait on lock]
    a --> b
    b -->|hit| c
    b -->|miss| d
    d -->|yes| e --> f --> c
    d -->|no| g --> b

Two simultaneous lookups for the same key see only one execution of the user closure. The other waits and reads the cached result.

Integration points

• Used by Cloudflare-internal services (DNS lookups in particular).
• Re-exported under pingora- prefix as a published crate but not surfaced via pingora::*.

Entry points for modification

• Different admission policy → would mean swapping the tinyufo dependency.
• Adding TTL semantics → read_through.rs.

Key source files

File	Purpose
pingora-memory-cache/src/lib.rs	MemoryCache API
pingora-memory-cache/src/read_through.rs	RTCache with lock coalescing