File cache

Active contributors: Maxim Dounin, Sergey Kandaurov

Purpose

The file cache stores upstream responses on disk so that subsequent requests for the same key can be served without contacting the backend. Used by proxy_cache, fastcgi_cache, uwsgi_cache, scgi_cache. Implementation: src/http/ngx_http_file_cache.c — at ~2,300 lines, the largest non-protocol-module file in the HTTP tree.

Directory layout

src/http/
├── ngx_http_cache.h           # cache types
└── ngx_http_file_cache.c      # the implementation

The proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=foo:10m max_size=1g use_temp_path=off; directive bootstraps:

• A directory tree on disk (with optional levels=1:2 hashing) where cache files live.
• A shared-memory zone (keys_zone=foo:10m) holding the in-memory cache index.
• A cache loader process that scans the directory once at startup.
• A cache manager process that periodically expires entries past inactive or above max_size.

Key abstractions

Type / function	Role
ngx_http_cache_t	Per-request cache state (key, file, status)
ngx_http_file_cache_t	Per-zone state — paths, zone, sh memory pointer
ngx_http_file_cache_node_t	rbtree node in the in-memory index
ngx_http_file_cache_open()	Cache lookup — returns hit / stale / miss / updating
ngx_http_file_cache_create()	Allocate a placeholder for an in-flight fetch
ngx_http_file_cache_update()	Commit an open temp file as the final cached response
ngx_http_file_cache_send()	Stream a hit straight to the client
ngx_http_file_cache_valid()	Test whether a cached entry's status is still cacheable
ngx_http_cache_status_t	Enum: MISS, BYPASS, EXPIRED, STALE, UPDATING, REVALIDATED, HIT, SCARCE

On-disk layout

Each cached response is one file. The path is derived by hashing the cache key and applying levels=1:2:

key = "GET" + scheme + host + uri
md5 = ngx_md5(key)
levels=1:2  =>  /<root>/<md5[31]>/<md5[29..30]>/<md5>

So proxy_cache_path /var/cache/nginx levels=1:2 ... puts a file with hash abcdef...12345...0f at /var/cache/nginx/f/45/abcdef...12345...0f.

Each file starts with a binary header (the ngx_http_file_cache_header_t) recording the response's metadata: status, content-length, last-modified, etag, vary, expiry. Then the response headers as nginx serializes them, then the body.

Cache key

The cache key is configurable via proxy_cache_key. Default for proxy_cache is $scheme$proxy_host$request_uri. The framework hashes the key (MD5) for both the rbtree lookup and the file path.

Lookup flow

sequenceDiagram
    participant U as Upstream framework
    participant C as ngx_http_file_cache_open
    participant Idx as Shared rbtree (in-mem index)
    participant FS as Disk
    participant Mgr as Cache manager process

    U->>C: open(r, &cache)
    C->>Idx: lookup hash key
    alt hit
        Idx-->>C: node found, ts valid
        C->>FS: open() the cache file
        C->>U: NGX_OK + file fd
        U->>U: ngx_http_file_cache_send to client
    else stale (expired but proxy_cache_use_stale)
        C->>U: NGX_HTTP_CACHE_STALE
    else miss
        Idx-->>C: not found
        C->>Idx: insert "updating" placeholder
        C->>U: NGX_DECLINED — go fetch
        U->>U: contact backend
        U->>FS: write into temp file
        U->>C: ngx_http_file_cache_update
        C->>FS: rename(temp, final)
        C->>Idx: mark hit
    end
    Mgr->>Idx: periodically expire / evict
    Mgr->>FS: remove evicted files

The "updating" placeholder is the request collapsing mechanism: while a worker fetches a miss, other workers and clients hitting the same key see status UPDATING and (with proxy_cache_use_stale updating) get the stale entry instead of all stampeding on the backend.

Validity

ngx_http_file_cache_valid() decides whether a cached response is still fresh. Inputs:

• proxy_cache_valid — per-status TTLs (proxy_cache_valid 200 302 10m;)
• proxy_cache_revalidate — if expired but the entry has Last-Modified / ETag, do a conditional GET upstream
• proxy_cache_use_stale — conditions under which to serve stale (e.g., error, timeout, updating, http_500)
• proxy_cache_background_update — serve stale, fetch in background

Bypass and purge

• proxy_cache_bypass — comma-separated set of conditions that, if any is non-empty, skip the cache (read-through to backend).
• proxy_no_cache — same shape but for writes (don't store the response).
• There is no purge directive in OSS nginx — purging is an NGINX Plus feature. The OSS workaround is rm on the file.

Cache loader

Forked once on startup. Walks the cache directory in batches (loader_files files per scan, sleep loader_sleep between batches, max loader_threshold runtime) and inserts each on-disk file into the in-memory rbtree. Until the loader finishes, hits will go to backend even when the file exists — the index has to know about them first.

After the initial scan, the loader exits.

Cache manager

Forked once and runs forever. Wakes up every manager_sleep, walks the rbtree in LRU order, deletes:

• Entries inactive for longer than inactive.
• Enough entries to bring the total under max_size.

Variables

Variable	Source
$upstream_cache_status	The ngx_http_cache_status_t of the lookup
$upstream_cache_key	The actual cache key used

These are set during ngx_http_upstream_finalize_request so they're available in access logs.

Integration points

• Upstream subsystem — ngx_http_upstream_init calls ngx_http_file_cache_open. The cached body is sent via ngx_http_file_cache_send, which calls ngx_http_send_header + filter chain like any other content.
• Cycle — the cache zones are added at config time via proxy_cache_path etc.; the loader / manager processes are forked from the master.
• Logging — $upstream_cache_status for visibility.

Entry points for modification

Adding a new cache backend storage is unusual — most extension happens via cache key (proxy_cache_key accepts variables) and bypass / use_stale conditions. Touching ngx_http_file_cache_open is the high-bar work; mind the locking around the shared rbtree (ngx_http_file_cache_t.shpool->mutex).