pingora-header-serde

Active contributors: yuchen

Purpose

Compressed serialization of HTTP request and response headers using a shared zstd dictionary. The use case: storing many cached HTTP responses where the headers are a significant portion of the storage cost. A custom-trained dictionary cuts the per-header overhead substantially.

Directory layout

pingora-header-serde/src/
├── lib.rs              Public API: encode_req_header, encode_resp_header, ...
├── dict.rs             Shared dictionary handling
├── thread_zstd.rs      Per-thread zstd encoder/decoder pool
└── trainer.rs          Offline dictionary trainer

Key abstractions

Function / type	File	What it is
encode_req_header_to_buf / decode_req_header	pingora-header-serde/src/lib.rs	Request header (de)serialization
encode_resp_header_to_buf / decode_resp_header	pingora-header-serde/src/lib.rs	Response header (de)serialization
Dictionary	pingora-header-serde/src/dict.rs	Loaded zstd dictionary
Trainer	pingora-header-serde/src/trainer.rs	Offline tool to build a dictionary from sample headers

How it works

The crate writes the headers in a compact wire format (length-prefixed name/value pairs) and runs the result through a zstd encoder primed with a precomputed dictionary. The dictionary captures common header names, values, and patterns specific to your traffic mix. Per-thread encoders avoid initialization cost.

You train a dictionary offline against a corpus of representative headers (Trainer) and load it at runtime (Dictionary::from_path). The dictionary is shared across threads.

Integration points

• Used by Cloudflare-internal cache backends that serialize cached responses to disk.
• Not directly exposed via pingora::*.

Entry points for modification

• New header field → must round-trip through encode_* and decode_*.
• Compression algorithm change → would mean retraining dictionaries.

Key source files

File	Purpose
pingora-header-serde/src/lib.rs	Public encode/decode functions
pingora-header-serde/src/dict.rs	Dictionary loading
pingora-header-serde/src/thread_zstd.rs	Encoder/decoder pool
pingora-header-serde/src/trainer.rs	Offline dictionary trainer