Stream radix

The encoding used for stream values. A radix tree of listpacks, with consumer-group state stored in side raxes.

Source layout

File	Role
src/t_stream.c	The full stream type implementation (~250 KB). Commands, encoding, RDB save/load, AOF rewrite.
src/stream.h	Stream-specific types: stream, streamCG, streamConsumer, streamID, streamNACK.
src/rax.c	The underlying compressed radix tree.
src/listpack.c	The leaf format.

Layout

A stream is:

• A rax (stream.rax) keyed by entry id, whose leaves are listpacks containing a contiguous run of entries.
• The current last_id, max_deleted_entry_id, total length, total entries_added, and bookkeeping fields.
• A dict of consumer groups (stream.cgroups) — one entry per XGROUP CREATE.

A streamCG (consumer group) holds:

• The group's last_id (last delivered id).
• A dict of consumers (streamConsumer) keyed by name.
• A pending entries list (PEL), implemented as a rax keyed by entry id with streamNACK values.

A streamConsumer holds the consumer's name, last activity time, and a reference to the per-consumer pending entries (also a rax).

graph TD
    Stream[stream] --> Rax1[rax: id -> listpack node]
    Stream --> CG[dict: cgroups]
    Rax1 --> LP1[listpack node 1]
    Rax1 --> LP2[listpack node 2]
    CG --> G1[group A]
    CG --> G2[group B]
    G1 --> Cs[dict: consumers]
    G1 --> PEL[rax: pending entries]
    Cs --> C1[consumer x]
    Cs --> C2[consumer y]
    PEL --> N1[NACK: entry id 5-1]
    PEL --> N2[NACK: entry id 5-2]

Entry ids

Stream entry ids are 128-bit values: <ms-timestamp>-<sequence>. The auto-id form (* in XADD) uses the current time in ms for the timestamp; if multiple entries arrive in the same millisecond, the sequence increments. Explicit ids must be strictly greater than the current last_id.

The 128-bit id is encoded in the rax as 16 bytes big-endian, which makes radix-tree byte order match logical id order.

Listpack nodes

Each listpack holds entries with a shared "master" set of field names. The first entry in a listpack lists every field used in that node (the "master fields"); subsequent entries reference the master fields by index plus per-entry field values. This deduplicates the field names and makes typical streams (where every entry has the same fields) very compact.

The node-size limits are stream-node-max-bytes (default 4 KiB) and stream-node-max-entries (default 100). When a node fills up, the next XADD creates a fresh listpack.

Range queries

XRANGE start end COUNT n walks the rax from start forward, opening the first listpack and iterating. When the listpack ends, the iterator advances to the next rax leaf.

XREVRANGE end start COUNT n does the same in reverse.

XREAD STREAMS key id is built on top of XRANGE plus blocking semantics. If id is $, only entries added after this command was issued are returned; the blocking machinery (in src/blocked.c) wakes the client when a new XADD lands.

Consumer groups

XGROUP CREATE initialises a streamCG whose last_id defaults to $ (no past entries). XREADGROUP GROUP g c delivers entries past the group's last_id; the entry id is added to both the group's PEL and the per-consumer PEL, then last_id advances.

XACK removes an id from the PEL, marking it processed. XPENDING reports the PEL contents. XCLAIM and XAUTOCLAIM move an id from one consumer's pending list to another's (typical pattern when a consumer dies and another picks up its work).

XACKDEL, XDELEX, XNACK are recent additions that combine ack, delete, and explicit nack semantics.

Trimming

XTRIM key MAXLEN n and XTRIM key MINID id shrink the stream by deleting old entries. The implementation walks the rax from the head, drains complete listpacks, and trims the boundary listpack in place. With ~ (approximate) the trim only deletes complete listpacks, which is much cheaper.

XADD ... MAXLEN ~ n does an inline approximate trim on every add.

Persistence

RDB version 11+ encodes streams with their full structure: entries, groups, consumers, PEL. Older versions encode only the entries (groups are reconstructed from the configured XGROUP calls in the AOF, if any).

AOF rewrite emits XADD (one per entry, keeping the same field structure) plus XGROUP CREATE, XGROUP CREATECONSUMER, and per-PEL replay records. The replay is exact.

Where to start modifying

• Add a stream subcommand — the dispatcher is at the top of src/t_stream.c. Add a JSON spec, regenerate commands.def.
• Tune listpack-node sizing — streamNodeAlmostFull heuristic in src/t_stream.c.
• Improve consumer-group tracking — streamCG and streamNACK types in src/stream.h.

Related pages

• Rax, Listpack & quicklist.
• Data types — the Stream commands.
• Pub/Sub & tracking — XREAD BLOCK uses the blocked-client framework.