Change streams

A change stream is a long-running cursor over a logical change feed sourced from the oplog. Drivers subscribe to a change stream to receive a typed event for every insert, update, delete, replace, and DDL change. The full design — including the oplog representation, resume tokens, and the post-image strategy — is in docs/change_streams.md (~44 KB), the largest design doc in the repository.

Purpose

Change streams provide:

• Continuous tailing of a collection, database, or whole cluster.
• Resume tokens that survive primary failovers and let consumers resume exactly where they left off.
• Pre- and post-images of updated documents (when enabled by the collection options).
• Cluster-wide ordering — events are delivered in cluster timestamp order even when sourced from many shards.

Wire-protocol surface

A change stream looks like an aggregation pipeline:

const cs = db.collection.watch([{ $match: { 'fullDocument.region': 'US' } }]);

while (cs.hasNext()) {
  const event = cs.next();
  // event = { _id: <resumeToken>, operationType: "insert", ... }
}

The driver opens the cursor with aggregate whose first stage is $changeStream. The server rewrites it into an internal pipeline rooted at $_internalChangeStreamUnwindTransaction, $_internalChangeStreamCheckResumability, and other internal stages.

Internals

graph LR
    Oplog[local.oplog.rs]
    Pipeline[Aggregation pipeline]
    UnpackTxn[$_internalChangeStreamUnwindTransaction]
    Filter[$_internalChangeStreamMatchFilter]
    Transform[$_internalChangeStreamTransform]
    Resumability[$_internalChangeStreamCheckResumability]
    Tail[Tailing cursor on oplog]

    Oplog --> Tail
    Tail --> Pipeline
    Pipeline --> UnpackTxn
    UnpackTxn --> Filter
    Filter --> Transform
    Transform --> Resumability
    Resumability --> Output[Output to client]

Steps:

1. The pipeline opens a tailing cursor on local.oplog.rs starting at the requested timestamp.
2. Multi-statement transactions are unwound — a single applyOps oplog entry produces one event per inner statement.
3. Filters narrow events to the requested namespace and shape.
4. The Transform stage reshapes raw oplog entries into the public change-event schema.
5. Resumability checks ensure events are emitted in cluster-time order and that resume tokens encode enough state to resume after a failover.

For a sharded cluster, mongos opens change streams on every shard and merges the streams in cluster-time order — see src/mongo/s/change_streams/ and src/mongo/db/pipeline/document_source_change_stream*.

Resume tokens

A resume token is an opaque BSON document that encodes:

• The cluster timestamp of the last event delivered.
• The application-level identifier of that event (uuid, document key).
• A version tag so older drivers can decode newer tokens without crashing.

Drivers pass the token back via resumeAfter or startAfter to resume the stream. The token format is in src/mongo/db/pipeline/resume_token.cpp.

Pre- and post-images

By default, an update event reports the post-image by re-reading the updated document from the storage engine — but only if the document still exists at the time the change stream catches up. To support consumers that need the full document at the moment of the change, MongoDB persists configurable pre- and post-images:

• changeStreamPreAndPostImages is a per-collection option.
• Pre-images go in config.system.preimages.
• Post-images are stored inline on the oplog.

The truncation policy (how long pre-images are retained) is implemented in src/mongo/db/change_stream_pre_images_truncate_markers_sampling_strategy/ and the related change_stream_pre_images_collection* helpers under src/mongo/db/.

Sharded change streams

The router-side machinery is in src/mongo/s/change_streams/. Per-shard cursors are merged with a sort-merge that relies on cluster timestamps; if a shard goes silent, a periodic post-batch resume token keeps the stream advancing.

DDL events

Newer protocol versions also emit DDL events: create, drop, rename, shardCollection, reshardCollection, etc. These come from the c (command) oplog entries; the change-stream pipeline re-shapes them into typed events.

Key source files

File	Purpose
src/mongo/db/pipeline/document_source_change_stream.cpp	The user-visible $changeStream stage and its internal expansion.
src/mongo/db/pipeline/document_source_change_stream_*.cpp	One file per internal stage (unwind, transform, resumability, …).
src/mongo/db/pipeline/resume_token.cpp	Token format and serialization.
src/mongo/s/change_streams/	Router-side merging across shards.
src/mongo/db/change_stream_pre_images_truncate_markers_sampling_strategy/	Truncation policy for pre-image storage.
docs/change_streams.md	The canonical design document.

Integration points

• The oplog is the source of truth.
• The op observer writes oplog entries that change streams consume.
• Sharding — per-shard cursors are merged on mongos.
• The aggregation pipeline hosts change streams as a special pipeline.
• The transactions subsystem — change streams unwind each transaction's applyOps into one event per inner statement.

Entry points for modification

Most changes to the change-stream protocol touch one of the internal document_source_change_stream_* stages. Token-format changes need to update both reader and writer and bump the embedded version tag. Cross-shard ordering and resumability work happens in src/mongo/s/change_streams/. The dedicated change_streams resmoke suite plus the FSM-based change_stream_fsm suite are the primary CI gates.