Watch

A Watch is a streaming subscription to changes in a key range. It is etcd's most subtle feature, touching MVCC, the v3 RPC layer, the proxy, and the new cache module.

What clients see

A client opens a single bidirectional stream and sends:

• WatchCreateRequest — start a new watch on key/range_end, optionally from a prior start_revision, optionally with prev_kv.
• WatchCancelRequest — cancel an existing watch by id.
• WatchProgressRequest — request a heartbeat that confirms the watch is still alive even when no events are flowing.

Server replies on the same stream with WatchResponses that carry events (Put or Delete), the watch id, and protocol headers.

Server-side data flow

sequenceDiagram
    participant Client
    participant V3rpc as watchServer
    participant SWS as serverWatchStream
    participant MVCC as watchableStore
    participant Apply as apply pipeline

    Client->>V3rpc: stream Watch
    V3rpc->>SWS: new serverWatchStream
    Client->>SWS: WatchCreateRequest
    SWS->>MVCC: NewWatchStream(); Watch(key, end, rev)
    Apply->>MVCC: notify(events)
    MVCC->>SWS: events
    SWS->>Client: WatchResponse{events}
    Client->>SWS: WatchCancelRequest
    SWS->>MVCC: Cancel(watchID)

Where things live:

• server/etcdserver/api/v3rpc/watch.go — gRPC stream pump (~18 KB of subtle logic).
• server/storage/mvcc/watchable_store.go — synced and unsynced watcher groups.
• server/storage/mvcc/watcher_group.go — interval-tree-based range matching.
• server/storage/mvcc/watcher.go — per-watcher state + bookkeeping.

Synced vs unsynced

A new watcher is unsynced if its start_revision is below currentRev; the syncWatchersLoop pumps it forward from MVCC history until it catches up, at which point it joins the synced set and receives events directly.

If a watcher's start_revision is below compactMainRev, the server returns ErrCompacted and cancels the watch — the client must re-fetch + re-subscribe.

Fragmentation

Single events can be larger than --max-request-bytes. The watch service fragments responses into multiple WatchResponse messages with a fragment flag. Clients re-assemble before delivering to the user.

Progress notifications

WatchProgressRequest lets a quiet keyspace tell the client "yes, I'm still here, and the watch high-water mark is X". The cache module relies on this in cache/progress_requestor.go to keep its upstream watch fresh; the gRPC proxy does not forward RequestProgress.

prev_kv

When prev_kv = true, every event includes the value the key had immediately before the change. The flag is also used by transactional permission checks — recent CVE fixes (70a2b4871, d97dfbc3b at wiki time) tightened how prev_kv interacts with auth in nested txns.

Client-side behavior

• client/v3/watch.go retries the stream automatically when the gRPC connection drops, resuming from the last delivered revision.
• concurrency.WatchChan exposes the stream as a channel of WatchResponse.
• The cache/ module wraps clientv3.Watcher for in-process fan-out.
• The gRPC proxy's watchProxy (server/proxy/grpcproxy/watch.go) consolidates many client watches into one upstream watch.

Cross-references

• systems/mvcc — watcher fan-out.
• systems/v3rpc — streaming gRPC service.
• modules/cache — local fan-out cache.
• systems/grpc-proxy — proxy-side consolidation.