Coordination

src/Coordination/ is ClickHouse's native coordination service — clickhouse-keeper — a Raft-based, ZooKeeper-wire-compatible store. It can run standalone (the clickhouse keeper binary, see Apps → Keeper) or embedded inside clickhouse-server.

Layers

graph TD
    Client[ZooKeeper-protocol clients<br/>and ZooKeeper.cpp inside ClickHouse] --> Net[KeeperServer<br/>NuRaft glue]
    Net --> Raft[NuRaft consensus<br/>contrib/NuRaft]
    Raft --> Log[KeeperLogStore<br/>RocksDB or files]
    Raft --> SM[KeeperStateMachine]
    SM --> Storage[KeeperStorage<br/>in-memory KV + watches]
    SM --> Snap[KeeperSnapshotManager*]

Key files

File	Role
KeeperServer.cpp	Top-level Raft node. Wires NuRaft, the state machine, the log store, and the network listener.
KeeperStateMachine.cpp	The Raft state machine. Applies committed ops to KeeperStorage.
KeeperStorage.cpp (~120 KB)	The in-memory key-value store: znodes, ACLs, ephemerals, watches, sequential nodes.
KeeperLogStore.cpp, KeeperLogIO.cpp	Persistent Raft log. Backends: file-based, RocksDB.
KeeperSnapshotManager.cpp, KeeperSnapshotManagerS3.cpp	Snapshots: serialize the storage to disk; restore on startup. Optional S3 backup of snapshots.
KeeperConnectionStats.cpp, KeeperDispatcher.cpp	Per-connection stats and request dispatch.
CoordinationSettings.cpp	All Keeper-side settings.
FourLetterCommand.cpp	Classic ZooKeeper four-letter admin commands (mntr, stat, srvr, conf, dump, ruok, rqld, rcvr, crst, wchc, wchp, wchs, …).
KeeperFeatureFlags.cpp	Feature-flag negotiation between Keeper nodes.
Standalone/	Code shared with embedded mode.

State machine

KeeperStorage is a hierarchical KV store with znode metadata (cversion, mtime, ephemeral owner, ACL). It supports:

• create / delete / set / get / exists / list / multi.
• Sequential nodes (auto-increment suffix).
• Ephemeral nodes (deleted with the session).
• Watches (one-shot notifications).
• Multi-read (an extension over plain ZooKeeper).
• Filtered list (newer extension).
• ACLs (digest scheme).

Every committed Raft op is applied through KeeperStateMachine::pre_commit / commit against KeeperStorage.

Persistence

Two log backends:

• File log — append-only files under coordination/log/. Default for older deployments.
• RocksDB log — log entries stored in RocksDB (vendored under contrib/rocksdb). Better for high write throughput.

Snapshots periodically capture the entire KeeperStorage to coordination/snapshots/. The Raft log is truncated up to the latest snapshot. On startup, the latest snapshot is loaded and the log is replayed from there.

KeeperSnapshotManagerS3.cpp optionally backs up snapshots to S3 for disaster recovery.

Networking

TinyContext.cpp and the listener in KeeperServer.cpp accept ZooKeeper protocol connections. The wire format mirrors ZooKeeper closely; ClickHouse extends it with multi-read, async operations, and feature-flag negotiation. ZooKeeper clients (Curator, Kafka, …) connect with no changes.

Embedded mode vs standalone

In embedded mode (<keeper_server> in the server config), the engine runs inside clickhouse-server. Tables in this server use a built-in client to talk to the local Keeper; remote replicas connect over the network. This is convenient for single-node testing and small clusters but does not provide the durability of a dedicated quorum.

In standalone mode (the clickhouse keeper subcommand), Keeper runs in its own process, typically deployed as a 3- or 5-node Raft quorum on dedicated hosts. This is the recommended production layout.

Settings of note

CoordinationSettings.cpp:

• operation_timeout_ms, session_timeout_ms, min_session_timeout_ms.
• force_sync — fsync per write.
• snapshot_distance — ops between snapshots.
• quorum_reads, linearizable_reads — strong-consistency knobs.
• auto_forwarding, compress_logs, compress_snapshots_with_zstd_format.
• disk_local, disk_remote, latest_logs_cache_size_threshold.
• digest_enabled — server-side state digest for auditing.

ZooKeeper client inside ClickHouse

The other side of the wire — the ZooKeeper client — lives in src/Common/ZooKeeper/. It handles connection management, retry, multi-instance failover, ACL, leader following. This is the client used by StorageReplicatedMergeTree, the DDL worker, cluster discovery, replicated databases, the named collection store, and the replicated access storage.

Migration from ZooKeeper

clickhouse keeper-converter (programs/keeper-converter/) converts ZooKeeper transaction logs and snapshots into Keeper's snapshot format. The converted snapshot is dropped into Keeper's snapshots/ dir before first start.

Tools

• programs/keeper-client/ — interactive client (analogue of zkCli.sh).
• programs/keeper-bench/ — load generator.
• programs/keeper-data-dumper/ — forensics dumper.
• programs/keeper-utils/ — misc helpers.
• utils/keeper-overload/ — sustained-load harness.

Entry points for modification

• New op type → extend KeeperStorage and KeeperStateMachine::commit.
• New four-letter command → FourLetterCommand.cpp.
• New log backend → subclass IKeeperLogStore (or extend KeeperLogStore).
• New setting → CoordinationSettings.cpp.

Related pages

• Apps → Keeper
• Replicated MergeTree
• Replication
• Common utilities — Common/ZooKeeper/ client.