clickhouse-keeper

clickhouse-keeper is ClickHouse's native, in-process Raft-based coordination service. It speaks the ZooKeeper wire protocol and is intended as a drop-in replacement for ZooKeeper in ClickHouse deployments. Source: programs/keeper/ for the binary, src/Coordination/ for the engine.

For the deeper engine docs, see Coordination.

Purpose

ClickHouse needs strong coordination for:

• Replication of ReplicatedMergeTree tables (the replication log).
• Distributed DDL (ON CLUSTER).
• Cluster discovery (ClusterDiscovery).
• Replicated databases.
• The KeeperMap engine.
• Backup/restore coordination.

Historically that was ZooKeeper. clickhouse-keeper was added in 2021 (first commits to src/Coordination/ are from 2021-01-13) and is now the recommended deployment.

Why a new coordinator?

• Same binary as clickhouse-server — easier ops.
• Native protocol with extensions (multi-read, async ops).
• C++ implementation. RocksDB log + NuRaft consensus.
• Wire-protocol-compatible with ZooKeeper, so existing applications (Curator, Kafka, Flink) work.

Source layout

Path	Purpose
programs/keeper/Keeper.cpp	The Keeper server class (mirrors Server.cpp for clickhouse-server).
programs/keeper/Keeper.h	Declarations.
programs/keeper/keeper_config.xml	Default Keeper configuration.
src/Coordination/KeeperServer.cpp	Wraps NuRaft and exposes the ZooKeeper API.
src/Coordination/KeeperStateMachine.cpp	The Raft state machine — applies committed ops.
src/Coordination/KeeperStorage.cpp	The in-memory KV store (znodes + ephemerals + watches).
src/Coordination/KeeperLogStore.cpp	Raft log persistence on disk.
src/Coordination/KeeperSnapshotManager*.cpp	Snapshotting + recovery.
src/Coordination/CoordinationSettings.cpp	All Keeper-side settings.
src/Coordination/FourLetterCommand.cpp	The classic ZooKeeper four-letter admin commands.
src/Coordination/Standalone/	Code shared with embedded mode.

Modes of operation

Keeper can run in two modes:

• Standalone — clickhouse keeper --config-file=.... The Keeper.cpp binary entry point.
• Embedded in clickhouse-server — set <keeper_server> in the server config. The same engine code runs as part of the server's process. Useful for small clusters where a dedicated quorum is overkill.

In both cases the protocol surface is identical and existing ZooKeeper clients connect with no changes.

Cluster shape

A typical Keeper cluster is a 3- or 5-node Raft quorum. Each node has the same config naming the others. Leader election, log replication, and snapshotting are handled by NuRaft. Reads can be served from any replica with configurable consistency (linearizable_reads setting).

Storage

The on-disk layout:

/var/lib/clickhouse-keeper/
├── coordination/
│   ├── log/             # Raft log — RocksDB or file-based
│   └── snapshots/       # Periodic snapshots of KeeperStorage
└── logs/

The state machine is in-memory; persistence comes from the log + snapshots. Snapshots are taken every snapshot_distance ops (default 100,000) to bound recovery time.

Migrating from ZooKeeper

clickhouse keeper-converter (programs/keeper-converter/) converts an existing ZooKeeper transaction log + snapshot into Keeper's snapshot format. The standard procedure:

1. Stand up an empty Keeper cluster.
2. Stop ZooKeeper.
3. Run keeper-converter --zookeeper-logs-dir ... --zookeeper-snapshots-dir ... --output-dir keeper-snapshots/.
4. Drop the produced snapshot into Keeper's snapshots/ and start the cluster.

Tools

• clickhouse keeper-client — interactive client (similar to zkCli.sh). Source under programs/keeper-client/.
• clickhouse keeper-bench — benchmark.
• clickhouse keeper-data-dumper — forensics dump of Raft state.
• clickhouse keeper-utils — miscellaneous helpers.
• utils/keeper-overload/ — sustained-load harness.

Settings

Settings live in <keeper_server><coordination_settings> and are documented in src/Coordination/CoordinationSettings.cpp. Notable knobs:

• operation_timeout_ms, session_timeout_ms.
• force_sync, fsync_interval.
• snapshot_distance, auto_forwarding.
• quorum_reads, linearizable_reads.
• digest_enabled — server-side state digests for auditing.

Entry points for modification

• To add a new four-letter admin command, see src/Coordination/FourLetterCommand.cpp.
• To add a new Raft setting, edit src/Coordination/CoordinationSettings.cpp.
• The state machine logic for znodes, ACLs, watches, ephemerals lives in KeeperStorage.cpp.

Related pages

• Coordination — the engine details.
• Replication — the largest consumer of Keeper.