Replication

ClickHouse replicates data at the level of the MergeTree part. Each part is an immutable artifact; replication is a matter of agreeing on which parts exist and shipping them between replicas. The coordination layer is Keeper / ZooKeeper.

For the deep code walk, see Replicated MergeTree. This page is the "what does it do for me" view.

Components

graph TD
    subgraph "Replica A"
      AA[StorageReplicatedMergeTree A]
      AAQ[Per-replica queue]
    end
    subgraph "Replica B"
      BB[StorageReplicatedMergeTree B]
      BBQ[Per-replica queue]
    end
    Keeper[(clickhouse-keeper / ZooKeeper<br/>/clickhouse/tables/{shard}/{table})]
    AA -- write log entry --> Keeper
    Keeper -- pull entries --> AA
    Keeper -- pull entries --> BB
    AA <-- HTTP fetch (DataPartsExchange) --> BB

Each replicated table has a Keeper path with a global operation log. Replicas pull entries from the log into their own queues and either execute them (if they have the source data) or fetch the resulting part from a peer.

What gets replicated

• New parts created by INSERT.
• Merges between existing parts (the result is published to the log).
• Mutations (ALTER UPDATE/DELETE).
• Schema changes (ALTER TABLE ... ADD/DROP COLUMN ...).
• Partition operations (MOVE, DROP, ATTACH, REPLACE).
• TTL moves and column-clear operations.

What does not get replicated

• Engine-internal background tasks that don't change visible data.
• Per-server caches, query history, settings.
• Database / table existence (created via CREATE TABLE ... ENGINE = ReplicatedMergeTree(...) independently on each replica, unless using Replicated databases).

Quorum inserts

insert_quorum=N (or auto for majority) waits until N replicas acknowledge the part. Without quorum, an insert is durable on the originating replica and is replicated asynchronously. Quorum state is tracked in /quorum/ znodes.

Lock semantics

Replication uses Keeper to coordinate exclusive operations:

• EphemeralLockInZooKeeper — exclusive grants for partition-level moves/replaces.
• Leader election — the leader picks merges and moves.
• LeaderElection.h — built-in primitive used by other subsystems (MergeTreeData, Backups).

Failure modes and recovery

• Replica falls behind. system.replicas.queue_size grows; queue entries log an error in last_exception. Most issues self-resolve once Keeper / network recovers. The escape hatch is SYSTEM SYNC REPLICA <table>, which waits until the queue is drained.
• Replica's local disk is corrupted. ReplicatedMergeTreePartCheckThread detects mismatched checksums and re-fetches. Manual recovery is SYSTEM RESTORE REPLICA <table> (re-create from the log) or DETACH/ATTACH REPLICA.
• Replica gone. SYSTEM DROP REPLICA <name> removes the dead replica's znodes.
• Diverging replicas after split-brain. Quorum + select_sequential_consistency is the strongest consistency you can ask for. Otherwise you get eventual consistency.

Replicated databases

Replicated databases (src/Databases/DatabaseReplicated.cpp) replicate the DDL — every CREATE TABLE/ALTER/DROP is captured as a Keeper-coordinated entry, executed on every replica. Pairs with ReplicatedMergeTree to give cluster-wide schema changes without ON CLUSTER.

ON CLUSTER

SOMETHING ON CLUSTER cluster_name runs the statement on every shard's selected replicas. Implemented in src/Interpreters/DDLWorker.cpp (~65 KB). Each server picks tasks from /clickhouse/task_queue/ddl/ in Keeper. system.distributed_ddl_queue lets you watch progress.

Parallel replicas (read path)

A separate feature: a single SELECT is split across replicas of the same shard for more parallelism. The leader assigns mark ranges; followers pull them and stream results back. Implemented in:

• src/Storages/MergeTree/ParallelReplicasReadingCoordinator.cpp
• src/Storages/MergeTree/RequestResponse.cpp
• src/Interpreters/ClusterProxy/SelectStreamFactory.cpp
• src/Client/HedgedConnectionsFactory.cpp (also used)

Toggled by allow_experimental_parallel_reading_from_replicas / parallel_replicas_for_non_replicated_merge_tree.

Hedged reads

HedgedConnections issues a query to multiple replicas with a small delay; the first to respond wins. Used by both Distributed and parallel replicas to mitigate slow replicas.

Diagnostics

• system.replicas — health per replicated table.
• system.replication_queue — pending operations.
• system.zookeeper_log — Keeper request audit.
• system.distributed_ddl_queue — ON CLUSTER queue status.
• system.replicated_fetches — in-flight part fetches.

Related pages

• Replicated MergeTree — the engine.
• Coordination — the Keeper engine.
• Distributed queries — fan-out queries.
• Apps → Keeper