MergeTree

MergeTree is the columnar storage engine that powers analytical workloads in ClickHouse. Source: src/Storages/MergeTree/. It is by far the largest single subsystem in the codebase.

Big idea

Inserts produce immutable, sorted, columnar parts. A part is a directory of column files plus a sparse primary index, marks that point at granule boundaries, and metadata. The engine merges parts in the background to keep their count bounded and their sort order intact. Queries combine reads from many parts and skip irrelevant granules using the primary index, partition pruning, and skip indexes.

This shape — many small immutable artifacts that are continuously merged — gives ClickHouse high write throughput, fast point-range reads, and friendly behaviour under random updates expressed as mutations.

Anatomy of a part

A typical wide part on disk:

20240101_2024_01_42_3/
├── primary.idx               # the sparse primary key
├── columns.txt               # column → type mapping
├── count.txt                 # row count
├── checksums.txt             # blake3 / SipHash of every file
├── partition.dat             # partition key values
├── minmax_*.idx              # per-column min/max for partition
├── default_compression_codec.txt
├── ttl.txt
├── <col>.bin                 # column data (compressed)
├── <col>.mrk2                # marks: (offset_in_compressed, offset_in_decompressed) per granule
├── skp_idx_*.idx             # skip indexes
└── projections/<name>/...    # projection sub-parts

A compact part stores all columns in a single data.bin + data.mrk*. Choice between wide and compact is driven by part size (min_bytes_for_wide_part, min_rows_for_wide_part).

Key abstractions

Class	File	Role
MergeTreeData	MergeTreeData.h (~115 KB)	The base for every MergeTree-family engine. Owns the part set, schema, settings, partitioning, TTL, projections, indices.
IMergeTreeDataPart	IMergeTreeDataPart.h (~38 KB)	Abstract data part. Subclasses: MergeTreeDataPartWide, MergeTreeDataPartCompact, MergeTreeDataPartInMemory.
MergeTreeDataPartChecksum	MergeTreeDataPartChecksum.cpp	Per-file checksums recorded in checksums.txt.
MergeTreeDataPartType	MergeTreeDataPartType.h	Wide / compact / in-memory / unknown.
MergeTreePartInfo	MergeTreePartInfo.h	Parses part directory names like 20240101_42_42_0.
MergeTreePartition	MergeTreePartition.cpp	Encodes the partition key values for a part.
MergeTreeIndexGranularity	MergeTreeIndexGranularity.cpp	Maps row index to mark, supports adaptive granularity.
MergeTreeMarksLoader	MergeTreeMarksLoader.cpp	Loads .mrk* files into a MarksInCompressedFile cache.
KeyCondition	KeyCondition.h (~28 KB)	Translates WHERE predicates over the primary key into mark-range filters.
MergeTreeReadPool*	MergeTreeReadPool.cpp, MergeTreePrefetchedReadPool.cpp, MergeTreeIndexReadResultPool.cpp	Coordinates parallel reads across parts and threads.
MergeTreeRangeReader	MergeTreeRangeReader.cpp	Reads a slice of a part with prewhere/filter awareness.
MergeTreeDataMergerMutator	MergeTreeDataMergerMutator.cpp	Picks parts to merge and runs the merge.
MergeTask	MergeTask.cpp	The merge state machine (collect → execute → merge → write).
MutateTask	MutateTask.cpp	The mutation state machine.
BackgroundJobsAssignee	BackgroundJobsAssignee.cpp	Assigns merges/mutations/moves to background pools.
MergeList	MergeList.cpp	Live list of running merges (visible in system.merges).

Read path

graph TD
    Plan[ReadFromMergeTree<br/>QueryPlan step] --> Analyze[KeyCondition + partition pruning + skip indexes]
    Analyze --> Ranges[(MarkRange[] per part)]
    Ranges --> Pool[MergeTreeReadPool / Prefetched / InOrder]
    Pool --> Workers[Parallel source processors]
    Workers --> Reader[MergeTreeRangeReader<br/>with PREWHERE filter]
    Reader --> Out[Chunk stream]

1. Build KeyCondition from the query's WHERE filtered on the primary-key columns.
2. Skip whole parts via partition pruning (MergeTreePartition minmax).
3. Per surviving part, intersect the primary-index ranges with the KeyCondition to get a MarkRange[].
4. Apply skip indexes (MergeTreeIndices.cpp) to drop more granules.
5. Distribute mark ranges across reader threads via MergeTreeReadPool (one of several variants — work-stealing, prefetched, in-order).
6. Each reader hydrates columns through MergeTreeRangeReader, evaluating PREWHERE first to mask rows before reading the heavy columns.

Write path

graph TD
    Block[Block from INSERT] --> Sink[MergeTreeSink]
    Sink --> Sort[Sort by primary key per partition]
    Sort --> Tmp[/Write tmp_<n>/]
    Tmp --> Atomic[Atomic rename → active part]
    Atomic --> Bg[BackgroundJobsAssignee schedules merge]

MergeTreeSink.cpp (and ReplicatedMergeTreeSink.cpp for replication) handles inserts. Each block is sorted by the primary key, partitioned, and written into a temp directory. The directory is then atomically renamed to make the part active. The background pool eventually merges it with siblings.

Merging

MergeTreeDataMergerMutator::selectPartsToMerge runs every few seconds. It picks a covering range of adjacent parts where the cost (size, age, predicate) is best, then runs MergeTask. Large merges may be split into vertical (per-column) phases for memory friendliness. Sorting happens via MergingSortedAlgorithm (src/Processors/Merges/), which streams rows from each input part in primary-key order.

The merge selectors live in src/Storages/MergeTree/Compaction/ (e.g., SimpleMergeSelector, TTLMergeSelector, AllMergeSelector).

Mutations follow the same machinery via MutateTask. The mutation produces a new part identified by the source part name plus a mutation id; it then becomes the canonical active part for that range.

Indexes

• Primary index — sparse, in memory, sorted by the primary key. Maps mark → key value.
• Skip indexes (MergeTreeIndices.cpp):
  • minmax — per-granule min/max.
  • set — per-granule materialized set of distinct values.
  • bloom_filter — Bloom filter over a column or expression.
  • tokenbf_v1, ngrambf_v1 — token / n-gram Bloom filters for substring search.
  • text — full-text index (MergeTreeIndexConditionText.cpp, TextIndexUtils.cpp).
  • vector_similarity — approximate-nearest-neighbour using usearch / SimSIMD (MergeTreeIndexVectorSimilarity.h).
  • hypothesis — boolean-expression skip.
  • inverted (deprecated, replaced by text).
• Sampling expression — SAMPLE BY enables stable random sampling with SAMPLE 0.1.
• Partition pruning — the PARTITION BY expression is evaluated for each part; queries with predicates over it skip whole partitions.

See Text and vector search.

Settings

MergeTreeSettings (src/Storages/MergeTree/MergeTreeSettings.cpp) defines hundreds of per-table settings (index_granularity, merge_with_ttl_timeout, parts_to_throw_insert, min_rows_for_wide_part, replicated_*, …). Engine-wide defaults live in <merge_tree> of config.xml. Per-table overrides land on CREATE TABLE ... SETTINGS ....

Disk layout and storage policies

A MergeTree table can span multiple disks via a storage policy (<storage_configuration> in config.xml). Volumes are ordered; new parts land on the first volume; TTL TO DISK / TO VOLUME rules move parts later. The disk abstraction (IDisk) makes S3, Azure, HDFS, encrypted, and cached disks first-class. See IO and disks.

On-disk format

Format versions are tracked in MergeTreeDataFormatVersion.h. The current default is the modern adaptive-granularity format (.mrk2/.mrk3). A handful of legacy variants exist (MergeTreeDataPartCompactSettings, the old .mrk format) and remain readable for backward compatibility.

MergeTreeDataPartCompact.cpp and MergeTreeDataPartWide.cpp are the two on-disk layouts in use today. MergeTreeDataPartInMemory is for testing and short-lived parts.

Mutation, projection, and special parts

• A patch part (ReplicatedMergeTreeSinkPatch.cpp) carries lightweight updates and is folded into the data over time.
• A projection part lives under projections/<name>/ of its host part. The optimizer rewrites queries to read from a projection when cheaper.
• A part with _row_exists mask is the result of a lightweight DELETE. The mask is folded into the data in subsequent merges.

Common diagnostics

• system.parts — every active and detached part with size, rows, and stats.
• system.parts_columns — per-column sizes inside parts.
• system.merges — merges in progress.
• system.mutations — mutations and their progress.
• system.replication_queue — replicated-only.
• system.detached_parts — parts that were quarantined.
• OPTIMIZE TABLE ... FINAL — force a full merge.
• ALTER TABLE ... DETACH/ATTACH PART — surgical part operations.

Entry points for modification

• New skip index → subclass IMergeTreeIndex and register in MergeTreeIndices.cpp.
• New part layout → subclass IMergeTreeDataPart and add a MergeTreeDataPartType.
• New merge selector → subclass IMergeSelector under src/Storages/MergeTree/Compaction/.
• New mutation kind → extend MutationCommands and the MutateTask state machine.
• New TTL action → extend TTLMode / TTLDescription.

Related pages

• Replicated MergeTree
• Replication
• Projections and TTL
• Text and vector search
• IO and disks
• Compression