Text and vector search

ClickHouse has been extending its analytical core with two specialised retrieval features.

Full-text search

A native inverted index for String columns. The implementation lives in src/Storages/MergeTree/ (MergeTreeIndexText.cpp, MergeTreeIndexBloomFilter*.cpp) and the matching tokenizers under src/Interpreters/ITokenExtractor.cpp and friends.

Two tiers exist:

• Token bloom filter (tokenbf_v1) — a per-granule Bloom filter of the column's tokens. Cheap and approximate. Skips granules whose tokens definitely don't include the literal. Good for LIKE, hasToken, multiSearchAny, regex with literal sub-strings.
• Text index (text skipping index) — a more proper inverted index with optional positional information. Allows phrase queries and supports searchAny, searchAll, searchAnyWithBM25 style functions when paired with text_index_settings.

ALTER TABLE logs ADD INDEX idx_message message TYPE text(...);

The tokenizer used by these indexes is configurable: default, ngram, split, lowercase, etc. (see MergeTreeIndexUtils.cpp).

Functions

• hasToken, hasTokenCaseInsensitive, multiSearchAny, multiMatchAny, like, match.
• For the proper inverted index: searchAny, searchAll, plus aggregate-style ranking functions when applicable.

How granule skipping works

Each granule's index payload is consulted before reading the column data. If the predicate cannot be satisfied for the granule, it is skipped entirely. See MergeTree → indexes.

Vector search

MergeTreeIndexVectorSimilarity.cpp adds an approximate-nearest-neighbour index on Array(Float32) (or Array(BFloat16) / Tuple(...)) columns, backed by USearch (vendored under contrib/usearch and contrib/SimSIMD).

ALTER TABLE embeddings ADD INDEX idx_vec embedding
TYPE vector_similarity('hnsw', 'cosineDistance', 1024, 'bf16');

Parameters:

• Algorithm: hnsw (the only currently supported).
• Distance: cosineDistance, L2Distance, etc.
• Dimension: must be fixed.
• Storage type: f32 (default), bf16, f16, i8 (compressed).

The index is consulted for queries of the form ORDER BY <distance>(embedding, query) LIMIT N. The optimizer (src/Processors/QueryPlan/Optimizations/optimizeUseHNSWIndex.cpp-style steps) detects the pattern, computes neighbour candidates from the index, and skips reading the rest.

Distance functions

src/Functions/array/arrayDistance.cpp and friends provide:

• cosineDistance, dotProduct.
• L2Distance, L2SquaredDistance, L1Distance, LinfDistance, LpDistance.
• arrayJaccardIndex, arrayHammingDistance.
• BFloat16 arithmetic via arrayBFloat16ToFloat32.

Hardware acceleration: SimSIMD picks AVX-512, AVX2, NEON or Arm-SVE depending on the CPU.

What you typically build

CREATE TABLE docs (
    id UInt64,
    text String,
    embedding Array(Float32),
    INDEX idx_vec embedding TYPE vector_similarity('hnsw', 'cosineDistance', 768, 'bf16'),
    INDEX idx_txt text TYPE text(...)
) ENGINE = MergeTree ORDER BY id;

Then a hybrid query combines the two:

SELECT id, text
FROM docs
WHERE searchAny(text, 'climate change')
ORDER BY cosineDistance(embedding, [...]) ASC
LIMIT 50;

Operational notes

• Both index families are skipping indexes — they're per-granule. Granule size and index_granularity matter.
• Large vector indexes use significant memory (HNSW graphs are kept resident).
• system.parts.secondary_indices_compressed_bytes and system.data_skipping_indices show usage.
• Backups capture the indexes alongside the parts.

Related pages

• MergeTree — secondary indexes anatomy.
• Functions — distance and search functions.