Processors

src/Processors/ and src/QueryPipeline/ implement the physical execution engine — a pull-based, push-back-pressured operator graph. This is the layer that actually turns a QueryPlan into bytes on the wire.

The processor model

A Processor (alias IProcessor, src/Processors/IProcessor.h) is a node with input and output Ports. It exposes:

• prepare() — examine its ports and return a Status (Ready, NeedData, PortFull, Async, ExpandPipeline, Finished).
• work() — when Status::Ready, do a unit of work (consume from inputs, produce on outputs).
• schedule() and asynchronousJobIndex() — for processors that wait on async I/O.

Data flows as Chunks — typed columnar batches. A Port has at most one connected peer at any time. A processor can dynamically expand the graph (e.g. when a JOIN needs to add a build pipeline).

graph LR
    Source1[ISource] --> P1[Transform]
    Source2[ISource] --> P1
    P1 --> P2[ResizeProcessor]
    P2 --> P3[Aggregating]
    P3 --> Sink[ISink]

Categories

src/Processors/ is organized by role:

Subdirectory	What lives there
Sources/	Source processors (SourceFromChunks, RemoteSource, MergeTreeSource, KafkaSource, …).
Sinks/	Sink processors (EmptySink, NullSink, RemoteSink, table sinks via IStorage).
Transforms/	The big family of transforms — filters, projections, joins, sort, distinct, array joins, etc.
Merges/	Sorted-merge transforms (MergingSortedTransform, CollapsingSortedTransform, AggregatingSortedTransform, …).
TTL/	TTL-aware merging during background merges.
Formats/	IInputFormat / IOutputFormat and concrete format readers/writers.
Executors/	The PipelineExecutor and PullingPipelineExecutor / PushingPipelineExecutor.
QueryPlan/	Logical plan steps + the optimizer (covered on the Planner page).

The flat root holds the most fundamental processors: IProcessor.cpp, Port.h, Chunk.cpp, LimitTransform.cpp, OffsetTransform.cpp, ResizeProcessor.cpp, ConcatProcessor.cpp, ForkProcessor.cpp, DelayedPortsProcessor.cpp.

Port and chunk

class Port { ... };
class InputPort : public Port { ... };
class OutputPort : public Port { ... };

Ports carry a ColumnsWithTypeAndName header, the streamed Chunks, and signalling state (needed, finished, pulled). The header lets processors validate column compatibility at connection time.

Chunk (src/Processors/Chunk.h) is a Columns plus row count plus optional ChunkInfo payload (used to carry e.g. partial aggregation state, mark-range hints, or quota info).

The executor

src/Processors/Executors/PipelineExecutor.cpp is the run-loop:

1. Walk the graph, determine which processors are ready or need data.
2. Distribute ready processors over a thread pool.
3. Each thread pops a processor, calls work(), then re-checks the affected nodes.
4. Async processors (e.g. remote reads) sit on a separate I/O thread.
5. Continue until all processors are Finished.

Variants:

• PullingPipelineExecutor — produces Blocks for the caller (used by interpreters).
• PullingAsyncPipelineExecutor — same, async.
• PushingPipelineExecutor — caller pushes blocks into a sink (used for INSERT).
• PipelineExecutor — the bare graph engine.

Common transforms

Transform	File	Purpose
FilterTransform	Transforms/FilterTransform.cpp	Apply a boolean expression.
ExpressionTransform	Transforms/ExpressionTransform.cpp	Project / compute.
AggregatingTransform	Transforms/AggregatingTransform.cpp	First-phase aggregation.
MergingAggregatedTransform	Transforms/MergingAggregatedTransform.cpp	Second-phase merge.
JoiningTransform	Transforms/JoiningTransform.cpp	Wraps an IJoin.
SortingTransform / MergeSortingTransform / PartialSortingTransform / FinishSortingTransform	Transforms/Sorting*	ORDER BY variants.
WindowTransform	Transforms/WindowTransform.cpp	Window functions.
LimitTransform, OffsetTransform, LimitByTransform	flat root + Transforms/	LIMIT/OFFSET/LIMIT BY.
DistinctTransform	Transforms/DistinctTransform.cpp	DISTINCT.
ArrayJoinTransform	Transforms/ArrayJoinTransform.cpp	ARRAY JOIN.
CountingTransform	Transforms/CountingTransform.cpp	Updates progress.
MaterializingTransform	Transforms/MaterializingTransform.cpp	Force-materialize columns.
TotalsHavingTransform, RollupTransform, CubeTransform	Transforms/	GROUP BY extensions.

Merges (sorted)

src/Processors/Merges/ implements n-way sorted merges. Variants:

• MergingSorted — plain.
• CollapsingSorted — with sign column (CollapsingMergeTree).
• VersionedCollapsingSorted — with version + sign columns.
• ReplacingSorted — keep the latest version per key (ReplacingMergeTree).
• AggregatingSorted — combine aggregate states (AggregatingMergeTree).
• SummingSorted — sum numeric columns (SummingMergeTree).
• GraphiteRollupSorted — Graphite-aware rollup.

These power both background merges in MergeTree and read-time FINAL/SELECT FROM merge_tree FINAL queries.

QueryPipeline glue

src/QueryPipeline/QueryPipeline.cpp and QueryPipelineBuilder.cpp are the high-level helpers that interpreters and plan steps use to compose processors. They expose convenience methods like addSimpleTransform(...), addMergingAggregatedMemoryEfficientTransform(...), and connect(...).

Async processors

For S3/HTTP/Kafka reads ClickHouse uses Async status. The processor returns Async and registers a future with the executor; once it resolves, the executor wakes up and resumes the work. This keeps the thread pool small even with hundreds of in-flight remote reads.

Profiling

When log_processors_profiles=1 (or the matching MergeTree settings), each processor records its work() time, idle time, and rows in/out into system.processors_profile_log.

Entry points for modification

• New transform → subclass ISimpleTransform or IInflatingTransform/IAccumulatingTransform, or the bare IProcessor for unusual cases.
• New source/sink → subclass ISource/ISink (or ISinkToStorage).
• New format → see Formats — formats are processors too.
• New schedule/work pattern → study IProcessor::Status and PipelineExecutor.

Related pages

• Planner — produces the steps that build pipelines.
• Interpreters — Aggregator, IJoin, expression engine.
• Formats, IO and disks