Parallel

Active contributors: Laurens Kuiper, Carlo Piovesan, Tishj

Purpose

src/parallel/ orchestrates query execution. It splits a PhysicalOperator tree into pipelines, schedules pipeline tasks on a fixed-size thread pool, and coordinates completion across the pipeline DAG.

Directory layout

src/parallel/
├── executor.cpp                  Top-level Executor: owns the pipeline DAG
├── meta_pipeline.cpp             Group of pipelines that share a sink
├── pipeline.cpp                  A single linear pipeline
├── pipeline_executor.cpp         Drives a pipeline to completion in a worker
├── task_scheduler.cpp            Thread pool + task queues
├── executor_task.cpp             Task wrapper for executor work
├── task_executor.cpp             Generic task executor used outside queries
├── event.cpp                     Cross-pipeline synchronization events
├── base_pipeline_event.cpp       Base for pipeline lifecycle events
├── pipeline_initialize_event.cpp Pipeline initialization
├── pipeline_event.cpp            Main pipeline run event
├── pipeline_finish_event.cpp     Sink finalization
├── pipeline_prepare_finish_event.cpp Just-before-finalize hook
├── pipeline_complete_event.cpp   Pipeline-completed signal
├── interrupt.cpp                 Interrupt/cancellation glue
├── async_result.cpp              Async result delivery
├── thread_context.cpp            Per-thread query context
└── task_notifier.cpp             Cross-thread task completion signal

Key abstractions

Type	File	Role
Executor	src/parallel/executor.cpp	Builds the pipeline DAG for a query, schedules events, collects results. One per query.
MetaPipeline	src/parallel/meta_pipeline.cpp	A group of pipelines that share a sink and a top-level scope. Used to factor out subplan reuse.
Pipeline	src/parallel/pipeline.cpp	A linear chain of operators from a source through optional intermediate operators to a sink.
PipelineExecutor	src/parallel/pipeline_executor.cpp	Per-thread driver that pulls chunks from the source and pushes them through the pipeline.
TaskScheduler	src/parallel/task_scheduler.cpp	Owns the worker pool. Workers pull tasks from per-priority queues.
Event	src/parallel/event.cpp	Lifecycle marker. Pipelines emit events when they start, finish a sink, etc.; downstream pipelines depend on those events.
ThreadContext	src/parallel/thread_context.cpp	Per-thread query-scoped state used by operator local states.

How it works

graph TD
    Phys[PhysicalOperator tree] -->|MetaPipeline::Build| MP[MetaPipeline]
    MP -->|Pipeline::Build| Pipes[Pipelines + dependencies]
    Pipes -->|Schedule events| Events[Init / Run / Finish / Complete events]
    Events -->|TaskScheduler| Workers[Worker threads]
    Workers --> Done[Result chunks]

Pipeline construction

Pipelines are built top-down from the physical plan:

1. The root sink becomes the top of a MetaPipeline.
2. MetaPipeline walks down the tree until it finds a source. The chain from source to sink becomes one Pipeline.
3. When a sink is encountered mid-walk (e.g., a hash-aggregate inside a join probe), a new child MetaPipeline is started; the parent depends on the child finishing.

A query like SELECT a, sum(b) FROM t GROUP BY a produces:

Pipeline 1: Scan(t) -> HashAggregate Build (sink)
Pipeline 2: HashAggregate Source -> Result (sink)

Pipeline 2 cannot start before Pipeline 1's sink has finalized — that dependency is enforced by Event chains.

Threading

• The TaskScheduler owns a fixed pool sized by PRAGMA threads = N (default: number of CPUs).
• Each pipeline can be executed by multiple threads simultaneously if its source and sink advertise ParallelSource / ParallelSink.
• Each worker has its own LocalSourceState / LocalSinkState. Global state (GlobalSourceState, GlobalSinkState) is shared and synchronized internally.
• When a pipeline is partitioned (e.g., partitioned hash aggregate), each partition becomes a separate task.

Events

Pipelines progress through a sequence of events: initialize → run → prepare-finish → finish → complete. Downstream pipelines hold an Event dependency on their upstream sink's "finish" event; they are only scheduled once that finishes.

sequenceDiagram
    participant Exec as Executor
    participant Sched as TaskScheduler
    participant W as Worker thread
    Exec->>Sched: Schedule InitializeEvent(P1)
    Sched->>W: Run P1.Initialize
    W->>Sched: Done. Schedule PipelineEvent(P1)
    Sched->>W: Pull chunks from P1.source
    W->>W: Sink chunks into P1.sink
    W->>Sched: Done. Schedule FinishEvent(P1)
    Sched->>W: Run P1.sink.Finalize
    W->>Sched: CompleteEvent(P1) -> Schedule P2.InitializeEvent

Cancellation and interrupts

interrupt.cpp handles user-initiated cancellation (e.g., the CLI sending SIGINT). ClientContext::Interrupt wakes pending tasks; operators check InterruptCondition between chunks to bail out promptly.

Async results

AsyncResult (async_result.cpp) wraps a query whose results are consumed by the caller's own thread without holding a worker. This is what powers streaming Connection::SendQuery in non-blocking mode.

Integration points

• Input: PhysicalOperator tree from execution.
• Output: Materialized or streaming QueryResult returned to the main layer (ClientContext::ExecuteInternal).
• Storage: Source operators read from buffer-managed blocks (src/storage/).
• Operators: All worker code paths live inside per-operator LocalSinkState / LocalSourceState classes in src/execution/operator/.

Entry points for modification

• Tuning thread count: PRAGMA threads = N is wired through src/main/config.cpp. The scheduler honors this setting.
• Adding a new pipeline event: subclass BasePipelineEvent, schedule it from Executor::ScheduleEvents.
• Changing partitioning: radix_partitioned_hashtable.cpp and partitioned_execution.cpp (in the optimizer) determine how many partitions are produced; Pipeline::ScheduleSequentialTask decides how to schedule them.
• Adding a non-query task type: see task_executor.cpp for the generic executor that runs background work outside the per-query pipeline.

Key source files

File	Purpose
src/parallel/executor.cpp	Owns the pipeline graph for a query.
src/parallel/meta_pipeline.cpp	Groups pipelines that share a sink.
src/parallel/pipeline.cpp	Single source-to-sink pipeline.
src/parallel/pipeline_executor.cpp	Worker-thread driver.
src/parallel/task_scheduler.cpp	Worker pool and queues.
src/parallel/event.cpp	Cross-pipeline synchronization.
src/parallel/thread_context.cpp	Per-thread query-scoped state.

Continue to transaction for the MVCC layer that pipelines read against, or storage for the data they read.