Executor

The executor turns the planner's PhysicalPlan into a tree of runtime operators that produce result chunks.

Purpose

pkg/executor/ is the runtime side of the SQL pipeline. It contains:

• The Executor interface and a runtime tree of operator implementations.
• The Compiler and ExecStmt adapter that bridge planner output to runtime.
• DDL job submission paths (executors that hand work over to the DDL framework).
• Implementations of EXPLAIN, ANALYZE, LOAD DATA, IMPORT INTO, SHOW, SET, traffic capture, BR/restore, plan replayer, and more.
• Memory-table / cluster-table readers that back INFORMATION_SCHEMA and PERFORMANCE_SCHEMA tables.

Directory layout

pkg/executor/
├── adapter.go                # ExecStmt: wraps a plan + statement
├── adapter_slow_log.go       # Slow log emission from adapter
├── builder.go                # PhysicalPlan → Executor tree (huge file)
├── compiler.go               # Top-level Compiler driving optimize → build
├── select.go                 # Common select / projection / table reader glue
├── insert.go, update.go, delete.go  # DML operators
├── insert_common.go          # Shared insert/update preconditions
├── point_get.go, batch_point_get.go # Fast-path KV reads
├── distsql.go                # TableReader, IndexReader, IndexLookUp
├── index_merge_reader.go     # IndexMerge runtime
├── parallel_apply.go         # Parallel APPLY operator
├── projection.go             # Projection operator
├── union_scan.go             # MVCC union scan over txn buffer
├── analyze*.go               # ANALYZE TABLE pipelines
├── join/                     # All join operators (hash, merge, index, …)
├── aggregate/, aggfuncs/     # Hash and stream aggregation, agg fns
├── sortexec/, unionexec/     # Sort, Union All
├── windows/                  # Window functions
├── importer/                 # IMPORT INTO worker (uses lightning lib)
├── compact_table.go          # ALTER TABLE … COMPACT for TiFlash
├── checksum.go               # ADMIN CHECKSUM TABLE
├── cte.go, cte_table_reader.go # CTE materialisation
├── infoschema_reader*.go     # information_schema/cluster table readers
├── memtable_reader.go, mem_reader.go  # Performance/info schema memtables
├── memtable_predicate_extractor.go    # Predicate pushdown into memtables
├── slow_query.go             # SLOW_QUERY table parsing
├── show.go, show_*.go        # SHOW statements
├── simple.go                 # Single-statement non-cursor commands (BEGIN, COMMIT, SET, …)
├── grant.go, revoke.go       # ACL statements
├── brie.go, brie_utils.go    # BR (BACKUP/RESTORE/IMPORT) statements via SQL
├── traffic.go                # TRAFFIC SHARE/CAPTURE for replay
├── trace.go                  # TRACE FORMAT … runtime
├── plan_replayer.go          # PLAN REPLAYER DUMP/LOAD
├── stmtsummary.go            # Stmt summary memtable surface
├── shuffle.go                # Shuffle operator
├── parallel_apply.go         # Parallel APPLY
├── load_data.go              # LOAD DATA infile / from cloud
├── import_into.go            # IMPORT INTO statement entry
├── mppcoordmanager/          # MPP coordinator registry
├── mpp_gather.go             # MPP gather operator (TiFlash)
├── lockstats/                # LOCK STATS / UNLOCK STATS
├── staticrecordset/          # Static record set wrappers
├── metrics_reader.go, metrics/  # Metrics-table runtime
├── opt_rule_blacklist.go     # Optimizer rule blacklist
├── reload_expr_pushdown_blacklist.go # Pushdown blacklist
├── set_config.go             # ADMIN SET CONFIG
├── split.go                  # SPLIT TABLE, SPLIT INDEX
└── workloadrepo.go           # Workload repository wiring

Key abstractions

Type	File	Purpose
Executor	pkg/executor/internal/exec/exec.go	The runtime operator interface (Open/Next/Close)
BaseExecutor	same	Shared base struct: schema, runtime stats, child executors
ExecStmt	pkg/executor/adapter.go	A statement bundle: plan, AST, runtime stats, ResultSet
Compiler.Compile	pkg/executor/compiler.go	Calls into the planner, returns ExecStmt
executorBuilder	pkg/executor/builder.go	Visits a PhysicalPlan and constructs the matching executor tree
MppCoordinatorManager	pkg/executor/mppcoordmanager/manager.go	Tracks MPP queries and their coordinators
analyzeExec and friends	pkg/executor/analyze*.go	Runtime for ANALYZE TABLE
LoadDataExec	pkg/executor/load_data.go	LOAD DATA / IMPORT INTO logical entry points
BRIEExec	pkg/executor/brie.go	SQL-level BACKUP / RESTORE / SHOW BACKUPS

How it works

graph LR
  ast[AST] --> compile[Compiler.Compile<br/>compiler.go]
  compile --> opt[Planner.Optimize]
  opt --> stmt[ExecStmt<br/>adapter.go]
  stmt --> build[executorBuilder<br/>builder.go]
  build --> tree[Executor tree]
  tree -->|Open / Next / Close| chunks[Chunk batches]
  chunks --> server[pkg/server result writer]

1. Compiler.Compile calls into the planner and wraps the result in ExecStmt.
2. ExecStmt.Exec invokes executorBuilder.build(...). Builder is a giant switch on PhysicalPlan type.
3. The resulting executor tree implements Open(ctx), Next(ctx, chunk), Close(). Next is called repeatedly with a reusable chunk.Chunk and fills it with rows; an empty chunk signals end of stream.
4. Reads and aggregations push down to TiKV/TiFlash via pkg/distsql/ and the coprocessor (see Storage).
5. Writes go through kv.Transaction mutations buffered on the session (pkg/session/txn.go); commit happens at SQL COMMIT time via two-phase commit.
6. The session writes result chunks back through the MySQL protocol layer (pkg/server/).

Operator categories

• Access — TableReader, IndexReader, IndexLookUp, IndexMergeReader, PointGet, BatchPointGet (in distsql.go, point_get.go, batch_point_get.go, index_merge_reader.go, table_reader.go).
• DML — Insert, Replace, Update, Delete, LoadData (in insert.go, update.go, delete.go, replace.go, load_data.go).
• Joins — hash, merge, index, index-hash, parallel-apply, anti-semi (pkg/executor/join/).
• Aggregations — hash agg, stream agg, parallel hash agg (pkg/executor/aggregate/, pkg/executor/aggfuncs/).
• Sort and TopN — chunk-based sort with optional spill (pkg/executor/sortexec/).
• Set ops — Union, UnionAll (pkg/executor/unionexec/); INTERSECT/EXCEPT are realised via plans + hash agg.
• Windows — window functions (pkg/executor/windows/).
• CTE — common table expression materialisation (cte.go).
• Misc — Projection, Selection, Limit, Sort, Shuffle, UnionScan, Trace, Explain, Compact, Checksum.

builder.go is the single integration point. New operators almost always require a new branch there plus the matching PhysicalPlan definition.

Memtables and INFORMATION_SCHEMA

infoschema_reader.go, memtable_reader.go, and mem_reader.go implement the runtime for memtable-backed tables. Predicate-extraction logic lives in memtable_predicate_extractor.go and the plan-side counterpart in pkg/planner/core/memtable_predicate_extractor.go. Adding a new memtable typically involves:

1. Defining the schema in pkg/infoschema/tables.go.
2. Adding a fetcher in pkg/executor/infoschema_reader.go.
3. Optionally extending the predicate extractor for pushdown.

Statement adapter and slow log

adapter.go is the shared lifecycle around any executor tree. It manages:

• Statement context (pkg/sessionctx/stmtctx) initialisation and reset.
• Runtime stats accounting (pkg/util/execdetails/).
• Slow query logging and the optional plan-replayer "auto capture".
• Result-set writing for non-DML statements.
• Statement summary updates.

adapter_slow_log.go serialises a single statement's slow-log line, including the executed plan in textual and binary forms.

Integration points

• Planner: consumes PhysicalPlan from pkg/planner/core.
• Expression: every operator builds and evaluates expressions via pkg/expression/ (evaluator.go, vectorized.go, chunk_executor.go).
• Storage: read paths use pkg/distsql/; writes use pkg/kv/Transaction; both are configured via pkg/store/.
• DDL: DDL executors delegate to pkg/ddl/ddl.go to submit jobs.
• Statistics: analyze*.go writes histograms back through pkg/statistics/handle/.
• MPP: mpp_gather.go and mppcoordmanager/ integrate with pkg/store/copr/mpp.go for TiFlash MPP queries.

Entry points for modification

• Adding or modifying SQL operator semantics → edit the operator file under pkg/executor/ and the matching builder branch in builder.go. Add unit tests next to the operator and an integration test under tests/integrationtest/.
• New memtable or INFORMATION_SCHEMA table → schema in pkg/infoschema/tables.go, fetcher in infoschema_reader.go.
• New ANALYZE strategy → analyze*.go and pkg/statistics/handle/.
• New EXPLAIN output → explain.go and pkg/planner/core/encode.go.