Planner

The planner turns a parsed AST into an executable plan tree. It is the largest, deepest subsystem in the repository.

Purpose

pkg/planner/ is responsible for:

• Building a logical plan from the AST.
• Resolving names against the schema (pkg/infoschema/) and statistics (pkg/statistics/).
• Applying logical rewrite rules (predicate pushdown, projection elimination, decorrelation, etc.).
• Choosing physical operators by cost (Volcano-style enumeration in find_best_task.go).
• Running a parallel Cascades-style optimizer for a growing set of queries (pkg/planner/cascades/).
• Caching prepared-statement plans.
• Producing a runnable plan tree for the executor.

Directory layout

pkg/planner/
├── optimize.go              # Top-level Optimize entry point
├── core/                    # The current default optimizer
│   ├── logical_plan_builder.go     # AST → logical plan
│   ├── planbuilder.go              # Builds non-DML plans (DDL, SET, SHOW, …)
│   ├── preprocess.go               # Name resolution, validation
│   ├── optimizer.go                # Logical → physical pipeline
│   ├── rule_*.go                   # Logical rewrite rules
│   ├── find_best_task.go           # Cost-based physical enumeration
│   ├── exhaust_physical_plans.go   # Generate physical alternatives
│   ├── plan_cost_ver1.go / _ver2.go # Cost models
│   ├── plan_cache*.go              # Prepared-statement plan cache
│   ├── point_get_plan.go           # Fast-path for primary-key/unique-index lookups
│   ├── indexmerge_path.go          # Index-merge access path
│   ├── index_join_path.go          # Index-join access path
│   ├── operator/logicalop/         # Logical operator structs
│   ├── operator/physicalop/        # Physical operator structs
│   ├── base/                       # Shared interfaces (LogicalPlan, PhysicalPlan)
│   ├── rule/                       # Newer rule framework
│   ├── stats/                      # Stats derivation per logical plan
│   ├── cost/                       # Cost-related helpers
│   ├── joinorder/                  # Join reorder algorithms
│   └── casetest/                   # Golden-file planner tests
├── cascades/                # Cascades-style optimizer (newer)
│   ├── cascades.go                 # Entry point
│   ├── memo/                       # Group/memo data structures
│   ├── pattern/, rule/, task/      # Pattern matching, rules, tasks
│   └── impl/                       # Physical implementation rules
├── property/                # Required physical properties (sort order, distribution)
├── funcdep/                 # Functional dependency reasoning
├── implementation/          # Logical → physical bridges (older code path)
├── memo/                    # Older memo prototype
├── cardinality/             # Cardinality estimation helpers
├── indexadvisor/            # Index recommendation
├── extstore/                # External-storage-aware planning
├── plannersession/          # Per-session planner state
├── planctx/                 # Lightweight planner context
└── util/                    # Shared utilities

Key abstractions

Type	Where	Purpose
core.LogicalPlan	pkg/planner/core/base/	Logical plan node interface. Every logical operator implements it.
core.PhysicalPlan	pkg/planner/core/base/	Physical plan node interface used by the executor.
core.PlanBuilder	pkg/planner/core/planbuilder.go	Top-level plan builder.
core.LogicalPlanBuilder	pkg/planner/core/logical_plan_builder.go	Builds logical plan trees from AST.
core.preprocess(ctx, …)	pkg/planner/core/preprocess.go	Resolves names, expands *, checks privileges, validates.
core.physicalOptimize	pkg/planner/core/optimizer.go	Logical → physical pipeline runner.
core.findBestTask	pkg/planner/core/find_best_task.go	Top-down cost enumeration over physical alternatives.
core.PlanCache	pkg/planner/core/plan_cache_instance.go	Prepared-statement plan cache (LRU per session, with global tier).
core.PointGetPlan, core.BatchPointGetPlan	pkg/planner/core/point_get_plan.go	Fast paths that skip the optimizer entirely.
cascades.Optimizer	pkg/planner/cascades/cascades.go	Cascades-style optimizer entry point.
property.PhysicalProperty	pkg/planner/property/	Required ordering/partitioning during physical enumeration.
funcdep.FDSet	pkg/planner/funcdep/	Functional dependency tracking used by several rules.
cardinality.AdjustRowCountForTableScanByLimit	pkg/planner/cardinality/	Cardinality estimation entry points.

How a plan is built

graph TD
  ast[AST<br/>ast.StmtNode] --> pre[preprocess<br/>preprocess.go]
  pre --> build[LogicalPlanBuilder<br/>logical_plan_builder.go]
  build --> log[Logical plan tree]
  log --> rules[Logical rewrite rules<br/>rule_*.go]
  rules --> stats[Stats derivation<br/>stats.go]
  stats --> phys[Physical enumeration<br/>find_best_task.go]
  phys --> cost[Cost models<br/>plan_cost_ver1/2.go]
  cost --> chosen[Chosen physical plan]
  chosen --> exec[Executor]

The complete pipeline, in code order:

1. optimize.Optimize (pkg/planner/optimize.go) is the top-level entry. The session calls this for every query.
2. Preprocess (pkg/planner/core/preprocess.go): name resolution, privilege checks, schema metadata binding. Also gathers statistics handles for tables touched.
3. Plan build: core.PlanBuilder.Build dispatches by AST type. DML statements go through LogicalPlanBuilder; DDL/SET/SHOW go through planbuilder.go's direct constructors and bypass the optimizer entirely.
4. Logical optimisation: core.logicalOptimize runs an ordered list of rules — the rule list is in core.Optimize and includes column pruning, predicate pushdown, decorrelation (rule_decorrelate.go), join reorder (rule_join_reorder*.go), aggregation pushdown (rule_aggregation_push_down.go), projection elimination (rule_eliminate_projection.go), generated-column substitution, runtime-filter generation, etc.
5. Stats derivation: core.deriveStats (in stats.go) annotates each logical plan node with row counts and column NDV based on pkg/statistics/.
6. Physical optimisation: core.findBestTask enumerates physical alternatives top-down. For each logical operator it asks the children for their best plan under specific PhysicalProperty requirements. Operators that own access paths (DataSource) generate alternatives across TableScan, IndexScan, IndexMerge, PointGet, IndexJoin, etc.
7. Cost evaluation: physical plans are scored by plan_cost_ver1.go (the legacy model) or plan_cost_ver2.go (the cost model in active use). The model is selected per session via tidb_cost_model_version.
8. Plan cache (for prepared statements): cached entries live in plan_cache_instance.go; rebuild logic in plan_cache_rebuild.go. Plans are evicted by an LRU in plan_cache_lru.go. Cacheability is checked by plan_cacheable_checker.go.

Cost model

• Version 1 (plan_cost_ver1.go): the original cost model. Still used for some operators and as a baseline.
• Version 2 (plan_cost_ver2.go): the current default. Costs are computed in TiDB cost units (close to "ns of expected wall-clock") and combined per operator. The unit-test golden file is pkg/planner/core/plan_cost_ver2_test.go.

Cost-related helpers live in pkg/planner/core/cost/. Network and storage cost factors come from session variables in pkg/sessionctx/variable/sysvar.go (see tidb_opt_*_factor).

The Cascades optimizer (pkg/planner/cascades/)

The repository is migrating toward a Cascades-style top-down optimizer. It is not yet the default for all queries; it currently handles a curated set and is gated by configuration. Files of note:

• cascades.go — entry, Optimize wraps the memo-based exploration.
• memo/ — group/expression structures.
• pattern/ — bind patterns for transformation rules.
• rule/ — transformation and implementation rules.
• task/ — exploration/implementation task scheduling.
• impl/ — physical-implementation rules.
• old/ — earlier cascades prototype kept while migration is in progress.

The main optimizer in pkg/planner/core/ remains authoritative for now. Both share access to logical plan types, statistics, and the physical operator layer.

Tests

Planner tests are heavy on golden-file comparisons. Inputs and expected outputs live in:

• pkg/planner/core/casetest/ — JSON/YAML-driven golden tests for plans.
• pkg/planner/core/casetest/rule/testdata/ — rule-specific golden files.
• tests/integrationtest/t/planner/... and tests/integrationtest/r/planner/... — SQL-level integration tests.

When changing planner output, follow .agents/skills/tidb-test-guidelines for which casetest bucket to update, and use tests/integrationtest/run-tests.sh -r <name> to record integration-test diffs (see Testing).

Integration points

• AST input: pkg/parser/ast/ (read-only).
• Schema: pkg/infoschema/ for table/column metadata, charset, indexes.
• Statistics: pkg/statistics/ for histograms/CMS sketches; pkg/statistics/handle/ for cache.
• Hints: pkg/parser/hintparser parses inline hints; core.hint_utils.go and core.hint_test.go apply them to the plan tree. SQL bindings (pkg/bindinfo/) are consulted before the optimizer runs.
• Output: core.PhysicalPlan is consumed by pkg/executor/builder.go to produce executors.
• Plan replayer: pkg/domain/plan_replayer.go dumps the planner's inputs (SQL, schema, stats, config) for offline replay.

Entry points for modification

• New rewrite rule → add a rule_<name>.go file under pkg/planner/core/, wire it into the optimizer rule list in core.logicalOptimize, and add a casetest under core/casetest/rule/.
• New physical operator → define the struct under pkg/planner/core/operator/physicalop/, implement the cost function (plan_cost_ver2.go), implement the matching LogicalPlan.exhaustPhysicalPlans enumeration (in exhaust_physical_plans.go), and add the executor builder branch in pkg/executor/builder.go.
• New access path → extend pkg/planner/core/indexmerge_path.go, index_join_path.go, or find_best_task.go's DataSource enumeration.
• Tweaking cost factors → only via tidb_opt_*_factor system variables in pkg/sessionctx/variable/sysvar.go.

The expected validation set for planner work is laid out in AGENTS.md -> Task -> Validation Matrix.