Expression engine

The expression engine evaluates SQL expressions over column-batches. It is one of the largest packages in the repo because TiDB implements MySQL's full builtin function set.

Purpose

pkg/expression/ provides:

• The Expression tree (constants, columns, scalar functions).
• A registry of every MySQL builtin function and its TiDB-specific extensions.
• Two evaluation modes: scalar (row-at-a-time) and vectorized (chunk-at-a-time).
• Pushdown declaration tables that say which expressions can be pushed to TiKV / TiFlash.
• Helpers for charset/collation, constant folding, and constant propagation.
• The expr_to_pb and distsql_builtin codecs that serialise expressions to TiKV's coprocessor protocol.

Directory layout

pkg/expression/
├── expression.go              # Expression interface, Constant, base utilities
├── column.go                  # Column expression
├── scalar_function.go         # ScalarFunction expression
├── constant.go                # Constant expression
├── constant_fold.go           # Constant folding
├── constant_propagation.go    # Constant propagation across predicates
├── builtin.go                 # Function class registry, eval interfaces
├── builtin_*.go               # Individual builtin classes (math, time, string, json, …)
├── builtin_*_vec.go           # Vectorized counterparts
├── builtin_*_vec_generated.go # Generated vectorized helpers
├── builtin_threadsafe_generated.go
├── builtin_threadunsafe_generated.go
├── evaluator.go               # Scalar evaluation driver
├── vectorized.go              # Vectorized evaluation driver
├── chunk_executor.go          # ChunkExecutor wraps Expression for executor.go
├── infer_pushdown.go          # Pushdown classification (TiKV / TiFlash / cop)
├── expr_to_pb.go              # Encode an Expression to coprocessor protobuf
├── distsql_builtin.go         # Decode/encode coprocessor builtin descriptors
├── collation.go               # Collation handling
├── grouping_sets.go           # GROUP BY ROLLUP / GROUPING SETS support
├── extension.go               # Extension hooks
├── exprctx/                   # ExprContext interface
├── expropt/                   # Optional expression bits (per-extension)
├── exprstatic/                # Stateless context wrappers
├── sessionexpr/               # Session-bound expression context
├── aggregation/               # Aggregate function descriptors
├── helper.go, util.go         # Shared helpers
└── generator/                 # Code generator for *_vec_generated.go

Key abstractions

Type	File	Purpose
Expression	pkg/expression/expression.go	Top-level interface. Implemented by Column, Constant, ScalarFunction, CorrelatedColumn.
Column	pkg/expression/column.go	A reference to a result-set column.
Constant	pkg/expression/constant.go	Literal value with optional deferred constant (parameter).
ScalarFunction	pkg/expression/scalar_function.go	A function call: name, args, return type, function class.
functionClass	pkg/expression/builtin.go	Registers a builtin function and produces a builtinFunc per signature.
builtinFunc	pkg/expression/builtin.go	The runtime function. Implements both scalar (evalInt/evalReal/...) and vectorized (vecEvalInt/...) entry points.
EvalContext	pkg/expression/exprctx/	Minimal context for eval: time zone, current user, statement context.
ExprContext	pkg/expression/exprctx/	Bigger context used during planning.
ChunkExecutor	pkg/expression/chunk_executor.go	Glues Expression to chunk.Chunk filling.
aggregation.AggFuncDesc	pkg/expression/aggregation/	Describes an aggregate (e.g., SUM, COUNT, APPROX_COUNT_DISTINCT).

How evaluation works

graph LR
  exec[Executor calls<br/>Expression.Eval / VecEval] --> sf[ScalarFunction]
  sf --> bf[builtinFunc impl]
  bf -->|vectorized| chunk[Chunk column writes]
  bf -->|scalar| row[Single Row eval]
  row --> ret[Datum]
  chunk --> ret2[Column batch]

• Scalar evaluation: Expression.Eval(EvalContext, Row) returns a types.Datum. evaluator.go walks the expression tree and folds children. Used in row-oriented contexts (e.g., point-gets, single-row predicates).
• Vectorized evaluation: Expression.VecEval{Int,Real,String,Decimal,Time,Duration,JSON}(EvalContext, Chunk, Column) writes a whole column at once. Most operators use this path because it avoids per-row interface dispatch and keeps memory traffic linear.

The scalar/vectorized split is per type. Each builtinFunc declares its return type and implements only the matching pair of methods. The *_vec_generated.go files are produced by pkg/expression/generator/ from declarative templates and contain the vectorized fast paths for monomorphic helpers (e.g., per-type compare).

Function categories (builtin_*.go)

File pair	What it covers
builtin_math.* / _vec.*	ABS, MOD, RAND, LOG, EXP, …
builtin_arithmetic.* / _vec.*	+, -, *, /, decimal/integer arithmetic
builtin_compare.* / _vec*.go	=, <, <=>, IF, LEAST, GREATEST, IN
builtin_control.* / _vec_generated.*	CASE, WHEN, IF, IFNULL, NULLIF
builtin_string.* / _vec*	CONCAT, SUBSTRING, LIKE, LENGTH, LPAD, …
builtin_time.* / _vec*	NOW, DATE_ADD, STR_TO_DATE, EXTRACT, …
builtin_json.* / _vec.*	JSON_EXTRACT, JSON_OBJECT, JSON_VALID, …
builtin_cast.* / _vec.*	All cross-type casts
builtin_op.* / _vec.*	AND, OR, XOR, NOT, bit ops
builtin_other.* / _vec*	Misc (COALESCE, BIT_COUNT, INET_NTOA, …)
builtin_info.* / _vec.*	Server info functions (USER, VERSION, DATABASE, TIDB_VERSION)
builtin_miscellaneous.* / _vec.*	UUID, sleep, get_lock, IP helpers
builtin_encryption.* / _vec.*	MD5, SHA*, AES_ENCRYPT, RANDOM_BYTES
builtin_grouping.*	GROUPING_ID, ROLLUP support
builtin_regexp.*	REGEXP_LIKE, REGEXP_REPLACE, REGEXP_INSTR
builtin_like.*, builtin_ilike.*	LIKE, ILIKE (TiDB extension)
builtin_fts.go, fts_helper.go	Full-text search builtins
builtin_vec.*, builtin_vec_vec.*, vs_helper.go	Vector-search builtins (VEC_*, cosine distance)

Pushdown classification

infer_pushdown.go declares which functions can be pushed down to:

• TiKV via the coprocessor (the row-store engine).
• TiFlash via MPP / batch coprocessor (columnar engine).

Some functions are only safe to push when arguments are of certain types, or when collation is compatible. The classification is consulted by the planner's predicate-pushdown rule.

Constant folding and propagation

• constant_fold.go folds expressions like 1+1 to 2 at planning time.
• constant_propagation.go propagates constraints across predicates (a = 1 AND a = b → a = 1 AND b = 1). It is invoked by the planner's rule_predicate_push_down.go pipeline.

Tests

• bench_test.go — micro-benchmarks for hot eval paths.
• *_test.go next to each builtin family.
• typeinfer_test.go — golden tests for return-type inference, an enormous fixture file because every signature contributes a row.
• expr_to_pb_test.go — confirms the protobuf encoding for every pushdown-eligible function.
• pkg/expression/integration_test/ — end-to-end SQL-level tests for builtins.

Integration points

• Planner uses pkg/expression/ to build Expression trees during plan construction.
• Executor calls VecEval* per chunk during query execution.
• Storage layer (pkg/store/copr) calls expr_to_pb.go to push expressions to TiKV/TiFlash.
• Statistics and stats handles (pkg/statistics/) re-use vectorized builtins for sample/CMS sketch construction.
• Extensions (pkg/extension/) can register new builtins via extension.go.

Entry points for modification

• Adding a new builtin → add a functionClass and builtinFunc in the right builtin_<family>.go, register it in builtin.go's funcs map, implement the vectorized counterpart in builtin_<family>_vec.go, declare pushdown applicability in infer_pushdown.go, and add to pkg/parser only if a new keyword is needed.
• Type promotion or implicit-cast tweaks → builtin_cast.go and the corresponding vectorized file. Watch typeinfer_test.go golden output.
• Pushdown changes → only infer_pushdown.go and expr_to_pb.go. Coordinate with the corresponding TiKV / TiFlash builtin implementations.

The validation matrix in AGENTS.md -> Task -> Validation Matrix requires targeted expression unit tests with edge-case coverage for any change here.