Substrait

Active contributors: Weston Pace, Antoine Pitrou, David Li

Substrait is a cross-engine specification for query plans. The Arrow Substrait adapter lives in cpp/src/arrow/engine/substrait/ and converts Substrait plans into Acero ExecPlans, letting Arrow serve as an execution backend for any frontend that produces Substrait.

Purpose

Decouple plan production from plan execution. A frontend like DuckDB, pandas-on-arrow, or DataFusion can produce a Substrait plan and ask Acero to execute it without baking in any Acero-specific assumptions. Conversely, Acero can be the engine behind any Substrait-emitting tool.

Files

cpp/src/arrow/engine/
├── api.h
├── arrow-substrait.pc.in
├── simple_extension_type_internal.h
└── substrait/
    ├── api.h                      # Public API
    ├── plan_internal.cc/.h        # Substrait Plan ↔ Acero Declaration
    ├── relation_internal.cc/.h    # Relations: Read, Project, Filter, Aggregate, Join, Sort, Set, Fetch
    ├── expression_internal.cc/.h  # Substrait expressions ↔ Arrow expressions
    ├── type_internal.cc/.h        # Substrait types ↔ Arrow types
    ├── extension_set.cc/.h        # Function and type extension URIs
    ├── extension_types.cc/.h      # Built-in extension types (UUID, JSON, ...)
    ├── extended_expression_internal.cc/.h
    ├── options.h                  # ConversionOptions
    ├── serde.cc/.h                # Top-level serialize/deserialize entry points
    ├── serde_test.cc (~6,300 lines)  # Round-trip tests for every relation type
    ├── util.cc/.h                 # Helpers
    └── visibility.h

Pipeline

graph LR
    SubstraitPlan["Substrait Plan (protobuf)"] --> Deserialize["DeserializePlan"]
    Deserialize --> Relations["Walk relations bottom-up"]
    Relations --> Convert["Convert each relation to a Declaration"]
    Convert --> Sequence["Compose Declarations into an ExecPlan"]
    Sequence --> Acero["Acero executes the plan"]

Relation mapping

Substrait defines a small set of relational operators. The adapter maps each to an Acero node:

Substrait relation	Acero node
read_rel (NamedTable, LocalFiles, VirtualTable)	source or dataset scanner
filter_rel	filter
project_rel	project
aggregate_rel	aggregate
join_rel	hashjoin
sort_rel	order_by
set_rel (UNION, INTERSECTION, ...)	union (with extras)
fetch_rel (LIMIT/OFFSET)	fetch
extension_rel	Custom — used for ExecPlan-only constructs

Each conversion lives in relation_internal.cc.

Expression mapping

Substrait expressions (literals, field references, scalar functions, IF/THEN/ELSE) map to arrow::compute::Expression (expression_internal.cc). Function names are namespaced by an extension URI — Substrait doesn't have a global "+" function; instead, a plan declares it imports from https://github.com/substrait-io/substrait/blob/main/extensions/functions_arithmetic.yaml#add:i32_i32. The adapter resolves these to Arrow compute functions by lookup.

extension_set.h is the bridge between Substrait extension URIs and Arrow's compute function registry. Built-in extensions cover arithmetic, comparison, boolean, datetime, string, aggregation, and round-cast functions.

Type mapping

Substrait types map cleanly to Arrow types (type_internal.cc):

Substrait	Arrow
i8 … i64	int8 … int64
fp32, fp64	float32, float64
string, binary, fixed_char(N), var_char(N)	utf8, binary, fixed_size_binary
boolean	bool
date, time, timestamp, timestamp_tz	date32/date64, time32/time64, timestamp[ns], timestamp[ns, tz=UTC]
decimal(p, s)	decimal128(p, s)
list<T>	list<T>
struct<...>	struct<...>
map<K, V>	map<K, V>
User-defined types via extension URIs	ExtensionType

Read relations

The most complex conversion is the read relation. Substrait can describe:

• A named table (just the name; the consumer resolves to its catalog).
• A local files read with a list of file paths.
• A virtual table with literal rows.

The adapter resolves named tables via a callback the user provides (ConversionOptions::named_table_provider). Local files reads can be wired to the dataset framework if the option is set.

Round-trip support

serde.cc provides both directions: DeserializePlan (Substrait → Acero) and SerializePlan (Acero → Substrait). Not every Acero plan is expressible in pure Substrait — for those, the adapter emits Substrait's extension_rel to carry an Acero declaration verbatim.

Test suite

serde_test.cc is 6,345 lines — among the largest test files in the project. It round-trips every relation, every expression node, every type, every aggregation function, and exercises the extension-set machinery exhaustively.

PyArrow integration

python/pyarrow/_substrait.pyx exposes pyarrow.substrait.run_query(plan_bytes) to execute a Substrait plan and return a RecordBatchReader. This is the integration point used by libraries like Ibis when they target Acero as the execution engine.

Entry points for modification

• Adding a new relation: extend relation_internal.cc with the relation's conversion. Add a corresponding test in serde_test.cc.
• Adding a new function extension: register the function URI in the extension set; map it to an Arrow compute function name.
• Adding a new type: extend type_internal.cc to convert in both directions; add a format string in the C data interface where relevant.

For the engine that consumes the converted plan, see Acero.