Dataset

Active contributors: Antoine Pitrou, Joris Van den Bossche, Felipe Aramburu, Sutou Kouhei

The dataset framework (cpp/src/arrow/dataset/) lets a single logical query span multiple files in different formats across local disk and remote object stores. It builds on the I/O layer and uses Acero for execution.

Purpose

Read and write partitioned, multi-file, multi-format datasets with column projection, filter pushdown, and efficient parallelism. Datasets are how PyArrow and the R package handle larger-than-memory data on Parquet, IPC, CSV, JSON, and ORC files.

Concepts

• Dataset. A logical collection of one or more Fragments sharing a unified schema. Subclasses: FileSystemDataset, InMemoryDataset, UnionDataset.
• Fragment. A discrete unit, typically a single file or row group. Subclasses: ParquetFragment, IpcFragment, CsvFragment, JsonFragment, OrcFragment, InMemoryFragment.
• FileFormat. A pluggable adapter that knows how to read/write a specific format. Built-in: FileFormatParquet, FileFormatIpc, FileFormatCsv, FileFormatJson, FileFormatOrc.
• Partitioning. A scheme that maps directory/file paths to column values. Built-in: HivePartitioning, DirectoryPartitioning, FilenamePartitioning.
• Scanner. Executes a query over a dataset: applies projection, filter pushdown, and parallel I/O. Returns an AsyncGenerator<RecordBatch> or a RecordBatchReader.

Files in this directory

File	Purpose
dataset.h / .cc	Dataset, Fragment, factories.
discovery.h / .cc	DatasetFactory. Builds a dataset from a filesystem path by inferring schema and partitioning.
partition.h / .cc	Partitioning types.
file_base.h / .cc	FileFormat and FileFragment shared logic.
file_parquet.h / .cc (~49 KB)	Parquet adapter. The biggest format adapter — Parquet has the most options.
file_ipc.h / .cc	Arrow IPC adapter.
file_csv.h / .cc	CSV adapter.
file_json.h / .cc	JSON adapter.
file_orc.h / .cc	ORC adapter.
scanner.h / .cc (~49 KB)	The scanner. Implements push-down, parallelism, and the streaming output.
scan_node.cc	The Acero node that wraps a scanner.
dataset_writer.h / .cc	Writes a RecordBatchReader to a partitioned dataset.
forest_internal.h, subtree_internal.h	Helpers for partition tree management.
parquet_encryption_config.h	Pass-through for Parquet modular encryption.
plan.h / .cc	Convenience wrappers around Acero plan construction.
projector.h / .cc	Column projection helper.

Scan pipeline

graph LR
    DSF["DatasetFactory.Finish()"] --> DS["FileSystemDataset"]
    DS --> Fragments["Iterate fragments"]
    Fragments --> FilterPart["Filter by partition expression"]
    FilterPart --> OpenFile["FileFormat.OpenReader(filesystem, fragment)"]
    OpenFile --> Stats["Apply pushdown filter against per-row-group stats"]
    Stats --> ReadBatches["Read batches with column projection"]
    ReadBatches --> AceroPlan["Optional: feed into Acero ExecPlan"]

Steps in detail:

1. The user constructs a DatasetFactory from a filesystem path or an explicit list of fragments.
2. Finish() resolves a unified schema (either user-supplied or inferred) and produces a Dataset.
3. NewScan() returns a ScannerBuilder that the user configures with projection, filter, and threading options.
4. ToRecordBatches() (or ToTable, ToBatches, Head) starts the scan. The scanner enumerates fragments, applies the filter expression to partition values to skip whole fragments, opens the surviving fragments, and pushes down the projection + filter to the format adapter when the format supports it.
5. The Parquet adapter pushes down by:
   • Reading the file metadata.
   • Filtering row groups by the file's column statistics.
   • Reading only the requested columns (column-index projection).
   • Optionally reading only the requested row ranges using the page index.
6. The format adapter returns an AsyncGenerator<RecordBatch> that the scanner consumes via arrow::AsyncGenerator operators.
7. The user receives a stream of record batches. If the user pipelines into an Acero plan, scan_node.cc is the bridge.

Partitioning

HivePartitioning parses paths like year=2024/month=01/file.parquet into (year=2024, month=01).

DirectoryPartitioning parses positional paths like 2024/01/file.parquet against a fixed schema, producing the same column values.

FilenamePartitioning extracts column values from the filename instead of the directory.

The C++ types are extensible: the user can subclass Partitioning to add a custom scheme (e.g., S3 prefix conventions specific to their lake).

Writing datasets

dataset_writer.cc (~30 KB) implements writing a RecordBatchReader to a partitioned dataset. It can:

• Partition output by Hive or directory partitioning.
• Limit rows per file (max_rows_per_file) and rows per group (max_rows_per_group).
• Detect existing files and choose a deletion or overwrite strategy.
• Hand off to a FileWriter from the configured FileFormat.

The recently added Acero node write_node_test.cc covers writing as part of an ExecPlan.

Schema unification

When a dataset spans multiple files with subtly different schemas (e.g., one file has an extra column, or some files have different but compatible types), the scanner attempts to unify schemas. dataset_internal.h and discovery.cc hold the unification logic. Promotion rules are conservative — incompatible types raise an error rather than silently coercing.

Tests

scanner_test.cc is enormous (~123 KB) and covers every combination of partitioning, projection, filter, format, and threading. partition_test.cc, file_parquet_test.cc, file_csv_test.cc, file_json_test.cc, file_ipc_test.cc, and file_orc_test.cc exercise each format in isolation. file_parquet_encryption_test.cc covers the Parquet modular encryption path.

test_util_internal.h (~85 KB) provides shared fixtures.

Language wrapper integration

• PyArrow. python/pyarrow/_dataset.pyx (~165 KB) wraps the entire surface. Format-specific extras live in _dataset_parquet.pyx, _dataset_orc.pyx. Public API in pyarrow.dataset (dataset.py).
• R arrow. r/R/dataset.R and friends. The dplyr backend (r/R/dplyr-*.R) builds Acero plans on top of dataset scanners, giving R users SQL-like analytics over partitioned Parquet directly.

Entry points for modification

• Adding a new file format: subclass FileFormat and FileFragment. The CSV adapter (file_csv.cc) is a good minimal template; the Parquet adapter is the maximal one.
• Adding a partitioning scheme: subclass Partitioning. Most consumers use the built-in three.
• Optimizing scanner concurrency: scanner.cc orchestrates the IO and CPU pools. scanner_benchmark.cc exists for measuring changes.