Parquet

Active contributors: Antoine Pitrou, Zehua Zou, mwish, Sutou Kouhei

Apache Parquet C++ in cpp/src/parquet/ is a sister project to Arrow. It originated as the standalone parquet-cpp repository in 2014, predates Arrow itself, and was merged into the Arrow repo in 2018. The library implements a complete Parquet reader and writer with bloom filters, page indexes, modular encryption, and an Arrow-friendly API in cpp/src/parquet/arrow/.

Purpose

Provide a high-performance, fully-featured Parquet implementation that integrates with Arrow's columnar format and ecosystem.

Layout

cpp/src/parquet/
├── api/                # Public umbrella headers (api/reader.h, api/writer.h, api/io.h, api/schema.h)
├── arrow/              # Arrow integration: read Parquet → Arrow Table, write Arrow → Parquet
├── encryption/         # Modular encryption (column-level encryption keys, KMS clients)
├── geospatial/         # WKB Geometry/Geography logical type support
├── parquet.thrift      # Thrift schema (the format spec)
├── file_reader.{h,cc}  # ParquetFileReader
├── file_writer.{h,cc}  # ParquetFileWriter
├── column_reader.{h,cc} (~89 KB)   # Column-level read API
├── column_writer.{h,cc} (~114 KB)  # Column-level write API
├── column_page.h       # Page abstractions (DataPage, DictionaryPage, IndexPage)
├── encoding.h, encoder.cc, decoder.cc  # Plain, dictionary, RLE, delta, byte-stream-split codecs
├── statistics.{h,cc}, size_statistics.{h,cc}  # Per-column/per-page stats
├── metadata.{h,cc} (~83 KB)  # File/row group/column chunk metadata
├── schema.{h,cc}       # SchemaDescriptor: nested + repetition/definition levels
├── level_conversion.{h,cc}   # Compact rep/def levels representation
├── bloom_filter.{h,cc}, bloom_filter_writer.{h,cc}, bloom_filter_reader.{h,cc}
├── page_index.{h,cc} (~43 KB)  # Per-page min/max statistics
├── chunker_internal.{h,cc} + chunker_internal_codegen.py  # Content-defined chunking
├── stream_reader.{h,cc}, stream_writer.{h,cc}  # Row-oriented convenience API
├── properties.h, properties.cc                  # Reader/Writer properties (config)
├── exception.h, exception.cc, types.{h,cc}, type_fwd.h
└── thrift_internal.h   # Thrift parsing helpers

Read pipeline

graph LR
    Open["ParquetFileReader::Open"] --> Footer["Read + parse FileMetaData (Thrift)"]
    Footer --> RowGroups["Iterate row groups (filter by stats)"]
    RowGroups --> ColumnChunk["For each column chunk"]
    ColumnChunk --> Pages["Read DictionaryPage + DataPages"]
    Pages --> Decode["Decode by encoding (PLAIN, RLE_DICT, DELTA_*, BYTE_STREAM_SPLIT)"]
    Decode --> RepDef["Reconstruct rep/def levels"]
    RepDef --> Output["Emit Arrow ArrayData"]

Key types:

• ParquetFileReader (file_reader.h) — opens a file, parses the footer, exposes RowGroupReaders.
• RowGroupReader — exposes ColumnReaders.
• ColumnReader (column_reader.h) — pulls pages, decodes them, returns batches of values + def/rep levels.
• ColumnScanner (column_scanner.h) — high-level wrapper for sequential column scans.

Write pipeline

ParquetFileWriter (file_writer.h) wraps ParquetFileWriterContents. Each call to AppendRowGroup produces a RowGroupWriter; each call to NextColumn produces a ColumnWriter that the user feeds with values + def/rep levels (or, more commonly, with Arrow Arrays through cpp/src/parquet/arrow/).

column_writer.cc is the largest file in the Parquet directory at ~114 KB and handles all the encoding strategies, dictionary management, statistics accumulation, page boundary decisions, and bloom filter / page index writing.

Encodings

encoding.h declares every encoding the format supports:

• PLAIN — values stored as-is.
• PLAIN_DICTIONARY / RLE_DICTIONARY — dictionary-encoded indices with run-length packing.
• RLE / BIT_PACKED — for repetition/definition levels.
• DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY — delta encodings for numeric and binary columns.
• BYTE_STREAM_SPLIT — column splitting that improves compressibility for floats.

encoder.cc (70 KB) and decoder.cc (98 KB) implement every encoder and decoder. Hot paths use the bit-packing kernels from cpp/src/arrow/util/bpacking_*.h.

Bloom filters and page indexes

Both are forms of zone-map-style row skipping:

• Bloom filter (bloom_filter.h): per-column probabilistic filter. BloomFilterReader lets a query like WHERE id = 12345 skip whole row groups.
• Page index (page_index.h): per-page min/max statistics + page locations. Lets WHERE date < '2024-01-01' skip individual pages within a row group.

Page index reading is implemented by the column reader; writing happens at the ColumnWriter level when properties enable it.

Modular encryption

cpp/src/parquet/encryption/ implements Parquet's modular encryption (PME) — column-level encryption keys with optional integration with a Key Management Service (KMS):

File	Purpose
crypto_factory.cc	Builds encryption configurations from properties.
encryption.h	Public encryption API.
encryption_internal.cc	Block ciphers (AES-GCM, AES-CTR).
kms_client.h, kms_client_factory.h	Pluggable KMS interface.
local_wrap_kms_client.cc	Built-in local-wrap KMS.
key_metadata.cc, key_material.cc	Key wrapping.

The dataset framework wires this through cpp/src/arrow/dataset/parquet_encryption_config.h.

Geospatial

cpp/src/parquet/geospatial/ (added recently) handles the Geometry and Geography logical types from the Parquet GeoParquet specification. Stores WKB-encoded values with bounding-box statistics.

Arrow integration

cpp/src/parquet/arrow/ is the bridge between Parquet's row-group + column-chunk model and Arrow's record batches:

• cpp/src/parquet/arrow/reader.cc — reads Parquet files into Arrow tables/batches with column projection, predicate pushdown, async IO, and parallelism.
• cpp/src/parquet/arrow/writer.cc — writes Arrow tables to Parquet, deciding chunk sizes, encodings, and compression.
• cpp/src/parquet/arrow/schema.cc — round-trips schemas: Arrow types ↔ Parquet logical types.

The dataset adapter cpp/src/arrow/dataset/file_parquet.cc builds on cpp/src/parquet/arrow/ to drive partitioned Parquet datasets.

Stream API

For applications that prefer row-by-row access over batch access, stream_reader.h and stream_writer.h provide an iostream-like API:

parquet::StreamReader reader{ParquetFileReader::Open(infile)};
int32_t id;
std::string name;
while (!reader.eof()) {
  reader >> id >> name >> parquet::EndRow;
}

This is a thin wrapper over the column-level API.

Test infrastructure

The Parquet test suite is one of the largest in the repo:

• arrow/arrow_reader_writer_test.cc (~6,000 lines) — comprehensive Arrow ↔ Parquet round trips.
• column_reader_test.cc, column_writer_test.cc (each ~70-100 KB) — encoding round-trips.
• metadata_test.cc, statistics_test.cc, schema_test.cc (each 30-100 KB) — metadata correctness.
• bloom_filter_test.cc, page_index_test.cc, bloom_filter_reader_writer_test.cc — skip-feature tests.
• chunker_internal_test.cc (~74 KB) — content-defined chunking.

Cross-implementation tests compare the C++ implementation against Parquet-MR (Java) and parquet-rs (Rust) outputs.

Performance work

Recent commits (2026-04) include:

• GH-47657: integer overflow fix when coercing timestamps.
• GH-49896: short-buffer rejection in IPC reader (also affects Parquet metadata parsing).
• Continuing AVX2/AVX-512 work in encoding.cc and the bpacking kernels.

encoding_benchmark.cc (~67 KB) and column_io_benchmark.cc are the main benchmarks.

Language wrapper integration

• PyArrow. python/pyarrow/_parquet.pyx (~83 KB) wraps everything. Public API in pyarrow.parquet.
• R. r/R/parquet.R and r/src/parquet.cpp.
• C-GLib + Ruby. c_glib/parquet-glib/ and ruby/red-parquet/.

Entry points for modification

• Adding a new encoding: implement an Encoder subclass in encoder.cc, a Decoder subclass in decoder.cc, register in encoding.cc, and add a test in encoding_test.cc.
• Adding a logical type: extend cpp/src/parquet/types.h, the Thrift schema (regenerate cpp/src/generated/parquet_types.{h,cc,tcc}), and the Arrow ↔ Parquet schema converter in arrow/schema.cc.
• Performance: profile with column_io_benchmark.cc or encoding_benchmark.cc.