Lore

Apache Arrow is a long-running project with a layered history. The codebase started as a fork of Cloudera's Parquet C++ implementation and has accumulated reference implementations in eight languages, two sister projects (Parquet, Gandiva), a streaming execution engine (Acero), an RPC framework (Flight), a JIT compiler (Gandiva), and a federated dataset framework. This page traces those eras using git history.

Eras

Genesis: Parquet C++ as the seed (Jun 2014 – Feb 2016)

The earliest commits in this repo, found under cpp/src/parquet/, date back to 2014-06-03 — a year and a half before the apache/arrow repo itself existed. They originated in the standalone parquet-cpp repository, which was eventually merged into Arrow. By the time Arrow was announced in early 2016, Parquet C++ already had readers, writers, and schemas; that DNA still shows in cpp/src/parquet/'s separate API surface and standalone CMake configuration in cpp/src/parquet/CMakeLists.txt.

Key events:

• Jun 2014: First Parquet C++ commits.
• Feb 2016: Arrow public announcement; format/ and cpp/src/arrow/ first appear (2016-02-16).

Founding the format (Feb 2016 – early 2017)

The first months of Arrow were dominated by the format specification and the C++ skeleton. The FlatBuffers schemas in format/Schema.fbs, Message.fbs, and File.fbs were drafted, and the C++ core grew the array hierarchy (array.h was the original monolithic header that later split into cpp/src/arrow/array/), buffer infrastructure, and the IPC stream/file readers and writers. PyArrow appeared early — its first python/pyarrow/ commits are from 2016-03-09, only a few weeks after Arrow's public launch.

Defining decisions from this era still in the codebase:

• arrow::Status and arrow::Result<T> for error handling.
• Reference-counted arrow::Buffer for memory ownership.
• A pluggable MemoryPool for allocator control.
• The IPC envelope: schema → dictionary batches → record batches.

Polyglot expansion (2017 – 2018)

The second wave brought the rest of the language wrappers in this repo:

• 2017-03-16: c_glib/ — GObject bindings, the foundation for many bindings beyond C++.
• 2018-05-24: ruby/red-arrow/ — Ruby gems on top of c_glib.
• 2018-05-29: cpp/src/gandiva/ — Gandiva LLVM JIT, contributed by Dremio.
• 2018-09-07: r/R/ — R6 classes wrapping the C++ library.
• 2018-09-08: matlab/ — MATLAB bindings.
• 2018-09-20: cpp/src/arrow/flight/ — Flight RPC over gRPC.

Each new language wrapper used the same pattern: bind to the C++ library through the most idiomatic FFI for that language. Java, Go, JS, and Rust pursued the alternative path of independent implementations — verifying compatibility through the integration test suite at cpp/src/arrow/integration/.

The dataset and execution engine years (2019 – 2023)

Arrow's ambition expanded from "an in-memory format" to "a query execution stack" during this period:

• 2019-06-13: cpp/src/arrow/dataset/ — the dataset framework, supporting partitioned and remote data sources with format adapters for Parquet, IPC, CSV, JSON, and ORC.
• 2019–2022: Compute kernel growth in cpp/src/arrow/compute/ — scalar arithmetic, casts, comparisons, vector operations (sort, filter, take), aggregates, and a hash-aggregate engine.
• 2022: The "exec engine" lived inline in cpp/src/arrow/compute/exec/ for several releases.
• Apr 2023: Acero was carved out into its own cpp/src/arrow/acero/ directory, becoming a first-class subsystem with its own CMake target, package, and pkg-config file.

Substrait support — the cross-engine plan format — landed in cpp/src/arrow/engine/substrait/ during the same window, allowing Acero to consume plans produced by other engines.

Modern hardware and storage (2024 – 2026)

The most recent era has been about extending Parquet, taking advantage of newer SIMD and ARM hardware, and integrating with cloud-native filesystems:

• Modular Parquet encryption and bloom filter writers under cpp/src/parquet/encryption/ and cpp/src/parquet/bloom_filter*.
• Page index reader and writer (cpp/src/parquet/page_index.h).
• Geospatial Parquet support (cpp/src/parquet/geospatial/).
• Azure Blob Storage filesystem (cpp/src/arrow/filesystem/azurefs.cc) joining S3, GCS, and HDFS.
• Aggressive ARM SVE dispatch and constexpr SIMD work (commits in Apr 2026: GH-49756 "SVE dynamic dispatch", GH-49835 "constexpr dynamic dispatch with static dispatch when possible").
• DLPack bridge (cpp/src/arrow/c/dlpack.cc, python/pyarrow/_dlpack.pxi) for zero-copy interchange with PyTorch/JAX/TensorFlow.

Releases moved to roughly quarterly cadence, with major versions 19.0.0 (Jan 2025), 20.0.0 (Apr 2025), 21.0.0 (Jul 2025), 22.0.0 (Oct 2025), 23.0.0 (Jan 2026), and 24.0.0 (Apr 2026). Maintenance branches (maint-23.0.x, maint-24.0.0) carry patch releases.

Longest-standing features

• Parquet C++ readers and writers (cpp/src/parquet/file_reader.cc, file_writer.cc) — the oldest sustained code in the repo, originally from 2014 and still actively maintained.
• Arrow IPC reader/writer (cpp/src/arrow/ipc/reader.cc, writer.cc) — present since Arrow's first months in 2016 and still the canonical entry point for streaming Arrow data.
• The C++ array hierarchy (cpp/src/arrow/array/) — the array base class and primitive/binary/list/struct subclasses have been stable since 2016, even as the package was reorganized from one big array.cc into the current many-file layout.
• PyArrow's Cython surface (python/pyarrow/lib.pyx) — has been the public Python entry point since March 2016.

Major rewrites and reorganizations

• Array layer split. What started as a single cpp/src/arrow/array.cc was split into cpp/src/arrow/array/array_base.cc, array_primitive.cc, array_binary.cc, array_nested.cc, array_dict.cc, and array_run_end.cc to keep individual files manageable.
• Compute → Acero extraction. The execution engine (cpp/src/arrow/compute/exec/) was promoted to cpp/src/arrow/acero/ in April 2023 as its responsibilities outgrew "compute".
• Parquet metadata churn. Parquet's metadata.cc has gone through several refactors as page index support, modular encryption, and column-chunk size statistics were added.
• Filesystem abstraction. The cpp/src/arrow/filesystem/ API was introduced after several iterations of separate S3FileSystem/HdfsFileSystem interfaces, unifying everything under arrow::fs::FileSystem.

Deprecated or replaced features

• Feather V1. The original on-disk single-table format was replaced by Feather V2, which is the Arrow IPC file format with a .feather extension. The V1 reader still exists in cpp/src/arrow/ipc/feather.cc for backward compatibility.
• Tensorflow adapter. cpp/src/arrow/adapters/tensorflow/ is a small remnant of an early TensorFlow integration.
• HDFS via libhdfs. The HDFS filesystem in cpp/src/arrow/io/hdfs.cc and cpp/src/arrow/filesystem/hdfs.cc is still present but has seen less investment than the cloud filesystems built on top of REST APIs (S3, GCS, Azure).
• JVM compat shim. python/pyarrow/jvm.py is a small adapter between PyArrow and Arrow Java; it has remained largely unchanged for years.

Growth trajectory

By release cadence and contributor counts:

• 2016–2017: a few dozen contributors, mostly dual-hatted between Cloudera and Databricks.
• 2018–2020: the polyglot expansion grew the contributor base to several hundred — language-specific maintainers (Sutou Kouhei for Ruby/c_glib, Neal Richardson and later Nic Crane for R, Wes McKinney and Joris Van den Bossche for Python).
• 2021–2026: 1,477 unique committers total, with a stable core of ~10 maintainers driving each subsystem. Sutou Kouhei has the most commits in the last year (214) and is the single most prolific maintainer, particularly across Ruby, c_glib, and CI infrastructure.

The repo currently sits around 18,792 commits with active development across every subsystem listed above.