Fun facts

A handful of curiosities discovered while reading the Arrow codebase.

The oldest code in the repo predates Arrow

The earliest commit in this repository is from June 2014, almost two years before Arrow was announced in February 2016. That's because cpp/src/parquet/ was originally the standalone parquet-cpp repository, which was merged into Arrow as a sister project. Files like cpp/src/parquet/types.cc carry comments and idioms from that earlier project. Even today, Parquet keeps a separate API in cpp/src/parquet/api/ that resembles its pre-merger interface more than the modern Arrow style.

"Acero" was named after a kind of maple tree

The streaming execution engine was carved out of cpp/src/arrow/compute/exec/ into cpp/src/arrow/acero/ in April 2023. The name comes from the Latin/Italian for maple — fitting for a query engine whose pipelines branch and merge data flows. The package even has its own pkg-config file (cpp/src/arrow/acero/arrow-acero.pc.in) and downstream consumers like the R package use arrow::acero as a hard module boundary.

A 11,000-line generated C++ header

cpp/src/arrow/util/bpacking_simd512_generated_internal.h is the single longest source file at over 11,000 lines. It's not handwritten — it's emitted by bpacking_simd_codegen.py, which generates AVX-512 bit-packing kernels for every bit-width from 1 to 32. There's a sibling bpacking_scalar_generated_internal.h (5,936 lines) for the scalar path. Together they form the bit-packing fast path used by Parquet's RLE/dictionary decoders.

Gandiva ships a precompiled bitcode blob

Gandiva, the LLVM-based expression compiler in cpp/src/gandiva/, doesn't JIT-compile every helper from scratch at runtime. Many helpers — decimal arithmetic, regex, hashing, casting, datetime ops — are written in C++ and pre-compiled to LLVM bitcode at build time. The bitcode is then linked into the JIT'd module at runtime. The build artifact lives at cpp/src/gandiva/precompiled/ and is generated by make_precompiled_bitcode.py. This is why Gandiva needs clang available even when the rest of Arrow is built with GCC.

A Python script that lives next to a debugger

cpp/gdb_arrow.py is a 70 KB GDB Python pretty-printer for Arrow types. Once loaded with source gdb_arrow.py it teaches GDB how to render arrow::Array, arrow::DataType, arrow::Schema, and most other Arrow types in a way that mirrors the Arrow C++ pretty printer. CMake installs it next to libarrow.so so distros can ship it as part of -dev packages.

The TPC-H benchmark generator is built into the engine

Acero ships its own TPC-H data generator at cpp/src/arrow/acero/tpch_node.cc (~149 KB — one of the largest single source files). It's an ExecNode like any other. To benchmark Acero's hash join or aggregate, you can plug a TpchNode into an ExecPlan and stream synthetic line items, customers, and orders without touching the disk.

R has 22 dplyr files

The R package's dplyr backend isn't a single file — it's split across dplyr-arrange.R, dplyr-by.R, dplyr-collect.R, dplyr-count.R, dplyr-distinct.R, dplyr-filter.R, dplyr-funcs-agg.R, dplyr-funcs-augmented.R, dplyr-funcs-conditional.R, dplyr-funcs-datetime.R, dplyr-funcs-doc.R, dplyr-funcs-math.R, dplyr-funcs-simple.R, dplyr-funcs-string.R, dplyr-funcs-type.R, dplyr-funcs.R, dplyr-glimpse.R, dplyr-group-by.R, dplyr-join.R, dplyr-mutate.R, dplyr-select.R, dplyr-slice.R, dplyr-summarize.R, dplyr-union.R, and dplyr.R. Each one translates a category of dplyr verbs into Acero ExecPlans via query-engine.R. The split keeps any single file readable.

The merge_arrow_pr.py script

dev/merge_arrow_pr.py is the script committers run to merge a PR. It enforces the project's commit message conventions (the GH-12345: [Component] Title (#PR) form), squashes the PR, and posts a comment on the issue. The companion merge_arrow_pr.sh and test_merge_arrow_pr.py mean the merge tool itself has its own test suite — one of the few project tools that does.