apache/arrow
Systems
The C++ library is structured as a collection of subsystems under cpp/src/arrow/ plus two sister projects (cpp/src/parquet/ and cpp/src/gandiva/). This section provides a deep dive into each subsystem.
Pages
- Arrays and types —
cpp/src/arrow/array/,cpp/src/arrow/type.h, scalars, tables, record batches. - I/O and filesystem —
cpp/src/arrow/io/,cpp/src/arrow/filesystem/, local + S3 + GCS + Azure + HDFS. - Compute kernels —
cpp/src/arrow/compute/, scalar/vector/aggregate kernels and dispatch. - Acero —
cpp/src/arrow/acero/, the streaming execution engine. - Dataset —
cpp/src/arrow/dataset/, partitioned multi-format datasets. - Parquet —
cpp/src/parquet/, the Parquet C++ implementation. - Gandiva —
cpp/src/gandiva/, LLVM-based expression compilation. - CSV and JSON —
cpp/src/arrow/csv/,cpp/src/arrow/json/. - Substrait —
cpp/src/arrow/engine/substrait/, plan-format adapter. - Util —
cpp/src/arrow/util/, the kitchen sink (bit ops, futures, threading, compression, telemetry, ...).
Subsystems by file count (cpp/src/arrow only)
A rough sense of weight, using subdirectory file counts inside cpp/src/arrow/:
| Subsystem | Files | Highlights |
|---|---|---|
compute/ (incl. kernels/ and row/) |
~120 | Function registry, ~150 kernels |
acero/ |
~80 | 14 ExecNode types, hash-join engines, TPC-H generator |
flight/ (incl. sql/, transport/) |
~70 | Client, server, gRPC + UCX transports, Flight SQL |
dataset/ |
~50 | Format adapters, scanner, partitioning |
array/ |
~50 | Array hierarchy, builders, validation |
util/ |
~250 | Bit ops, futures, threading, compression, telemetry, RLE, hashing |
io/ + filesystem/ |
~60 | Local + cloud filesystems, compressed/buffered streams |
ipc/ |
~30 | IPC reader/writer + Feather |
csv/ + json/ |
~50 | CSV and JSON readers and writers |
Subsystems outside arrow/
| Subsystem | Path | Focus |
|---|---|---|
| Parquet | cpp/src/parquet/ |
Parquet reader/writer, encryption, page index, bloom filters |
| Gandiva | cpp/src/gandiva/ |
LLVM-based expression compiler |
| ORC adapter | cpp/src/arrow/adapters/orc/ |
Apache ORC integration |
| TensorFlow adapter | cpp/src/arrow/adapters/tensorflow/ |
Legacy TensorFlow IO |
| GPU support | cpp/src/arrow/gpu/ |
CUDA buffers and contexts |
| Tensor I/O | cpp/src/arrow/tensor/ |
Sparse + dense tensors |
| Telemetry | cpp/src/arrow/telemetry/ |
OpenTelemetry plumbing |
Integration
The cpp/src/arrow/integration/ directory hosts the cross-language integration test driver. It implements:
integration_command.cc— the CLI used bydev/archery/archery/integration/.- JSON ↔ IPC encoding so any implementation can dump or read data in JSON form for round-trip checks.
- A test harness that invokes the C++ Arrow library's IPC reader and writer against fixture files generated by other implementations.
Cross-cutting reads
A few pieces span multiple subsystems and are worth highlighting separately:
- Async machinery.
cpp/src/arrow/util/future.h,async_generator.h,thread_pool.h,task_group.h. This is the runtime that backs Acero, datasets, and Flight. - Validity bitmap utilities.
cpp/src/arrow/util/bit_util.h,bitmap.h,bitmap_ops.cc. Touched by every array operation that supports nulls. - Compression codecs.
cpp/src/arrow/util/compression*.cc. Used by IPC, Parquet, ORC, and the dataset framework. - Hashing.
cpp/src/arrow/compute/key_hash_internal.cc(with AVX2 variant) andarrow/util/hashing.h. Used by hash aggregates, hash joins, and dictionaries. - CPU dispatch.
cpp/src/arrow/util/cpu_info.cc,dispatch_internal.h. Picks the best SIMD code path at runtime (SSE4.2, AVX2, AVX-512, NEON, SVE).
Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.