Open-Source Wikis

/

Apache Arrow

/

Systems

apache/arrow

Systems

The C++ library is structured as a collection of subsystems under cpp/src/arrow/ plus two sister projects (cpp/src/parquet/ and cpp/src/gandiva/). This section provides a deep dive into each subsystem.

Pages

  • Arrays and typescpp/src/arrow/array/, cpp/src/arrow/type.h, scalars, tables, record batches.
  • I/O and filesystemcpp/src/arrow/io/, cpp/src/arrow/filesystem/, local + S3 + GCS + Azure + HDFS.
  • Compute kernelscpp/src/arrow/compute/, scalar/vector/aggregate kernels and dispatch.
  • Acerocpp/src/arrow/acero/, the streaming execution engine.
  • Datasetcpp/src/arrow/dataset/, partitioned multi-format datasets.
  • Parquetcpp/src/parquet/, the Parquet C++ implementation.
  • Gandivacpp/src/gandiva/, LLVM-based expression compilation.
  • CSV and JSONcpp/src/arrow/csv/, cpp/src/arrow/json/.
  • Substraitcpp/src/arrow/engine/substrait/, plan-format adapter.
  • Utilcpp/src/arrow/util/, the kitchen sink (bit ops, futures, threading, compression, telemetry, ...).

Subsystems by file count (cpp/src/arrow only)

A rough sense of weight, using subdirectory file counts inside cpp/src/arrow/:

Subsystem Files Highlights
compute/ (incl. kernels/ and row/) ~120 Function registry, ~150 kernels
acero/ ~80 14 ExecNode types, hash-join engines, TPC-H generator
flight/ (incl. sql/, transport/) ~70 Client, server, gRPC + UCX transports, Flight SQL
dataset/ ~50 Format adapters, scanner, partitioning
array/ ~50 Array hierarchy, builders, validation
util/ ~250 Bit ops, futures, threading, compression, telemetry, RLE, hashing
io/ + filesystem/ ~60 Local + cloud filesystems, compressed/buffered streams
ipc/ ~30 IPC reader/writer + Feather
csv/ + json/ ~50 CSV and JSON readers and writers

Subsystems outside arrow/

Subsystem Path Focus
Parquet cpp/src/parquet/ Parquet reader/writer, encryption, page index, bloom filters
Gandiva cpp/src/gandiva/ LLVM-based expression compiler
ORC adapter cpp/src/arrow/adapters/orc/ Apache ORC integration
TensorFlow adapter cpp/src/arrow/adapters/tensorflow/ Legacy TensorFlow IO
GPU support cpp/src/arrow/gpu/ CUDA buffers and contexts
Tensor I/O cpp/src/arrow/tensor/ Sparse + dense tensors
Telemetry cpp/src/arrow/telemetry/ OpenTelemetry plumbing

Integration

The cpp/src/arrow/integration/ directory hosts the cross-language integration test driver. It implements:

  • integration_command.cc — the CLI used by dev/archery/archery/integration/.
  • JSON ↔ IPC encoding so any implementation can dump or read data in JSON form for round-trip checks.
  • A test harness that invokes the C++ Arrow library's IPC reader and writer against fixture files generated by other implementations.

Cross-cutting reads

A few pieces span multiple subsystems and are worth highlighting separately:

  • Async machinery. cpp/src/arrow/util/future.h, async_generator.h, thread_pool.h, task_group.h. This is the runtime that backs Acero, datasets, and Flight.
  • Validity bitmap utilities. cpp/src/arrow/util/bit_util.h, bitmap.h, bitmap_ops.cc. Touched by every array operation that supports nulls.
  • Compression codecs. cpp/src/arrow/util/compression*.cc. Used by IPC, Parquet, ORC, and the dataset framework.
  • Hashing. cpp/src/arrow/compute/key_hash_internal.cc (with AVX2 variant) and arrow/util/hashing.h. Used by hash aggregates, hash joins, and dictionaries.
  • CPU dispatch. cpp/src/arrow/util/cpu_info.cc, dispatch_internal.h. Picks the best SIMD code path at runtime (SSE4.2, AVX2, AVX-512, NEON, SVE).

Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.

Systems – Apache Arrow wiki | Factory