Arrays and types

Active contributors: Antoine Pitrou, Felipe Aramburu, Wes McKinney, Rossi Sun

The array and type system is the heart of the Arrow C++ library. It defines what data Arrow can represent and how that data lives in memory. See Columnar format for the cross-language spec; this page covers the C++ implementation.

Directory layout

cpp/src/arrow/
├── type.h, type.cc                      # DataType hierarchy
├── type_fwd.h                           # Forward declarations
├── type_traits.h, type_traits.cc        # Compile-time type → C++ mapping
├── visit_array_inline.h                 # Visitor dispatch for arrays
├── visit_data_inline.h                  # Visitor dispatch for ArrayData
├── visit_scalar_inline.h                # Visitor dispatch for scalars
├── visit_type_inline.h                  # Visitor dispatch for types
├── visitor.h, visitor.cc                # Runtime visitor
├── visitor_generate.h                   # Codegen helper
├── scalar.h, scalar.cc                  # Scalar values
├── datum.h, datum.cc                    # Compute input/output union
├── chunked_array.h, chunked_array.cc
├── record_batch.h, record_batch.cc
├── table.h, table.cc                    # Table = chunked columns
├── chunk_resolver.h, chunk_resolver.cc  # Logical index → (chunk, offset)
├── compare.h, compare.cc                # Array equality
├── extension_type.h, extension_type.cc
├── extension/                           # Built-in extension types
└── array/                               # Concrete arrays + builders
    ├── data.h, data.cc                  # ArrayData
    ├── array_base.{h,cc}                # Array, ArrayBuilder
    ├── array_primitive.{h,cc}           # Int*, Float*, Date*, Time*, Timestamp, Duration, Bool
    ├── array_binary.{h,cc}              # Binary, String, FixedSizeBinary, large variants, view variants
    ├── array_decimal.{h,cc}             # Decimal128, Decimal256
    ├── array_dict.{h,cc}                # DictionaryArray
    ├── array_nested.{h,cc}              # List, FixedSizeList, ListView, Map, Struct, Union (sparse + dense)
    ├── array_run_end.{h,cc}             # Run-end encoding
    ├── builder_*.{h,cc}                 # ArrayBuilder subclasses
    ├── concatenate.{h,cc}               # Concatenate same-type arrays
    ├── diff.{h,cc}                      # Compute array diffs
    ├── statistics.{h,cc}                # Per-array statistics (min, max, distinct count)
    ├── util.{h,cc}                      # Helpers
    └── validate.{h,cc}                  # Array validation

Type hierarchy

arrow::DataType (cpp/src/arrow/type.h) is the abstract base. Concrete types use the curiously recurring template pattern. Key subclasses:

• Primitive types — Int8Type, UInt8Type, ..., Int64Type, Float16Type, Float32Type, Float64Type, BooleanType, Date32Type, Date64Type, Time32Type, Time64Type, TimestampType, DurationType, IntervalType (with month, day-time, month-day-nano variants).
• Binary types — BinaryType, LargeBinaryType, FixedSizeBinaryType, Utf8Type, LargeUtf8Type, BinaryViewType, Utf8ViewType.
• Decimal types — Decimal32Type, Decimal64Type, Decimal128Type, Decimal256Type.
• Nested types — ListType, LargeListType, FixedSizeListType, ListViewType, LargeListViewType, MapType, StructType, SparseUnionType, DenseUnionType.
• Special types — NullType, DictionaryType, RunEndEncodedType, ExtensionType.

Each type carries a Type::type enum value (Type::INT32, Type::LIST, ...) used for runtime dispatch when compile-time dispatch isn't possible.

Builders

Every array type has a builder. The builder API is uniform:

Int32Builder builder;
RETURN_NOT_OK(builder.Reserve(n));
for (int32_t v : data) {
  RETURN_NOT_OK(builder.Append(v));
}
ARROW_ASSIGN_OR_RAISE(auto arr, builder.Finish());

Builders manage their own buffers and call MemoryPool::Reallocate when growing. Reserve(n) reserves capacity; Append(v) adds a value (potentially growing); AppendNull adds a null; AppendNulls(n) adds many; AppendValues(span) appends a contiguous range.

The dictionary builder (DictionaryBuilder<T>) adds a hash table for indexing into the dictionary array. The struct builder is recursive — it owns child builders for each field.

Slicing

Array::Slice(offset, length) produces a new Array that shares buffers with the original. The implementation just constructs a new ArrayData with the same buffer pointers and a different offset and length. This is O(1).

ChunkedArray::Slice slices across chunks. RecordBatch::Slice slices every column. Table::Slice does the same for chunked arrays.

Validation

arrow::ValidateArray (cpp/src/arrow/array/validate.h) walks an array and confirms the buffer layout is internally consistent (offsets are monotonic, validity bitmap dimensions match, child arrays have the right lengths, dictionary types make sense). Two levels:

• ValidateArray — fast checks that don't require touching every value.
• ValidateArrayFull — exhaustive checks (e.g. that every UTF-8 string is valid UTF-8, every offset in a list is in range).

Visitors

The visitor macros in cpp/src/arrow/visit_*.h produce specialized code for every type at compile time. Pattern:

struct MyVisitor {
  Status Visit(const Int32Array& arr) { ... }
  Status Visit(const StringArray& arr) { ... }
  // ... one Visit per array type
};

MyVisitor v;
RETURN_NOT_OK(VisitArrayInline(*arr, &v));

VisitArrayInline is a constexpr-dispatched template that picks the right Visit overload from the array's type_id. The recent commit GH-49835 ("constexpr dynamic dispatch with static dispatch when possible") tightened this further. The compiler unrolls the dispatch to a single switch, removing virtual function overhead.

Tabular containers

• RecordBatch is a schema plus a vector of arrays of equal length. The implementation in cpp/src/arrow/record_batch.cc uses ArrayData directly to keep slicing zero-copy.
• ChunkedArray is a vector of Array chunks of the same type. Useful when streaming data — append a chunk and you have a bigger column without copying.
• Table is a schema plus chunked arrays per column. Tables can have unequal chunk boundaries across columns, but every column has the same total length.

Statistics

cpp/src/arrow/array/statistics.h defines arrow::ArrayStatistics — a struct that carries optional min, max, distinct count, null count, and average byte width. Statistics are emitted by Parquet, the dataset scanner, and various compute kernels, and consumed by query planning to skip data ranges.

How types are exposed across the API

The same type hierarchy is exposed to:

• PyArrow via Cython in python/pyarrow/types.pxi (~163 KB).
• R via R6 classes in r/R/type.R and r/src/datatype.cpp.
• C-GLib via GObject classes in c_glib/arrow-glib/data-type.cpp and friends.
• The C data interface via format strings (see C data interface).
• IPC via FlatBuffers messages (see IPC).

The single source of truth is cpp/src/arrow/type.h. When a new type is added (e.g. Decimal32Type recently), the schema FlatBuffer is updated, the C++ class is added, the C data interface format string is documented, and each language wrapper picks it up via its bridge code.

Entry points for modification

• Adding a new primitive or nested type: extend format/Schema.fbs, add a class in cpp/src/arrow/type.h, register in cpp/src/arrow/type_fwd.h, add Visit overloads in cpp/src/arrow/visit_*.h, register in arrow/c/bridge.cc, and add Cython/R/C-GLib bindings.
• Optimizing an array operation: look in cpp/src/arrow/array/*.cc and the matching test files. Most arrays have a *_test.cc and a *_benchmark.cc (array_test.cc, array_benchmark.cc).
• Adding a new builder: subclass arrow::ArrayBuilder and implement Append, AppendNull, Reserve, Resize, and Finish.