Columnar format

Active contributors: Antoine Pitrou, Felipe Aramburu, Wes McKinney

The columnar format is Arrow's in-memory layout for arrays. It is defined by format/Schema.fbs and implemented in cpp/src/arrow/array/ and cpp/src/arrow/type.h. This page summarizes the layout and explains how the C++ types reflect the spec.

The shape of an array

Every Arrow array consists of:

1. A type (a DataType such as int32, utf8, list<int32>, struct<x: int, y: int>, ...).
2. A length (number of logical values).
3. An offset into the underlying buffers (for zero-copy slicing).
4. A null count.
5. A list of buffers (validity bitmap, offsets, values, etc.).
6. Zero or more child arrays (for nested types).
7. Optional dictionary array (for dictionary-encoded types).

The base C++ class is arrow::Array (cpp/src/arrow/array/array_base.h). The physical representation is arrow::ArrayData (cpp/src/arrow/array/data.h):

struct ArrayData {
  std::shared_ptr<DataType> type;
  int64_t length;
  int64_t null_count;
  int64_t offset;
  std::vector<std::shared_ptr<Buffer>> buffers;
  std::vector<std::shared_ptr<ArrayData>> child_data;
  std::shared_ptr<ArrayData> dictionary;
};

The split between Array (the typed view) and ArrayData (the physical representation) lets two Arrays of different views (e.g. a slice and the full array) share the same buffers.

Types and their physical layouts

The type hierarchy is defined in cpp/src/arrow/type.h. Each type has a fixed buffer layout described in the columnar spec:

Type	Buffers	Children
Primitive (int8 … float64, decimal, date, time, timestamp)	validity + values	—
Bool	validity + values (bit-packed)	—
Variable-length binary (utf8, binary)	validity + offsets + values	—
Large variable-length binary (large_utf8, large_binary)	validity + 64-bit offsets + values	—
StringView / BinaryView	validity + views + 0..N variadic data buffers	—
FixedSizeBinary	validity + values (fixed stride)	—
List	validity + offsets	one child array
LargeList	validity + 64-bit offsets	one child array
FixedSizeList	validity	one child array
ListView / LargeListView	validity + offsets + sizes	one child array
Struct	validity	N child arrays (one per field)
Sparse Union	type ids	N child arrays
Dense Union	type ids + offsets	N child arrays
Map	validity + offsets	one child array (struct of key, value)
Dictionary	indices buffers	indices array; dictionary held on the parent ArrayData
Run-end encoded	—	run_ends child + values child
Null	—	—

Concrete C++ classes for each type live in cpp/src/arrow/array/array_primitive.h, array_binary.h, array_dict.h, array_nested.h, array_run_end.h, array_decimal.h. Each has a corresponding builder in the same directory.

Validity bitmaps

Null/non-null is encoded by a bit per element: 1 means valid, 0 means null. The bitmap is the first buffer for any nullable type. Helpers live in cpp/src/arrow/util/bit_util.h, bitmap.h, and bitmap_ops.cc. Optimizations like OptionalBitmapAnd (recently added in GH-45819) operate on raw bitmaps and short-circuit when one side is fully valid.

Offsets

Variable-length arrays (utf8, binary, list, map) carry an offsets buffer of length N+1: position i is the start of the i-th element, and offsets[length] == data.size(). List sizes are offsets[i+1] - offsets[i].

The "view" types (utf8_view, binary_view, list_view) replace offsets with a per-element view containing length, prefix bytes, and a buffer index — a layout that allows out-of-order data.

Slicing

Array::Slice(offset, length) is O(1). It produces a new Array whose ArrayData shares buffers with the original but advances the logical offset. The validity bitmap and offsets buffer don't move; readers compute the right starting bit/index from the offset.

Dictionary encoding

A DictionaryArray carries indices (e.g. int32) and a dictionary array. The cpp/src/arrow/array/array_dict.h API exposes the indices and the dictionary; the bridge layer in cpp/src/arrow/c/bridge.cc handles dictionary lifecycle when crossing the C data interface.

Run-end encoding

Run-end encoded (REE) arrays compress runs of equal values. The layout is two children: a run_ends array (cumulative run lengths) and a values array. Operations on REE arrays are implemented in cpp/src/arrow/array/array_run_end.cc and cpp/src/arrow/util/ree_util.h.

Extension types

User-defined extension types attach a serialized payload to a storage type. The base class is arrow::ExtensionType (cpp/src/arrow/extension_type.h), and built-in extensions (UUID, JSON, OpaqueType, Bool8, Fixed-shape tensor, Variable-shape tensor) live in cpp/src/arrow/extension/.

How this maps to other layers

• IPC (cpp/src/arrow/ipc/) serializes record batches: a schema message, then record batch messages with field nodes describing each array's null count and length plus buffer descriptors with offsets/lengths.
• C data interface (cpp/src/arrow/c/) exposes the same logical structure (ArrowSchema, ArrowArray) over a raw C ABI.
• Compute kernels assume the standard layout. Most kernels accept ArraySpan (cpp/src/arrow/array/data.h), a non-owning view into an ArrayData.

Reading cpp/src/arrow/array/data.h and cpp/src/arrow/type.h is the fastest way to internalize the columnar format from a developer's point of view.