Format

Apache Arrow is, before anything else, a format specification. The actual binary contracts live in format/ at the repo root and are language-agnostic. Every reference implementation in this repo reads and writes data conforming to these specs; sister-language implementations elsewhere (Java, Go, Rust, JavaScript, Swift, Julia, .NET) re-implement the same specs.

What's in format/

File	Purpose
format/Schema.fbs	Logical and physical types, field metadata, schema
format/Message.fbs	The IPC envelope: schema messages, record batches, dictionary batches, tensor messages
format/File.fbs	Random-access Arrow file format (footer, blocks)
format/Tensor.fbs	Dense tensor wire format
format/SparseTensor.fbs	Sparse tensor wire formats (COO, CSR, CSF)
format/Flight.proto	Flight RPC service over gRPC
format/FlightSql.proto	Flight SQL extension to Flight
format/substrait/	Sub-spec: Arrow's Substrait extensions
format/README.rst	Pointer to the docs site for the format reference

The FlatBuffers schemas are compiled into C++ headers that live under cpp/src/generated/:

format/Schema.fbs        →  cpp/src/generated/Schema_generated.h
format/Message.fbs       →  cpp/src/generated/Message_generated.h
format/File.fbs          →  cpp/src/generated/File_generated.h
format/Tensor.fbs        →  cpp/src/generated/Tensor_generated.h
format/SparseTensor.fbs  →  cpp/src/generated/SparseTensor_generated.h

Re-generating these headers is part of the development workflow whenever you touch a .fbs (see Development workflow).

What this section covers

• Columnar format — the in-memory layout for arrays of every Arrow type.
• C data interface — the C ABI for in-process zero-copy exchange.
• IPC — the streaming and file formats Arrow uses on the wire and on disk.
• Flight — the gRPC-based RPC framework for high-throughput record-batch transport.

Why "format first"

Arrow's design priority is interoperability between independent implementations. The format spec is therefore expected to be read directly by humans and machines:

• The FlatBuffers schemas are the source of truth — a Java client and a C++ server agree because they both decode the same bytes against the same schema.
• The columnar layout (validity bitmap, offsets buffer, values buffer) is fixed: a Rust array can be wrapped around C++ buffers without copying.
• The C data interface guarantees that two languages running in the same process can exchange data with zero copies, even when neither of them links against the other's runtime library.
• Flight's .proto files are the source of truth for the RPC service; gRPC stubs in C++, Java, Python, Go, etc., are generated from them.

Anyone designing an Arrow consumer needs to read these specs at some point. The Arrow website (https://arrow.apache.org/docs/dev/format/Columnar.html) builds human-friendly versions of the same content from the docs/source/format/ RST files.