IPC

Active contributors: Antoine Pitrou, Wes McKinney, Felipe Aramburu

Arrow IPC is the on-the-wire and on-disk serialization for Arrow record batches. It is defined by the FlatBuffers schemas in format/Schema.fbs, format/Message.fbs, and format/File.fbs, and implemented in cpp/src/arrow/ipc/.

Two IPC formats

Format	Use	Layout
Stream	Pipes, sockets, fds, network streams. Sequential read only.	Schema message, then 0..N dictionary or record-batch messages, then EOS.
File	Local files, S3 objects, anywhere with random access.	ARROW1\0\0 magic + stream + footer + footer length + magic. The footer indexes record batches and dictionaries by file offset for random access.

Both formats share the same record-batch encoding. The file format is a superset.

The IPC envelope

Each message uses an "encapsulated message" framing:

[continuation indicator: int32 = 0xFFFFFFFF]
[metadata length: int32]
[metadata: FlatBuffer Message of length metadata_length]
[padding to 8-byte alignment]
[body: zero or more buffers, each padded to 8 bytes]

The metadata is a Message FlatBuffer (from format/Message.fbs) carrying:

• version — Arrow format version (currently V5).
• header — a union over Schema, RecordBatch, DictionaryBatch, Tensor, SparseTensor.
• bodyLength — number of bytes following the metadata.
• custom_metadata — optional key/value metadata.

For a RecordBatch, the metadata describes:

• length — number of rows.
• nodes — for each field (in pre-order traversal), length + null_count.
• buffers — for each buffer (in pre-order), offset + length into the body.
• compression — optional compression scheme (LZ4_FRAME, ZSTD).

The body contains the actual buffer bytes. Buffers are written contiguously in the order described by buffers, each padded to 8-byte boundaries.

C++ implementation

The reader and writer live in cpp/src/arrow/ipc/:

File	Purpose
cpp/src/arrow/ipc/message.h / .cc	The Message type and its FlatBuffer parsing.
cpp/src/arrow/ipc/reader.h / .cc	RecordBatchStreamReader, RecordBatchFileReader. The reader reads metadata, decodes field nodes, and slices the body buffers into Arrow ArrayData.
cpp/src/arrow/ipc/writer.h / .cc	RecordBatchStreamWriter, RecordBatchFileWriter. The writer flattens record batches into the encapsulated framing.
cpp/src/arrow/ipc/feather.h / .cc	The legacy Feather V1 format and the V2 Feather entry point (Feather V2 is the Arrow file format).
cpp/src/arrow/ipc/options.h	IpcReadOptions, IpcWriteOptions (compression, max recursion depth, validation level, custom field metadata).
cpp/src/arrow/ipc/metadata_internal.cc	The bridge between FlatBuffers and Arrow's type system.

Dictionary handling

Dictionary-encoded columns require special framing. Dictionaries are sent in DictionaryBatch messages that arrive before any record batch that references them. Dictionaries can be replaced (isDelta = false) or extended (isDelta = true). The reader maintains a DictionaryMemo (cpp/src/arrow/ipc/dictionary_memo.h) that tracks the active dictionary for each field.

In the file format, the footer separately tracks dictionary blocks by offset to support random access without re-reading the stream.

Compression

When IpcWriteOptions::compression is set, every buffer body is compressed individually. The supported codecs are LZ4_FRAME (default) and ZSTD. Compressed buffers carry an 8-byte uncompressed length prefix followed by the compressed bytes (or -1 indicating "this buffer was not compressed because compression made it larger"). The reader decompresses lazily.

Streaming over a socket / network

Arrow IPC is the wire format used by Flight: each DoGet response yields a stream of IPC messages. Flight's transport layer (cpp/src/arrow/flight/transport.cc) wraps the stream in gRPC frames.

Fuzz harnesses

cpp/src/arrow/ipc/file_fuzz.cc and stream_fuzz.cc are LibFuzzer entry points. They feed arbitrary bytes into the IPC reader and check that no UB occurs. cpp/src/arrow/ipc/generate_fuzz_corpus.cc produces a starting corpus that the fuzzers can mutate.

The recent fix GH-49896: [C++] Reject short buffer reads in IPC reader (Apr 2026) is an example of a bug uncovered by the fuzz infrastructure.

Cross-implementation tests

Integration tests under cpp/src/arrow/integration/ and dev/archery/archery/integration/ verify that IPC files written by one Arrow implementation can be read by every other implementation. Each implementation produces JSON dumps of test schemas and reads back binary files written by every other implementation, asserting equality. This is the contract the format spec promises.