C data interface

Active contributors: Antoine Pitrou, Wes McKinney, Joris Van den Bossche

The C data interface is a stable C ABI for moving Arrow data between language runtimes that share the same address space. It is defined in cpp/src/arrow/c/abi.h and implemented in cpp/src/arrow/c/bridge.cc and bridge.h. PyArrow exposes it through python/pyarrow/cffi.py and the corresponding _C Cython wrappers.

What it is for

In a single process you might want to:

• Pass a Polars DataFrame to PyArrow without copying.
• Hand a Pandas-via-PyArrow array to a NumPy consumer.
• Move a record batch from a custom C extension into Arrow Java via JNI.
• Stream batches from a Rust producer to a Python consumer in a notebook.

The C data interface lets all of those happen with zero copies and without either side linking against the other's runtime. Both sides agree on a set of plain C structs.

The structs

struct ArrowSchema {
  const char* format;
  const char* name;
  const char* metadata;
  int64_t flags;
  int64_t n_children;
  struct ArrowSchema** children;
  struct ArrowSchema* dictionary;
  void (*release)(struct ArrowSchema*);
  void* private_data;
};

struct ArrowArray {
  int64_t length;
  int64_t null_count;
  int64_t offset;
  int64_t n_buffers;
  int64_t n_children;
  const void** buffers;
  struct ArrowArray** children;
  struct ArrowArray* dictionary;
  void (*release)(struct ArrowArray*);
  void* private_data;
};

struct ArrowArrayStream {
  int (*get_schema)(struct ArrowArrayStream*, struct ArrowSchema*);
  int (*get_next)(struct ArrowArrayStream*, struct ArrowArray*);
  const char* (*get_last_error)(struct ArrowArrayStream*);
  void (*release)(struct ArrowArrayStream*);
  void* private_data;
};

struct ArrowDeviceArray {
  struct ArrowArray array;
  int64_t device_id;
  ArrowDeviceType device_type;
  int64_t* sync_event;
  int64_t reserved[3];
};

ArrowSchema describes the type. ArrowArray describes the physical buffers. ArrowArrayStream is a producer that emits a stream of arrays sharing one schema. ArrowDeviceArray extends ArrowArray with device placement (CPU, CUDA, ROCm, OpenCL).

The format field uses a small string-based grammar described in cpp/src/arrow/c/abi.h. Examples: i for int32, +s for struct, +l for list, c for int8, tdD for date32, +w:K for fixed-size list of K, +m for map.

Lifecycle

Both ArrowSchema and ArrowArray carry a release callback. The producer fills in the struct and sets release. The consumer is responsible for calling release exactly once; it must not call it more than once and must not free the struct itself (the private_data and release callback handle that). After release, the struct's release is set to NULL to mark it consumed.

This convention lets the producer choose whatever ownership model is convenient (shared_ptr, refcount, custom arena) without surfacing it to the consumer.

Implementation in this repo

cpp/src/arrow/c/bridge.cc (~96 KB — the largest single source file in cpp/src/arrow/c/) implements two directions:

• Export. arrow::ExportType, ExportArray, ExportRecordBatch, ExportRecordBatchReader, ExportArrayStream, and the device-aware variants populate the C structs from C++ Arrow objects.
• Import. arrow::ImportType, ImportArray, ImportRecordBatch, ImportRecordBatchReader consume the C structs and produce C++ Arrow objects.

Each direction has a corresponding Async variant for use with arrow::Future. The bridge handles dictionary arrays specially because dictionaries are shared across batches in a stream.

cpp/src/arrow/c/dlpack.cc adds a sibling DLPack adapter so that arrays can also be exchanged with PyTorch/JAX/TensorFlow's tensor format.

The test suite in cpp/src/arrow/c/bridge_test.cc is over 5,400 lines — round-tripping is exhaustively covered.

PyArrow surface

PyArrow exposes the C data interface through:

• pyarrow.Array._import_from_c(schema_ptr, array_ptr) and Array._export_to_c(...) (python/pyarrow/array.pxi).
• pyarrow.Schema._import_from_c, _export_to_c.
• pyarrow.RecordBatchReader._import_from_c.
• pyarrow.cffi — a CFFI module that exposes the C structs to other CFFI consumers.

Ecosystem use

• DuckDB can ingest a PyArrow table by passing the C struct pointer.
• Polars uses the C interface to interoperate with PyArrow.
• DataFusion (Rust) and arrow-rs support the interface natively.
• Java's Arrow JDBC adapter uses the JNI bridge built on this interface.
• R arrow uses it (r/src/bridge.cpp) to pass arrays between R and PyArrow over reticulate.

Because the interface is a C ABI, no language runtime depends on another's symbol table. Two programs only need to agree on the format string vocabulary and the buffer layout.