Apache Arrow

Apache Arrow is a cross-language development platform for in-memory and larger-than-memory data. It defines a standardized columnar memory format optimized for analytic operations on modern hardware, and provides reference implementations of that format together with tools for IPC, RPC, file format readers/writers, and a query execution engine.

This repository hosts the monorepo containing Arrow's columnar format specification and several of its reference language implementations: C++, C bindings via GLib (with Ruby wrappers), Python (PyArrow, built on the C++ library), R (built on the C++ library), and MATLAB (built on the C++ library). Other implementations such as Java, Go, JavaScript, Rust, Julia, Swift, and .NET live in separate repositories under the apache organization.

What this codebase ships

• The Arrow columnar format — a memory layout for flat and nested data that any language implementation can read with zero copies. The binary FlatBuffers schemas live in format/ (format/Schema.fbs, format/Message.fbs, format/File.fbs, format/Tensor.fbs, format/SparseTensor.fbs).
• The C++ reference library in cpp/src/arrow/ — arrays, builders, types, IPC readers/writers, IO, filesystems, compute kernels, datasets, the Acero streaming execution engine, Flight RPC, and an integration with Substrait.
• Apache Parquet C++ in cpp/src/parquet/ — a full Parquet reader and writer with bloom filters, page indexes, encryption, and tight Arrow integration.
• Gandiva in cpp/src/gandiva/ — an LLVM-based JIT compiler that turns expression trees into native code for filtering and projection.
• PyArrow in python/pyarrow/ — Cython bindings to the C++ library that expose Arrow types, IO, compute, datasets, Parquet, Flight, and CUDA.
• The R arrow package in r/ — R6 classes wrapping the C++ library plus a dplyr backend that pushes computation down to Arrow's compute engine via Acero.
• C bindings via GLib in c_glib/ — GObject wrappers around the C++ library that enable bindings into many other languages.
• Red Arrow in ruby/ — Ruby gems built on top of the GLib bindings.
• MATLAB bindings in matlab/ — MATLAB classes that wrap the C++ library.

Quick links

• Architecture overview: see Architecture.
• Build and run: see Getting started.
• Project vocabulary: see Glossary.
• Numbers about the codebase: see By the numbers.
• Project history and major milestones: see Lore.
• C++ subsystems: see Systems.
• Language wrappers: see Implementations.
• Format specifications: see Format.

How the pieces fit together

graph TD
    subgraph Spec["Format specification (format/)"]
        Schema[Schema.fbs]
        Message[Message.fbs]
        File[File.fbs]
        FlightProto[Flight.proto / FlightSql.proto]
    end

    subgraph Cpp["C++ reference library (cpp/src/arrow)"]
        ArrowCore[arrays / types / record batches]
        Compute[compute kernels]
        Acero[acero streaming engine]
        Dataset[dataset framework]
        IO[io + filesystem]
        IPC[ipc readers/writers]
        FlightRPC[flight RPC]
    end

    subgraph Sister["Sister C++ projects under cpp/src"]
        Parquet[parquet]
        Gandiva[gandiva]
    end

    subgraph Wrappers["Language wrappers"]
        Python[python / pyarrow]
        R[r / arrow]
        CGLib[c_glib]
        Ruby[ruby / red-arrow]
        Matlab[matlab]
    end

    Spec --> ArrowCore
    Spec --> IPC
    Spec --> FlightRPC
    ArrowCore --> Compute
    Compute --> Acero
    Acero --> Dataset
    IO --> IPC
    IO --> Dataset
    Parquet --> Dataset
    ArrowCore --> Parquet
    ArrowCore --> Gandiva
    ArrowCore --> Python
    ArrowCore --> R
    ArrowCore --> CGLib
    CGLib --> Ruby
    ArrowCore --> Matlab

The format specification is the contract. The C++ library is the workhorse implementation. The sister projects (Parquet, Gandiva) and the language wrappers all build on the C++ library. Sister-language implementations not in this repo (Java, Go, Rust, JavaScript, etc.) re-implement the same format independently and verify cross-language compatibility through the integration test suite.

Where to start reading the code

• New to columnar data? Start with format/Schema.fbs and the C++ types in cpp/src/arrow/type.h.
• Curious about query execution? Read cpp/src/arrow/acero/exec_plan.h and the node implementations in cpp/src/arrow/acero/.
• Tracking down a Parquet bug? Start at cpp/src/parquet/file_reader.h for reads and cpp/src/parquet/file_writer.h for writes.
• Want to see the Python bindings? Most of the surface lives in python/pyarrow/lib.pyx, with subsystem-specific .pyx files alongside it (_compute.pyx, _dataset.pyx, _flight.pyx).