Python (PyArrow)

Active contributors: Hyukjin Kwon, Raúl Cumplido, Rok Mihevc, Antoine Pitrou, Joris Van den Bossche

PyArrow is a Cython-based binding to the C++ library. It exposes Arrow's types, IO, compute, datasets, Parquet, Flight, CUDA, and Substrait integration to Python in idiomatic form. The package builds on top of libarrow, libparquet, libarrow-dataset, libarrow-flight, etc.

Purpose

Make Arrow available to the Python data ecosystem. PyArrow is the canonical Python implementation of Arrow, the engine behind pandas.read_parquet and pandas.DataFrame.to_arrow, and the typical conduit between Python and Arrow-aware engines like DuckDB, Polars, and DataFusion.

Layout

python/
├── pyproject.toml               # Build system config
├── setup.cfg                    # pytest, coverage config
├── CMakeLists.txt               # CMake glue invoked by scikit-build
├── _build_backend/              # Custom PEP 517 backend (uses CMake)
├── pyarrow/
│   ├── __init__.py              # Public API surface
│   ├── lib.pyx                  # Master Cython module: imports all .pxi files
│   ├── lib.pxd                  # C++-side declarations
│   ├── includes/                # Cython .pxd files mirroring C++ headers
│   ├── *.pxi                    # Per-domain Cython "include" files (array, scalar, ipc, io, table, types, ...)
│   ├── _compute.pyx             # Compute bindings
│   ├── _dataset.pyx             # Dataset bindings (~165 KB — the largest .pyx file)
│   ├── _flight.pyx              # Flight + Flight SQL bindings
│   ├── _parquet.pyx             # Parquet bindings
│   ├── _csv.pyx, _json.pyx      # CSV / JSON readers
│   ├── _fs.pyx, _s3fs.pyx,      # Filesystems
│   │  _gcsfs.pyx, _azurefs.pyx, _hdfs.pyx
│   ├── _cuda.pyx                # CUDA bindings
│   ├── _orc.pyx                 # ORC adapter
│   ├── _substrait.pyx           # Substrait integration
│   ├── _acero.pyx               # Acero engine bindings
│   ├── compute.py, dataset.py,  # Pythonic API on top of the .pyx modules
│   │  ipc.py, fs.py, ...
│   ├── parquet/                 # Parquet Pythonic API (`pyarrow.parquet.read_table`, ...)
│   ├── interchange/             # Dataframe interchange protocol
│   ├── tests/                   # pytest suite
│   ├── src/                     # Small C++ helpers compiled with the extension
│   └── vendored/                # Vendored Python helpers
├── pyarrow-stubs/               # PEP 561 stub package
├── benchmarks/                  # ASV benchmarks
├── examples/                    # Standalone examples
├── scripts/                     # Build / release helpers
└── requirements-*.txt           # Build, test, wheel dep manifests

How it builds

PyArrow uses scikit-build through a custom PEP 517 backend in python/_build_backend/. The backend:

1. Reads ARROW_HOME (or arrow.pc via pkg-config) to locate the C++ library.
2. Configures python/CMakeLists.txt, which invokes Cython to translate .pyx files into C++ source.
3. Links the compiled extension modules against libarrow.so and the optional component libraries.
4. Stages the .so files into python/pyarrow/ so Python can import pyarrow.

The build accepts PYARROW_WITH_PARQUET, PYARROW_WITH_DATASET, PYARROW_WITH_FLIGHT, PYARROW_WITH_GANDIVA, PYARROW_WITH_S3, etc., as environment variables to mirror the C++ component toggles.

Cython surface

python/pyarrow/lib.pyx is the master module — it includes every .pxi partial. The .pxi files are organized by domain:

• array.pxi — Array, all type-specific subclasses, builder helpers (~162 KB; the largest single Python file in the project)
• scalar.pxi — Scalar and subclasses
• table.pxi — Table, RecordBatch, ChunkedArray (~206 KB)
• types.pxi — the type system mirror (~163 KB)
• tensor.pxi, device.pxi, memory.pxi, error.pxi, io.pxi, ipc.pxi, builder.pxi, config.pxi
• pandas-shim.pxi — Pandas integration helpers

Domains that are large enough to compile separately have their own top-level .pyx files (_compute.pyx, _dataset.pyx, _flight.pyx, _parquet.pyx, etc.).

Pythonic Layer

On top of the Cython, a thin Python layer in pyarrow/*.py exposes idiomatic helpers:

• pyarrow.compute (compute.py) — function discovery, decorator helpers.
• pyarrow.dataset (dataset.py) — friendly dataset() factory, partitioning shortcuts.
• pyarrow.parquet (parquet/core.py) — read_table, write_table, ParquetDataset.
• pyarrow.ipc (ipc.py) — convenience wrappers over RecordBatchStreamWriter/Reader.
• pyarrow.fs (fs.py) — FileSystem factories.
• pyarrow.feather, pyarrow.json, pyarrow.csv, pyarrow.flight, pyarrow.cuda, pyarrow.acero, pyarrow.substrait, pyarrow.gandiva.

Pandas integration

python/pyarrow/pandas_compat.py (45 KB) implements bidirectional Pandas conversion. It handles:

• Dtype mapping (Pandas extension types ↔ Arrow types).
• The pandas metadata that PyArrow attaches to schemas to round-trip Pandas indexes and dtypes.
• Categorical and timezone-aware datetime handling.
• The ArrowExtensionArray protocol that Pandas 2.x uses.

Dataframe interchange

python/pyarrow/interchange/ implements the Dataframe Interchange Protocol, allowing PyArrow to exchange data with NumPy, Pandas, Polars, and other ecosystem libraries via a standard Python protocol.

Testing

Tests live under python/pyarrow/tests/. The conventions and run instructions are covered in Testing. Notable tests:

• tests/test_compute.py — every compute function.
• tests/test_dataset.py — dataset reads with all formats.
• tests/parquet/ — Parquet-specific tests.
• tests/test_flight.py — Flight client/server.
• tests/test_pandas.py — Pandas conversion.
• tests/test_extension_type.py — extension type round-tripping.

Build distributions

Wheels are built via Crossbow tasks under dev/tasks/python-wheels/. Conda packages are built via dev/tasks/conda-recipes/. The release pipeline at dev/release/post-09-python.sh uploads to PyPI.