Tooling

The Arrow project ships a kit of in-repo developer tools. This page lists the most useful ones and where to find them.

Archery — the Python developer CLI

dev/archery/ contains a Python CLI named archery. It is the umbrella entry point for project tooling and is installed via:

pip install -e dev/archery[all]

Sub-commands:

• archery lint — runs every linter the project uses (clang-format, cpplint, IWYU, cmake-format, ruff, lintr, rubocop, hadolint).
• archery integration — drives the cross-language integration test matrix (Java, Go, Rust, JS, etc.).
• archery release — generates release notes, manages release votes, prepares source archives.
• archery benchmark — wraps Google Benchmark runs with comparison and history.
• archery crossbow — submit, monitor, and report on Crossbow tasks.
• archery docker — orchestrates the compose.yaml services without manually invoking docker compose.

The implementation lives in dev/archery/archery/. Each sub-command is its own subpackage (archery.lint, archery.integration, ...). archery.utils has shared helpers.

Crossbow — CI fanout

Crossbow is the system that fans builds out to many platforms. Configuration:

• dev/tasks/tasks.yml — task list. Each task has a template (e.g. wheel-manylinux-2014, r-binary-packages, nightly-tests) and parameters.
• dev/tasks/<template>/ — Jinja templates that produce GitHub Actions workflow files for each task.
• dev/tasks/conda-recipes/ — conda-forge recipe sources used by conda-related tasks.
• dev/tasks/python-wheels/ — Python wheel build helpers.
• dev/tasks/r/ — R-package build scripts for various platforms.

Crossbow renders the templates into arrow.git/branches/<task> branches in a separate "crossbow" repo, which then triggers GitHub Actions. Reports flow back via dev/tasks/conda-recipes/.

Build-support scripts

cpp/build-support/ contains C++-specific helpers:

• run-clang-format.py — formats every C++ source file the project tracks.
• run-clang-tidy.py — runs clang-tidy with the project's .clang-tidy config.
• lint_cpp_cli.py — Arrow-specific lint rules beyond cpplint.
• iwyu/ — Include What You Use mappings.
• release/ — release-specific helpers (RAT license check, signature verification).

CMake modules

cpp/cmake_modules/ defines the CMake macros and find modules Arrow uses:

• BuildUtils.cmake — the add_arrow_lib, add_arrow_test, add_arrow_benchmark functions used throughout the build.
• FindLLVMAlt.cmake — Gandiva's LLVM finder.
• Find*.cmake — finders for the project's native dependencies (FindThriftAlt.cmake, FindgRPCAlt.cmake, FindAWSSDK.cmake, ...).
• ThirdpartyToolchain.cmake — orchestrates whether a dep is built from source or used from system.

Docker compose services

compose.yaml (~67 KB) is the source of truth for the build/test matrix. It defines services like:

• conda-cpp, conda-python, conda-r — conda-based builds.
• ubuntu-cpp, ubuntu-cpp-static, ubuntu-cpp-bundled — Ubuntu builds with different dep configurations.
• ubuntu-r, ubuntu-r-only-r — R builds.
• ubuntu-ruby, fedora-ruby — Ruby builds.
• ubuntu-c-glib — c_glib builds.
• python-wheel-manylinux-* — wheel builds.
• minio, azurite, gcs-server — local equivalents of S3/Azure/GCS for filesystem tests.

The shared base images come from ci/docker/, parametrized by .env.

Release tooling

dev/release/ contains the scripts that drive the Apache release process:

• 01-prepare.sh — bumps versions across the codebase.
• 02-source.sh — produces the source tarball.
• 03-binary-submit.sh — submits binary artifact builds via crossbow.
• 04-binary-download.sh — downloads binaries built by step 3.
• 05-binary-upload.sh — uploads binaries to the Apache distribution server.
• 06-rc-vote.sh — drafts the release vote email.
• post-* scripts — post-release tasks (announcement, PyPI upload, conda-forge feedstock bumps).

Per-language post-release helpers exist for PyPI, CRAN, RubyGems, etc.

Linting configuration

Tool	Config
clang-format	.clang-format, c_glib/.clang-format, r/src/.clang-format
clang-tidy	.clang-tidy, .clang-tidy-ignore
cpplint	CPPLINT.cfg, cpp/build-support/lint_cpp_cli.py
cmake-format	cmake-format.py
ruff/black	python/pyproject.toml
lintr (R)	r/.lintr
Air (R formatter)	r/air.toml
rubocop	.rubocop.yml
hadolint	.hadolint.yaml
shellcheck	.shellcheckrc
pre-commit	.pre-commit-config.yaml

CI scripts

ci/scripts/ holds the scripts that GitHub Actions invokes inside Docker containers. They are shared between the per-language workflows and the Docker compose services. Examples:

• cpp_build.sh, cpp_test.sh — C++ build and test entry points.
• python_build.sh, python_test.sh — PyArrow build and test.
• r_install.sh, r_test.sh, r_pkgdown.sh — R workflows.
• ruby_test.sh, c_glib_test.sh — Ruby and C-GLib tests.
• integration_arrow.sh, integration_dask.sh, ... — cross-language tests.
• install_minio.sh, install_azurite.sh, install_gcs_testbench.sh — local cloud-storage emulators for tests.