Development workflow

The Arrow monorepo has one branch (main), one release cadence (roughly quarterly major releases), and per-component development loops. This page walks through the workflows that contributors actually use.

Branching

• main is the only long-lived branch. All work targets main.
• Release branches (maint-23.0.x, maint-24.0.0, release-24.0.0-rc0, ...) are created at release time and used for backports.
• Feature work happens on contributor forks — there are no shared topic branches.

Local development loop

The cycle that maintainers run when developing C++:

# 1. Pick a component and configure its build
cmake --preset ninja-debug-python -S cpp -B cpp/build

# 2. Build incrementally
cmake --build cpp/build --target arrow_test

# 3. Run the test you're targeting
./cpp/build/release/arrow-array-test --gtest_filter='ArrayTest.Slice'

# 4. Repeat steps 2-3

For Python, the loop is:

cmake --build cpp/build --target install
cd python && pip install -e . --no-build-isolation
python -m pytest pyarrow/tests/test_array.py -k 'test_slice'

For R:

cd r
R -e 'devtools::load_all()'
R -e 'devtools::test(filter = "array")'

For Ruby:

cd ruby/red-arrow
bundle exec rake test TEST=test/test-array.rb

Crossbow

For changes that need to be tested across the full platform matrix (Linux distros, macOS, Windows, manylinux, conda, R-universe), maintainers use Crossbow — the project's CI fanout tool. It lives at dev/tasks/. A committer comments @github-actions crossbow submit <task> on a PR, and the system spawns the requested task.

The configuration lives in:

• dev/tasks/tasks.yml — task definitions (e.g. wheel-manylinux-2014, r-binary-packages).
• dev/tasks/conda-recipes/ — conda-forge recipe sources.
• dev/tasks/python-wheels/ — Python wheel build scripts.
• dev/tasks/r/ — R package builds for various platforms.

Pre-commit hooks

A .pre-commit-config.yaml is at the repo root. It runs:

• clang-format and cpplint on C++ changes
• cmake-format on CMake files
• ruff/black on Python (configured in python/pyproject.toml)
• rubocop on Ruby
• hadolint on Dockerfiles
• shellcheck on shell scripts
• prettier/markdownlint where applicable

Install with pip install pre-commit && pre-commit install. Hooks run on staged files; CI also enforces them.

Lint and format scripts

If you don't use pre-commit, the underlying scripts live at:

• cpp/build-support/run-clang-format.py
• cpp/build-support/lint_cpp_cli.py
• cpp/build-support/run-clang-tidy.py
• cpp/build-support/iwyu/
• dev/archery/archery/cli.py lint --help

archery lint is the umbrella entry point that knows how to run every linter the project uses.

Working in Docker

For reproducing CI locally without polluting your host:

docker compose run --rm conda-cpp           # build C++ in conda env
docker compose run --rm conda-python        # build pyarrow
docker compose run --rm ubuntu-r            # build R package
docker compose run --rm ubuntu-c-glib       # build c_glib + Ruby
docker compose run --rm ubuntu-cpp-static   # static-only build

compose.yaml is the source of truth — over 60 services are defined there, mirroring the platforms CI runs.

Updating generated code

Several files in the repo are generated, and contributors who touch them must regenerate the artifacts:

• FlatBuffers headers under cpp/src/generated/. Regenerate from format/*.fbs with flatc.
• Thrift Parquet types in cpp/src/generated/parquet_types.{h,cpp,tcc}. Regenerate from cpp/src/parquet/parquet.thrift using thrift --gen cpp.
• SIMD bit-packing kernels in cpp/src/arrow/util/bpacking_*_generated_internal.h. Regenerate via cpp/src/arrow/util/bpacking_simd_codegen.py and bpacking_scalar_codegen.py.
• Parquet chunker in cpp/src/parquet/chunker_internal_generated.h. Regenerate via cpp/src/parquet/chunker_internal_codegen.py.
• R arrowExports.cpp and arrowExports.R. Regenerate by running R -e 'cpp11::cpp_register()' from the r/ directory.
• C-GLib introspection data. Regenerated as part of the meson build.

The CI workflow dev.yml includes checks that fail when generated files drift from their sources.

Updating documentation

Reference docs are in docs/source/ and built with Sphinx (docs/Makefile, docs/source/conf.py). The build is reproduced in CI by .github/workflows/docs.yml. The R package has its own docs (r/_pkgdown.yml) built by pkgdown.