Tooling

The build system, linters, CI workflows, and dev scripts that surround the Spark source tree.

Build systems

Spark maintains both an SBT build (canonical for development) and a Maven build (canonical for artifact publishing).

SBT

• Definition: project/SparkBuild.scala. Lists every module, its dependencies, and per-module settings.
• Wrapper: build/sbt. Downloads a pinned SBT version on first run.
• Plugins: project/plugins.sbt (sbt-mima for binary compatibility, sbt-pom-reader for pom consumption, sbt-protobuf, sbt-scalastyle).

Common commands:

build/sbt compile
build/sbt 'core/test'
build/sbt 'sql/Test/compile'
build/sbt 'project sql' assembly

Maven

• Definition: pom.xml plus per-module pom.xml files. Profiles control optional features (-Phive, -Phive-thriftserver, -Pkubernetes, -Pyarn, -Psparkr, -Pjvm-profile, -Pscala-2.13).
• Wrapper: build/mvn. Picks up build/scala-build.sh for Scala compilation via Zinc.

build/mvn -Phive -DskipTests clean package

The Maven build is what gets used for releases and to publish artifacts to Maven Central.

Linters

Linter	Runs on	Wrapper script
Scalastyle	Scala	build/sbt scalastyle
Java code formatter	Java	build/sbt javaformat:format
flake8	Python	dev/lint-python
mypy	Python	dev/lint-python
black	Python	dev/reformat-python (manual)
lintr	R	dev/lint-r
xmllint, header check	XML, license headers	dev/lint-other
Markdown / docs	docs	dev/lint-other

The pre-commit configuration in .pre-commit-config.yaml wires several linters into a git hook:

pip install pre-commit
pre-commit install

CI

CI is GitHub Actions. The workflow files live in .github/workflows/:

• build_main.yml - the main build matrix (Scala, Java 17/21/25, Python versions).
• build_python_*.yml - per-Python-version PySpark builds.
• build_maven*.yml - Maven-based builds.
• build_sparkr_window.yml - SparkR on Windows.
• release.yml - tag-driven release builds.
• publish_snapshot.yml - nightly snapshot artifacts.
• build_infra_images_cache.yml - rebuilds the cached Docker images CI uses.

The workflows call into dev/run-tests and dev/run-tests.py, which decide which modules to exercise based on the file paths changed.

Code generators

Several generators run as part of the build or on demand:

• ANTLR -> Scala parser. The grammar is sql/catalyst/.../parser/SqlBaseParser.g4 and SqlBaseLexer.g4. Generated to target/generated-sources/.
• Protobuf -> Java/Python. Generated from sql/connect/common/src/main/protobuf/ and connector/protobuf/.
• Hive metastore stubs - shaded in sql/hive.
• SQL function docs - sql/gen-sql-functions-docs.py writes the per-function pages used by the website. sql/create-docs.sh is the entry point.
• Error class docs - docs/util/build-error-docs.py consumes the JSON in common/utils/src/main/resources/error/error-conditions.json.

Documentation site

User-facing docs are in docs/ and built with Jekyll. To preview locally:

cd docs
bundle install
bundle exec jekyll serve

API docs (Scaladoc, Javadoc, PyDoc, R docs) are wired into docs/_plugins/copy_api_dirs.rb and stitched together by docs/README.md.

Release tooling

• dev/create-release/ - the scripts maintainers run to cut a release.
• dev/merge_spark_pr.py - the helper that committers use to merge PRs. It rewrites the commit message to reference the JIRA ticket and closes the JIRA on push.
• dev/change-scala-version.sh - cross-Scala-version helper.
• dev/check-license - ASF header check.

Environment helpers

• dev/requirements.txt - pinned Python dev dependencies.
• dev/infra/Dockerfile, dev/spark-test-image/ - Docker images CI uses.
• dev/.rat-excludes - input to the Apache RAT license scan.

Editor support

• IntelliJ IDEA: import the pom.xml (Maven) or use the SBT plugin. The repo includes .idea/ ignores in .gitignore.
• VS Code: see python/conf_vscode/ for PySpark debug configurations.
• Java/Scala formatting profiles are documented in dev/codestyle/ (when present) and on the project website.