Getting started

This page is a fast path to a local Spark build, an interactive shell, and a passing test run. For deeper coverage see the project documentation at spark.apache.org/docs/latest/building-spark.html and the local guide at docs/building-spark.md.

Prerequisites

Tool	Version	Notes
Java (JDK)	17 or 21 (master also CI-builds on 25)	LTS Temurin works well
Scala	Bundled - the build/sbt wrapper uses the right version	Scala 2.13 is the default
Maven	3.9+ (only if building with Maven; bundled wrapper at build/mvn)	SBT is preferred for development
Python	3.10+	For PySpark and most dev tooling
R	4+ (optional)	For SparkR (R is deprecated for users but still maintained for tests)
Git, curl, perl	Any reasonably recent versions

Cloning

git clone https://github.com/apache/spark.git
cd spark

The repo's developer notes (AGENTS.md / CLAUDE.md) recommend origin for your personal fork and upstream for apache/spark.

Build

The bundled wrapper scripts in build/ download a pinned version of SBT or Maven and Zinc into build/, so you do not need to install them yourself.

SBT (recommended for development)

# Compile the whole project
build/sbt compile

# Compile a single module (faster)
build/sbt core/compile
build/sbt sql/compile

# Build a runnable distribution
build/sbt package

Module names map to Maven artifact IDs and are listed in project/SparkBuild.scala. Examples: core, catalyst, sql, hive, connect-common, connect-server, mllib, graphx, streaming, kubernetes, yarn.

Maven

build/mvn -DskipTests clean package

To build with Hive support enabled (required for PySpark tests and the Hive ThriftServer):

build/mvn -Phive -Phive-thriftserver -DskipTests clean package
# or
build/sbt -Phive package

To build for Kubernetes or YARN:

build/mvn -Pkubernetes -DskipTests clean package
build/mvn -Pyarn      -DskipTests clean package

Run

Interactive shells

./bin/spark-shell    # Scala REPL
./bin/pyspark        # Python REPL
./bin/spark-sql      # SQL CLI (needs Hive support to be built)
./bin/sparkR         # R REPL

Submit a job

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local[4] \
  examples/target/scala-*/jars/spark-examples_*.jar 100

MASTER URLs:

• local[N] - in-process, N threads
• spark://host:7077 - standalone master
• yarn, k8s://https://... - cluster managers
• See core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

Spark Connect

./sbin/start-connect-server.sh

Then connect with the Python or JVM client. See modules/connect.md.

Test

Build and tests can take a long time on a full clean build. The standard developer workflow is to run only the suite or test cases relevant to your change.

Scala / JVM

# Whole module
build/sbt 'core/test'

# One suite
build/sbt 'sql/testOnly *DataFrameSuite'

# One test name (substring match via -z)
build/sbt 'sql/testOnly *DataFrameSuite -- -z "select"'

For interactive use, keep SBT running:

build/sbt
> project sql
> testOnly *DataFrameSuite

Python

build/sbt -Phive package    # PySpark needs Hive built
python3 -m venv .venv && source .venv/bin/activate
pip install -r dev/requirements.txt

python/run-tests --testnames pyspark.sql.tests.test_dataframe
python/run-tests --testnames "pyspark.sql.tests.test_catalog CatalogTests.test_current_database"

Aggregated runner

./dev/run-tests runs the full developer test pipeline and is what CI uses. It detects which modules changed and runs the appropriate subset. Source: dev/run-tests.py.

Lint and style

Check	How to run
Scalastyle	build/sbt scalastyle test:scalastyle
Java code style	build/sbt javaformat:format
Python (flake8, mypy)	dev/lint-python (uses tools listed in dev/requirements.txt)
R	dev/lint-r
Markdown / docs	dev/lint-other and docs/README.md

A pre-commit hook is wired up in .pre-commit-config.yaml:

pip install pre-commit
pre-commit install

Useful directories for new contributors

Path	What it does
bin/, sbin/	User scripts (spark-shell, spark-submit, history server, ...)
build/	Pinned wrappers for SBT, Maven, Zinc
dev/	Developer scripts: lint, test runner, JIRA helper, release
examples/	Example programs (Scala, Java, Python, R)
docs/	User-facing documentation rendered to spark.apache.org
project/SparkBuild.scala	The SBT build definition; canonical module list
pom.xml	The Maven build definition (kept in sync with SBT)
.github/workflows/	CI pipelines

Where to file an issue or PR

• JIRA: issues.apache.org/jira/projects/SPARK
• PRs: open against the master branch on apache/spark, with a title of the form [SPARK-xxxxx][COMPONENT] Title. See how-to-contribute/index.md.