Testing

Spark's test culture is "every behavior change has a test", so most PRs include suite or golden-file additions. This page is the practical "how do I run X" reference.

The test runner

dev/run-tests is the entry point CI uses. The Python implementation lives in dev/run-tests.py, with module definitions in dev/sparktestsupport/modules.py. The runner:

1. Detects which modules changed in the PR diff (file path -> module mapping).
2. Runs Scalastyle, Java compile, and other static checks.
3. Runs the JVM tests for the affected modules.
4. Runs PySpark and SparkR tests if they are affected.
5. Runs pyspark-connect, pyspark-pandas, and other Python module tests as needed.

You generally do not run the full dev/run-tests locally - it is slow. Run the specific suite for your change.

Scala / JVM tests

Use SBT for incremental compile speed; Maven works too but is slower for development.

# Whole module
build/sbt 'core/test'
build/sbt 'sql/test'
build/sbt 'mllib/test'

# Single suite, anywhere in the project
build/sbt 'sql/testOnly *DataFrameSuite'

# Single test name (substring match via -z)
build/sbt 'sql/testOnly *DataFrameSuite -- -z "select"'

# Keep SBT open for fast iteration
build/sbt
> project sql
> testOnly *DataFrameSuite
> testOnly *.expressions.MathExpressionsSuite -- -z "tan"

Module names mirror the SBT projects in project/SparkBuild.scala. Frequently used: core, unsafe, kvstore, catalyst, sql, hive, hive-thriftserver, streaming, mllib, graphx, repl, connect-server, connect-client-jvm, kubernetes, yarn.

Python (PySpark) tests

PySpark tests need Hive support built first because they exercise the Hive-backed catalog:

build/sbt -Phive package

Activate a virtualenv (.venv is the documented default):

python3 -m venv .venv
source .venv/bin/activate
pip install -r dev/requirements.txt

Run a single suite or test case:

python/run-tests --testnames pyspark.sql.tests.test_dataframe
python/run-tests --testnames "pyspark.sql.tests.test_catalog CatalogTests.test_current_database"

Spark Connect Python tests live under python/pyspark/sql/tests/connect/, python/pyspark/ml/tests/connect/, etc. They run against a Connect server started by the test harness.

SQL golden-file tests

SQL semantics are tested with snapshot files. The driver class is SQLQueryTestSuite (and its variants) in sql/core/src/test/scala/org/apache/spark/sql/. Inputs live in sql/core/src/test/resources/sql-tests/inputs/ and golden outputs in .../results/.

# Run all SQL golden-file tests
build/sbt 'sql/testOnly *SQLQueryTestSuite'

# Regenerate golden files (only when intentional changes are made)
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt 'sql/testOnly *SQLQueryTestSuite'

Never edit the .sql.out files by hand. After regeneration, review the diff carefully - unexpected changes are usually a real regression.

R tests

build/sbt -Psparkr package
R/install-dev.sh
R/run-tests.sh

Integration / docker tests

Tests that need external services live in connector/docker-integration-tests/ and are guarded by Docker availability. Kubernetes integration tests are in resource-managers/kubernetes/integration-tests/; see that directory's README.

Benchmarks

Performance benchmarks are not part of the regular test runs but live alongside the code:

• core/benchmarks/, sql/core/benchmarks/, mllib/benchmarks/, python/benchmarks/.
• Each benchmark has a generated baseline .txt file. Re-run with the right SBT command (e.g., build/sbt 'sql/Test/runMain ...Benchmark') and commit the updated output.

Common pitfalls

• PySpark tests cannot find the JVM jars. You need a built distribution with Hive support; re-run build/sbt -Phive package after pulling.
• Tests pass locally but fail in CI. Many CI failures are flaky tests from streaming/ or sql/streaming/. Re-running the failed job is the standard first step. Persistent failures are tracked on JIRA.
• SQLQueryTestSuite golden file diff. Look at the actual SQL output difference. If it is intentional, regenerate; if not, fix the analyzer/optimizer change.
• Scalastyle ASCII errors. Caused by sneaking in non-ASCII characters such as smart quotes or em-dashes from a copy-paste. Run grep -rn -P "[^\x00-\x7F]" on changed files.