Patterns and conventions

A reference for the conventions that show up across the Spark codebase. Many are enforced by lint; the rest are unwritten rules that maintainers will flag in review.

Language strategy

• Scala 2.13 is the canonical implementation language for the engine. Cross-builds against earlier versions of Scala were dropped in 4.x.
• Java is used where a stable ABI matters: low-level primitives (common/unsafe, common/network-common), the public Java API surface (core/.../api/java/), and where the Scala compiler's incremental complications hurt (some shuffle code).
• Python is hand-written; types live in .pyi stubs alongside .py files.
• R is hand-written; SparkR is in maintenance mode.
• SQL test fixtures and golden files. Never the implementation language.

ASCII-only outside strings

The Scalastyle config (scalastyle-config.xml) flags non-ASCII characters in code and comments. The common offenders are em-dashes, smart quotes, ellipses, and non-breaking spaces sneaking in from copy-paste. Spot-check before committing:

grep -rn -P "[^\x00-\x7F]" <changed files>

String literals and test data can be non-ASCII when the content requires it (error messages, i18n test cases). Identifiers and comments may not.

Error classes

User-facing errors go through the error-class system instead of raw exception messages. The catalog of error classes is common/utils/src/main/resources/error/error-conditions.json. Adding a new error means:

1. Add a class to that JSON file.
2. Throw it from Scala via throw QueryCompilationErrors.foo(...) or one of the *Errors helpers in sql/catalyst/.../errors/ and sql/core/.../errors/.
3. Update the migration guide if it changes user-visible behavior.

PySpark mirrors this with python/pyspark/errors/.

Logging

• Always use the Logging trait. Never call println or use System.out/System.err directly in engine code (tests are an exception).
• Prefer the structured-logging helpers (common/utils/.../logging/) when emitting MDC fields - operator name, stage id, partition, etc.
• Log levels: error for irrecoverable problems; warn for recoverable problems the user should know about; info for lifecycle events (job start/end, stage retries); debug for developer-facing detail.

Configuration

• Engine config is declared via org.apache.spark.internal.config.ConfigBuilder in files named *Config*.scala. The single largest list is in core/.../internal/config/package.scala and sql/core/.../internal/SQLConf.scala.
• Each config has a version annotation indicating the version it was introduced in. Migration guides reference these versions.
• Dynamic config (set per session/query) goes through SQLConf. JVM-process-wide config goes through SparkConf.

Plan rewrites

Catalyst rules extend org.apache.spark.sql.catalyst.rules.Rule. Rules go through batches in the analyzer (sql/catalyst/.../analysis/Analyzer.scala) and the optimizer (sql/catalyst/.../optimizer/Optimizer.scala). Conventions:

• A rule should be idempotent - running it twice produces the same plan.
• A rule should change as little of the tree as possible. Use transformDown, transformUp, or resolveOperatorsWithPruning (sql/catalyst/.../trees/TreePatternBits.scala) to limit the visit.
• New rules need a unit test. Most rules have a corresponding suite in sql/catalyst/src/test/.../optimizer/.

Public vs developer API

Source files are tagged with annotations:

• @Experimental - users can call it but it may change without notice.
• @DeveloperApi - intended for advanced extension; may break across minor releases.
• @Stable / no annotation in public types - covered by the source-compatibility promise.

Run dev/mima (the MiMa binary-compatibility check) before changing public API.

Source headers

Every new file must start with the standard Apache 2.0 license header (the same one in LICENSE). dev/check-license verifies this.

Tests

• Each Scala test suite extends SparkFunSuite (core/.../SparkFunSuite.scala) which mixes in time-bounded test execution and shared cleanup.
• DataFrame tests use QueryTest and its helpers (checkAnswer, checkDataset).
• For new SQL behavior, prefer adding a golden-file test under sql/core/src/test/resources/sql-tests/inputs/ over a hand-written suite.
• Streaming tests use StreamTest (sql/core/.../execution/streaming/StreamTest.scala).
• PySpark tests use the standard unittest framework. Connect parity tests live next to the classic tests under */connect/.

PR titles

Title format: [SPARK-NNNNN][COMPONENT] Title. The component is the uppercase last word of the JIRA component:

• Project Infra -> [INFRA]
• Spark Core -> [CORE]
• Structured Streaming -> [SS] (some authors use [STREAMING])
• SQL -> [SQL]
• Connect -> [CONNECT]
• PySpark -> [PYTHON]
• Kubernetes -> [K8S]
• Build -> [BUILD]

A PR may carry multiple component tags if it crosses subsystems, e.g., [SPARK-12345][SQL][PYTHON].

Don't

• Do not edit golden .sql.out files by hand. Regenerate.
• Do not push to apache/spark directly - always to your fork.
• Do not force-push after review starts unless explicitly asked.
• Do not call spark.stop() in test cleanup if the suite extends SharedSparkSession - the harness handles it.
• Do not introduce new public API without a JIRA discussion (and ideally a SPIP).