Configuration

Spark has hundreds of configuration entries. They are declared in code via org.apache.spark.internal.config.ConfigBuilder so the type, default, version, and documentation live with the entry.

Where configs live

File	What it contains
core/.../internal/config/package.scala	The main core-side config catalog.
core/.../internal/config/ConfigBuilder.scala	The builder DSL.
core/.../internal/config/Deploy.scala	Deploy-related configs.
core/.../internal/config/DynamicAllocation.scala	Dynamic-allocation knobs.
core/.../internal/config/Network.scala	RPC and network configs.
core/.../internal/config/Status.scala	UI/listener-bus configs.
core/.../internal/config/Tests.scala	Test-only configs.
sql/core/.../internal/SQLConf.scala	SQL session configs (spark.sql.*).
sql/catalyst/.../internal/SQLConf.scala	The shared base used by both classic and Connect.
sql/connect/server/.../config/Connect.scala	Spark Connect server configs.
mllib/.../ml/Param.scala	ML pipeline params (different mechanism).

Each ConfigEntry records:

• Key (spark.foo.bar).
• Default value.
• Documentation string.
• The Spark version it was introduced in (.version("3.4.0")).
• A type converter (bytesConf, intConf, stringConf, ...).

How to read a config

To find what a config does, search the codebase for the literal key:

rg 'spark\.foo\.bar' --type scala

The match in a *Config*.scala will lead you to the ConfigBuilder block with the docstring and version.

Major families

Family prefix	Where declared
spark.driver.*	core/.../internal/config/package.scala
spark.executor.*	core/.../internal/config/package.scala
spark.dynamicAllocation.*	core/.../internal/config/DynamicAllocation.scala
spark.shuffle.*	core/.../internal/config/package.scala
spark.storage.*	core/.../internal/config/package.scala
spark.memory.*	core/.../internal/config/package.scala
spark.network.*	core/.../internal/config/Network.scala
spark.rpc.*	core/.../internal/config/Network.scala
spark.sql.*	sql/core/.../internal/SQLConf.scala
spark.sql.adaptive.*	sql/core/.../internal/SQLConf.scala (AQE section)
spark.sql.streaming.*	sql/core/.../internal/SQLConf.scala
spark.sql.connect.*	sql/connect/server/.../config/Connect.scala
spark.kubernetes.*	resource-managers/kubernetes/core/.../deploy/k8s/Config.scala
spark.yarn.*	resource-managers/yarn/.../config/
spark.history.*	core/.../internal/config/History.scala
spark.eventLog.*	core/.../internal/config/package.scala
spark.metrics.*	core/.../internal/config/Metrics.scala
spark.ui.*	core/.../internal/config/UI.scala
spark.master.*	core/.../deploy/master/
spark.worker.*	core/.../deploy/worker/
spark.log.*	common/utils/.../logging/

Setting configs

Configs can be set in any of these ways (in increasing priority):

1. spark-defaults.conf.
2. JVM system property.
3. CLI flag to spark-submit (--conf spark.foo.bar=value).
4. Programmatically: SparkConf().set("spark.foo.bar", "value").
5. SQL session: SET spark.sql.foo=value (for spark.sql.* only; the entry must allow runtime modification).

spark-submit (core/.../deploy/SparkSubmit.scala) merges (1)-(3) into a SparkConf that is then handed to the user code.

Generated documentation

The website's configuration reference is generated from the in-code definitions:

• sql/gen-sql-config-docs.py and friends emit sql-ref-configuration.md.
• The non-SQL entries are documented manually in docs/configuration.md.

Adding a new config

1. Add a ConfigEntry in the right *Config*.scala.
2. Pick a non-trivial version("X.Y.Z") (the version the config first ships in).
3. Use the entry via conf.get(MY_CONFIG) rather than reading the string key.
4. Add a doc string. It is what shows up on the website.
5. For SQL configs, mark whether the entry is staticConf (cannot be changed at runtime), dynamicConf, or session-scoped.
6. If the config affects a Connect server config, also expose it through the Connect protocol if needed.

Dangerous configs

• spark.driver.maxResultSize - too low truncates results; too high risks driver OOM.
• spark.shuffle.spill.compress and friends - changing compression mid-stream breaks long-running streaming jobs.
• spark.memory.fraction - lower values waste heap; higher values starve user code.
• spark.sql.shuffle.partitions - the most-tuned SQL knob.
• spark.scheduler.allocation.file (FAIR scheduler XML) - syntax errors silently fall back to FIFO.

Where to look for the user-facing description

After locating the ConfigEntry, the website rendering of its doc string is at:

• Core: https://spark.apache.org/docs/latest/configuration.html
• SQL: https://spark.apache.org/docs/latest/sql-ref-configuration.html
• K8s: https://spark.apache.org/docs/latest/running-on-kubernetes.html
• YARN: https://spark.apache.org/docs/latest/running-on-yarn.html