Lore

A timeline of how the Spark codebase grew. Dates are derived from git tags and commit history. This is a narrative tour, not a feature list. For raw stats see by-the-numbers.md.

Eras

The Berkeley research era (Mar 2010 - May 2014)

• Mar 29, 2010 - First commit df29d0ea4c8 Initial commit. Spark begins life as a research project at UC Berkeley's AMPLab.
• 2011-2013 - Donated to the Apache Software Foundation, accepted as an Apache Top-Level Project in February 2014.
• The original surface area is small: an RDD abstraction, a DAG scheduler, a Mesos backend, and a handful of broadcast and shuffle implementations. Most of what is now core/ was the whole project.

Spark 1.x: structured APIs are born (May 2014 - Jul 2016)

• Spark 1.0 (May 2014). Stable RDD API; first release as a TLP.
• Spark 1.3 (Mar 2015). The DataFrame API ships and begins displacing direct RDD use.
• Spark 1.4-1.6. The Project Tungsten effort lands. Off-heap memory, the unsafe row format in common/unsafe, and a new UnifiedMemoryManager in core/.../memory/ change how executors use heap.
• The Catalyst tree-rewrite framework (sql/catalyst/) and the Hive ThriftServer arrive.

Spark 2.x: Datasets, Structured Streaming, ML (Jul 2016 - Nov 2019)

• Spark 2.0 (Jul 2016). Dataset API consolidates with DataFrame. SparkSession replaces the zoo of contexts. Whole-stage code generation (WholeStageCodegenExec) compiles operator trees to one JVM method via Janino.
• Spark 2.1-2.2. Structured Streaming graduates from experimental. sql/core/.../execution/streaming/ becomes the strategic streaming codebase. The legacy streaming/ (DStreams) enters maintenance mode.
• Spark 2.3 (Feb 2018). First-class Kubernetes scheduler backend lands in resource-managers/kubernetes/. Continuous Processing arrives as an experimental mode.
• Spark 2.4. ML pipelines, image data source, native barrier execution mode for ML/MPI workloads (core/.../BarrierTaskContext.scala).

Spark 3.x: AQE, GPU, ANSI, K8s GA (Jun 2020 - Jun 2024)

• Spark 3.0 (Jun 2020). Adaptive Query Execution (sql/core/.../execution/adaptive/) becomes the default optimization layer. Dynamic partition pruning, accelerator-aware scheduling (GPU resource profiles in core/.../resource/), and SQL ANSI mode all land here.
• Spark 3.1. Kubernetes goes GA. State store APIs grow.
• Spark 3.2 (Oct 2021). The pandas API on Spark (originally Project Koalas) is merged into PySpark at python/pyspark/pandas/. This is the single largest external contribution in PySpark history.
• Spark 3.3-3.4. Push-based shuffle (core/.../storage/PushBasedFetchHelper.scala, common/network-shuffle) and executor decommissioning (core/.../storage/FallbackStorage.scala) ship. RocksDB is added as a state-store backend.
• Spark 3.4 (Apr 2023). Spark Connect arrives in sql/connect/. The gRPC protocol in sql/connect/common/src/main/protobuf/spark/connect/ lets thin Python and JVM clients drive the engine without embedding the JVM.
• Spark 3.5. Connect goes GA. The Python ML client and the Scala client mature.
• The JVM minimum is bumped from Java 8 to Java 11, then to Java 17 by the end of the 3.x series.

Spark 4.x: SQL scripting, declarative pipelines, Variant (Jun 2024 - present)

• Spark 4.0 (Jun 2024). The minimum JDK is Java 17. The Variant data type ships (common/variant/) and is wired through Catalyst. SQL scripting (sql/core/.../scripting/, sql/catalyst/.../trees/SqlScriptingContextManager.scala) brings PL/SQL-style control flow.
• Spark 4.1 / 4.2 (preview through 2025-2026). The recently-introduced declarative pipelines feature lives at sql/pipelines/ and python/pyspark/pipelines/. The Avro, Protobuf, and XML data sources have been promoted out of connector/ into first-class modules. The Hive Thrift server gets pruned and refactored.
• Java 21 is the recommended target; Java 25 is in CI.

Longest-standing code

• core/.../rdd/RDD.scala and core/.../SparkContext.scala trace back to the very first commits in 2010 and have survived every refactor.
• The DAGScheduler (core/.../scheduler/DAGScheduler.scala) is one of the oldest substantial files. It has been extended many times (barrier execution, push-based shuffle, decommissioning, AQE-aware re-submission) but the stage-DAG construction algorithm is recognizably the original one.
• The Mesos backend was removed in Spark 3.x, but standalone mode lives on in core/.../deploy/{master,worker}/ largely unchanged from its early form.

Major rewrites and migrations

• Hadoop 1 -> Hadoop 2 / YARN (2013-2014). The first resource-managers/yarn/ lands.
• Tungsten (Spark 1.4-1.6). Off-heap memory, the Unsafe row, code-generated stages.
• DataSource V2 (Spark 2.3 onward). Pluggable connectors are reworked under sql/catalyst/.../connector/ and sql/core/.../execution/datasources/v2/.
• Python typing (Spark 3.x). PySpark gains a stub-driven typing layer (python/pyspark/_typing.pyi, mypy.ini) and structural API parity with the Scala side.
• Spark Connect (Spark 3.4 onward). A second public surface emerges. The Scala API now has both a "classic" implementation (sql/core/.../sql/classic/) and a Connect-backed client (sql/connect/client/).
• Java 8 -> 11 -> 17 -> 21 (2020-2024). The minimum JVM version is bumped twice in five years; many CI matrices are pruned.
• Mesos removal (Spark 3.5). The Mesos scheduler backend, which was foundational in 2014, is dropped from the repo.
• Scala 2.12 -> 2.13 (Spark 3.2 default). The 2.13 cross-build becomes the default; Scala 2.13 is now the only supported version for Spark 4.

Deprecated / removed

• Mesos. Removed in 3.5.
• DStreams (streaming/). Maintained but not extended; Structured Streaming is the strategic API.
• SparkR (user-facing). Marked deprecated in Spark 4 release notes. The R code in R/ remains for transitional support and is still tested in CI.
• Hive 1.x support and several legacy SerDes were dropped in 3.x.

Growth signals

The three subdirectories that have grown most dramatically since Spark 3.0:

1. python/pyspark/pandas/ - born in Spark 3.2, now ~100k LoC.
2. sql/connect/ - born in Spark 3.4, now five subprojects (client, common, server, shims, bin).
3. sql/core/.../execution/streaming/ and the state store - reflects the heavy ongoing investment in Structured Streaming and stateful operators.