Apache Spark

Apache Spark is a unified analytics engine for large-scale data processing. It exposes high-level APIs in Scala, Java, Python, and R, and an optimized engine that runs general DAG-shaped computations on a cluster of machines. Spark ships with libraries for SQL and DataFrames, pandas-on-Spark, MLlib, GraphX, and Structured Streaming, all built on the same execution engine.

The codebase is hosted at github.com/apache/spark and is maintained under the Apache Software Foundation. The project's website is spark.apache.org.

What this wiki covers

This wiki is organized to help engineers navigate the Spark source tree. It describes:

• The high-level architecture and how requests flow through the engine
• How to get started building, running, and testing Spark locally
• The glossary of project-specific terms
• Each major source-tree module: core, sql, connect, mllib, graphx, Structured Streaming, PySpark, SparkR, and the resource managers
• Cross-cutting systems such as the DAG scheduler, block manager, shuffle service, RPC layer, and Web UI
• Reference material for configuration and dependencies

Repository at a glance

Layer	Where it lives
Core engine (RDDs, scheduler, storage, shuffle, RPC, UI)	core/src/main/scala/org/apache/spark/
Catalyst optimizer	sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/
SQL execution	sql/core/src/main/scala/org/apache/spark/sql/execution/
Hive integration	sql/hive/, sql/hive-thriftserver/
Spark Connect	sql/connect/{client,common,server}
MLlib	mllib/, mllib-local/
GraphX	graphx/
PySpark	python/pyspark/
SparkR	R/
Resource managers	resource-managers/yarn/, resource-managers/kubernetes/
Connectors	connector/avro, connector/kafka-0-10*, connector/protobuf, ...
Common subprojects	common/{kvstore,network-common,network-shuffle,unsafe,sketch,...}
Build / dev tooling	build/, dev/, project/SparkBuild.scala
User-facing docs	docs/ (rendered to spark.apache.org)

Where to go next

• New to the codebase? Read overview/architecture.md and then pick a module from modules/index.md.
• Building Spark for the first time? See overview/getting-started.md.
• Looking for a specific term (Catalyst, Tungsten, AQE, BlockManager, ...)? Check overview/glossary.md.
• Want to know how to contribute, run tests, or send a PR? Read how-to-contribute/index.md.