Modules

The Spark repo is a multi-module project. SBT and Maven both treat core, the SQL family, and the connectors as first-class modules. This section walks through each major top-level module: where it lives, what it does, and how it relates to the rest of the engine.

How modules are wired

• The canonical SBT module list is project/SparkBuild.scala. Search for lazy val to find the modules.
• Maven modules are declared in the root pom.xml and the per-directory pom.xml files.
• Module names map to artifact IDs published to Maven Central as org.apache.spark:spark-<name>_2.13.

Top-level modules

Page	Source roots	Summary
core	core/	RDDs, scheduler, storage, shuffle, RPC, Web UI
sql	sql/api, sql/core	DataFrames, Datasets, SQL execution, AQE
catalyst	sql/catalyst	The plan/expression tree-rewrite framework
hive	sql/hive, sql/hive-thriftserver	Hive metastore integration and the JDBC server
connect	sql/connect/{client,common,server,shims,bin}	gRPC layer for thin Spark clients
streaming	sql/core/.../execution/streaming, streaming/	Structured Streaming and the legacy DStream API
mllib	mllib, mllib-local	Machine learning library
graphx	graphx/	Graph processing on RDDs
pyspark	python/pyspark/	Python bindings, pandas-on-Spark, Connect client
sparkr	R/	R bindings (deprecated for users)
resource-managers	resource-managers/yarn, resource-managers/kubernetes	Cluster manager integrations
connectors	connector/*	Avro, Kafka, Kinesis, Protobuf, profiler, Ganglia
common	common/*	Low-level libraries shared by every module

Reading order

If you are new, the most useful path is:

1. core - the engine that everything else runs on.
2. catalyst - how plans get built and rewritten.
3. sql - how plans become RDDs and run.
4. connect - the modern client-server split.
5. Then any specific module that maps to your work.

For deeper internals, see the cross-cutting systems/ pages.