By the numbers

A quantitative snapshot of the Spark codebase. Data collected on 2026-04-30 from the master branch at commit c9e2d65176e.

Size

xychart-beta horizontal
    title "Lines of code by language"
    x-axis ["Scala", "Python", "Java", "SQL fixtures", "R", "Protobuf"]
    y-axis "Lines (thousands)" 0 --> 1800
    bar [1737, 443, 187, 71, 37, 7]

Language	Files	Total lines
Scala	5,935	1,736,888
Python	1,405	442,629
Java	1,300	187,222
SQL test fixtures	536	70,876
R	98	36,635
Protobuf	25	7,355

Counts are taken with find ... | xargs cat | wc -l on the working tree and exclude .git/. SQL fixtures are mostly golden files for SQLQueryTestSuite and its variants.

Top-level layout

Top-level directory	What lives there
core/	RDDs, scheduler, storage, shuffle, RPC, deploy, UI
sql/	Catalyst, SQL execution, Hive integration, Connect, pipelines
python/	PySpark (sql, ml, mllib, pandas, streaming, pipelines, testing)
R/	SparkR
mllib/, mllib-local/	Machine-learning library
graphx/	Graph processing on RDDs
streaming/	Legacy DStream API
connector/	Avro, Kafka, Kinesis, Protobuf, Ganglia, profiler, Docker tests
common/	Network, unsafe memory, kvstore, sketch, variant, utils, tags
resource-managers/	YARN and Kubernetes integrations
assembly/	Final assembled jars
examples/	Sample programs in Scala, Java, Python, R
docs/	Jekyll-rendered user-facing site
dev/	Linters, test runner, JIRA helpers, release scripts
bin/, sbin/	User-facing scripts

Activity

• Total commits on master: 47,973 (as of 2026-04-30).
• Commits on master in the last 6 months: 2,094.
• The first commit, df29d0ea4c8 Initial commit, is dated 2010-03-29.
• Recent tags include preview releases for Spark 4.2.0 and Spark 4.1.x maintenance branches.

Bot-attributed commits

A spot-check of the most recent 1,000 commits on master finds essentially zero bot co-authors (*[bot] accounts) on this repository. Apache Spark uses a manual review and merge workflow on JIRA-tracked tickets, so AI- and bot-assisted work, if present, leaves no consistent trace in git history.

Complexity

The single largest source files in core (the engine module) give a sense of where the heaviest logic lives:

File	Approximate size
core/.../SparkContext.scala	~148 KB
core/.../scheduler/DAGScheduler.scala	~167 KB
core/.../scheduler/TaskSchedulerImpl.scala	~59 KB
core/.../scheduler/TaskSetManager.scala	~66 KB
core/.../storage/BlockManager.scala	~95 KB
core/.../storage/BlockManagerMasterEndpoint.scala	~50 KB
core/.../storage/ShuffleBlockFetcherIterator.scala	~77 KB
core/.../MapOutputTracker.scala	~75 KB
core/.../ExecutorAllocationManager.scala	~49 KB

In Python, python/pyspark/worker.py is ~150 KB and houses the Python-side worker loop and UDF dispatch.

Module count

• SBT project modules: ~70+ (defined in project/SparkBuild.scala).
• Maven modules: 80+ (declared from pom.xml).
• Connectors: 9 (avro, kafka-0-10, kafka-0-10-assembly, kafka-0-10-sql, kafka-0-10-token-provider, kinesis-asl, kinesis-asl-assembly, profiler, protobuf, spark-ganglia-lgpl, docker-integration-tests).

Where to look for ratios

For per-module file counts, test-to-code ratios, and contributor history, see each module page under modules/. Subsystem-level numbers (scheduler, storage, shuffle, ...) are summarized in systems/.