Dependencies

A high-level catalog of the third-party libraries Spark relies on. The full lock files are in dev/deps/ (per Hadoop profile) and the per-module Maven pom.xml. Pinned Python dependencies for development live in dev/requirements.txt.

JVM core

Library	Why
Scala 2.13	Engine implementation language.
Hadoop client	HDFS access, Hadoop config, FileSystem, Kerberos, history-server FS.
Netty	RPC and block transfer.
Jetty	Embedded HTTP for the Web UI and the History Server.
Jackson	JSON parsing, listener-event log format.
Snappy / LZ4 / Zstd / Brotli	Block and shuffle compression codecs.
Janino	Runtime compilation of generated Java code (Tungsten).
RoaringBitmap	Compact partition bitmaps in MapStatus.
Guava (shaded)	Internal collections; carefully shaded to avoid version conflicts.
Apache Commons (lang3, math3, codec, text, compress, io)	Various utility code.
Joda Time / java.time	Catalyst date/time helpers (java.time is the canonical path now).
Apache Arrow	Columnar IPC for PySpark, Spark Connect, vectorized SQL.
ANTLR4	SQL grammar (sql/catalyst/.../parser/SqlBase*.g4).
RocksDBJNI	Backing store for common/kvstore and the RocksDB state store.
Dropwizard Metrics	Metrics system core.
Avro	Avro data source.
Protobuf 3	Spark Connect wire format and the protobuf data source.
gRPC (Java)	Spark Connect server and client.
HikariCP, ZooKeeper, Curator (in some profiles)	Spark Master HA and JDBC pooling.

The full version pins live in the root pom.xml properties section and in the per-Hadoop profile lock files under dev/deps/.

SQL stack

Library	Why
Parquet (Apache)	First-class data source, vectorized reader.
ORC	First-class data source, vectorized reader.
Apache Commons CSV	CSV parsing under sql/catalyst/.../csv/.
Univocity (replaced)	Older CSV parser; replaced by Commons CSV.
Hive metastore client	When -Phive is on. Multiple shaded versions via IsolatedClientLoader.
Hive serde and exec	When -Phive is on.
Calcite (small subset)	Legacy SQL utilities; most parsing is ANTLR-driven now.

Cluster managers

Library / dep	Where
Hadoop YARN client	resource-managers/yarn/
Fabric8 Kubernetes client	resource-managers/kubernetes/core/
Apache ZooKeeper / Curator	Standalone Master HA

MLlib and graphX

Library	Why
Breeze (small subset)	Numerical primitives; some legacy use.
netlib-java + JBLAS	BLAS/LAPACK bindings used by mllib-local.
OpenJDK Vector API	Where available (Java 21+), accelerates dense linear algebra.
Stanford MLE / lbfgsb4j	Optimization helpers in some MLlib algorithms.

Connectors

Module	Pulls in
connector/kafka-0-10*	Kafka clients 3.x.
connector/kinesis-asl	AWS SDK + Kinesis Client Library.
connector/protobuf	protobuf-java, protobuf-java-util.
connector/avro	Avro 1.11+.
connector/spark-ganglia-lgpl	Ganglia client (LGPL, separate artifact).
connector/profiler	async-profiler.

Python

dev/requirements.txt pins development dependencies for PySpark:

• pyarrow (runtime dep of the Arrow-optimized UDF path).
• pandas (runtime dep for pandas API on Spark).
• numpy, scipy (runtime deps for MLlib).
• grpcio, grpcio-status, grpcio-health-checking, googleapis-common-protos, protobuf (runtime deps for Spark Connect).
• mypy, pytest, flake8, black, coverage, mkdocs etc. (dev only).

The full lock is regenerated by dev/.gen-requirements.sh (when present).

R

R dependencies are listed in R/pkg/DESCRIPTION and installed by R/install-dev.sh. The core deps are methods, stats4, and (for Arrow-optimized paths) arrow.

Build

Tool / library	Where
SBT 1.x	build/sbt-launch-lib.bash, project/build.properties
Apache Maven 3.x	build/mvn
Zinc	Incremental Scala compiler.
sbt-pom-reader	Translates Maven POM info into SBT settings.
sbt-mima	Binary compatibility check.
sbt-protobuf, ScalaPB	Protobuf code generation.
sbt-scalastyle	Scalastyle integration.
Apache RAT	License header check (dev/check-license).

License notes

• Most code is Apache 2.0.
• connector/spark-ganglia-lgpl/ is LGPL and ships as a separate artifact.
• LGPL/EPL dependencies are kept opt-in via Maven profiles.
• LICENSE-binary and licenses-binary/ capture the redistribution licenses for the prebuilt distributions.