common

common/ is the home of small, low-level libraries that are shared across the engine and sit below the SQL stack. Each subdirectory is its own published Maven artifact. Most of the time you only touch these when working on memory, networking, the kvstore behind the UI, or sketches.

Modules

Subdirectory	Artifact	What it provides
common/utils/	spark-common-utils	Small utilities, the Logging trait, structured logging.
common/utils-java/	spark-common-utils-java	Pure-Java utilities used everywhere.
common/tags/	spark-tags	The @Experimental, @DeveloperApi, @Stable annotations.
common/unsafe/	spark-unsafe	Off-heap memory, unsafe-row format, Tungsten primitives.
common/network-common/	spark-network-common	Netty-based RPC and block transfer.
common/network-shuffle/	spark-network-shuffle	The external shuffle service and push-based shuffle protocol.
common/network-yarn/	spark-network-yarn	YARN-specific shuffle service entrypoints.
common/kvstore/	spark-kvstore	A small, embeddable key/value store with in-memory and RocksDB backends.
common/sketch/	spark-sketch	Count-min sketch, Bloom filter, and HyperLogLog++ implementations.
common/variant/	spark-variant	The VARIANT data type added in Spark 4.

Each lives behind a thin Maven module declaration; SBT defines them in project/SparkBuild.scala.

common/unsafe

The Tungsten effort (Spark 1.4-1.6) introduced an off-heap memory model and a binary row format. common/unsafe/ houses:

• Platform (common/unsafe/.../Platform.java) - allocation, copy, and primitive load/store via sun.misc.Unsafe (with a Java 17+ adapter using MemorySegment-style helpers).
• MemoryBlock and MemoryAllocator - on-heap and off-heap allocators.
• UTF8String - immutable UTF-8 byte string, used as the canonical Spark string.
• UnsafeRow, UnsafeArrayData, UnsafeMapData - the binary row format that flows through whole-stage code generation in sql/core.
• BytesToBytesMap and the UnsafeExternalSorter family - lock-free hash map and external sorter used by hash aggregation and shuffle.

common/network-common

Netty-backed transport used everywhere RPC happens:

• TransportClient / TransportServer - generic client/server with framing, retries, and authentication.
• SaslClientBootstrap / AuthEngine - SASL and the new auth-engine handshake.
• BlockTransferService - public interface used by Spark's BlockManager to fetch remote blocks.

The Spark RPC layer in core/.../rpc/netty/ is built on top of this module.

common/network-shuffle

Houses the external shuffle service, which lets shuffle blocks survive an executor's death:

• ExternalShuffleService - long-running daemon launched on each node.
• ExternalBlockHandler - implements the shuffle-fetch protocol on the server side.
• Push-based shuffle protocol messages (PushBlockStream, FinalizeShuffleMerge, ...).
• The MergeManager that assembles pushed blocks into mergers.

A YARN-specific entry point in common/network-yarn/ packages the service as a YARN auxiliary service.

common/kvstore

A tiny embedded K/V store used by:

• The Web UI / History Server (core/.../status/) to back AppStatusStore.
• The Spark Connect server to cache per-session state.

Two backends:

• InMemoryStore - hash-map-backed, used when the dataset is small.
• RocksDBStore - disk-backed, used by the History Server for very large event logs and by the K8s shuffle service for state.

common/sketch

Implementations of three approximate-data structures:

• BloomFilter - used by Catalyst's runtime-bloom-filter join optimization.
• CountMinSketch - approximate frequency counts.
• HyperLogLogPlusPlus - approximate distinct count, used by SQL's approx_count_distinct.

common/variant

The VARIANT data type added in Spark 4. A binary, schema-on-read JSON-like format that flows through Catalyst, Parquet, and the SQL parse_json and variant_get family of functions. Designed for analytics on semi-structured data.

common/utils

• org.apache.spark.internal.Logging - the trait every Scala class inherits to get a logger.
• org.apache.spark.internal.logging.* - the structured-logging layer that emits MDC fields in JSON.
• error/error-conditions.json - the catalog of user-facing error classes.

Integration points

• unsafe, network-common, and kvstore are dependencies of core/.
• network-shuffle is also a dependency of the YARN and K8s shuffle plugins.
• sketch is consumed by sql/catalyst for the runtime Bloom filter join.
• variant is consumed by sql/catalyst and sql/core's Parquet/Avro writers.

Entry points for modification

• Off-heap or row-format change: start in common/unsafe/ and trace the consumer in sql/catalyst/.../expressions/codegen/ and sql/core/.../execution/.
• New RPC message or protocol field: edit common/network-common/'s Message hierarchy and the relevant subprotocol (shuffle/protocol/ for shuffle, sasl/ for auth).
• Add a new sketch: drop a class under common/sketch/.../ and wire a SQL function in sql/catalyst/.../expressions/aggregate/.
• Add a new error class: edit common/utils/.../error/error-conditions.json.