core

core/ is the engine. It contains the public RDD API, the scheduler, the storage layer, the shuffle subsystem, the RPC framework, the Web UI, and the standalone deploy mode. Every higher-level Spark API ultimately submits jobs through core.

Purpose

• Define the RDD abstraction and its operations.
• Build a stage DAG from RDD lineage and run it on a cluster.
• Manage block storage (cache, broadcast, shuffle output) on each node.
• Provide pluggable cluster-manager integration.
• Serve the Web UI and the event-log replay (History Server).

For a tour of the engine in motion, see overview/architecture.md.

Directory layout

core/src/main/scala/org/apache/spark/
  SparkContext.scala            148 KB - the canonical low-level entry point
  SparkEnv.scala                 22 KB - per-JVM holder of services
  SparkConf.scala                36 KB - configuration parsing
  Dependency.scala              - RDD dependency types (narrow, shuffle)
  Partitioner.scala             - hash, range, custom partitioners
  api/                          - Java and Python API bridges
  broadcast/                    - TorrentBroadcast and friends
  deploy/                       - SparkSubmit, standalone master/worker, history server
  executor/                     - Executor, ExecutorBackend, plugins
  internal/                     - config, plugins, observability internals
  io/                           - compression codecs, file commit protocols
  memory/                       - UnifiedMemoryManager, MemoryStore
  metrics/                      - MetricsSystem, sinks (JMX, Prometheus, ...)
  network/                      - Spark-side wrappers around common/network-common
  rdd/                          - RDD core ops, MapPartitionsRDD, ShuffledRDD, ...
  resource/                     - ResourceProfile, GPU/FPGA discovery
  rpc/                          - RpcEnv, NettyRpcEnv, endpoints
  scheduler/                    - DAGScheduler, TaskScheduler, listener bus
  security/                     - Kerberos delegation, hadoop tokens
  serializer/                   - Java and Kryo serializers
  shuffle/                      - SortShuffleManager, IndexShuffleBlockResolver
  status/                       - AppStatusStore, AppStatusListener (UI backend)
  storage/                      - BlockManager, DiskBlockManager, ShuffleBlockFetcherIterator
  ui/                           - Jetty-based Web UI
  util/                         - Utils, ThreadUtils, AccumulatorV2, JsonProtocol, ...
core/src/main/java/org/apache/spark/
  api/java/                     - Java versions of the RDD/JavaSparkContext APIs
  shuffle/api/                  - Pluggable shuffle SPI for non-default shuffle backends

Key abstractions

Type / file	What it is
SparkContext (core/.../SparkContext.scala)	The driver-side root. Owns scheduler, RPC env, BlockManager, UI.
SparkSession (sql/core/.../sql/classic/SparkSession.scala)	The SQL-aware driver entry point. Wraps SparkContext.
SparkEnv (core/.../SparkEnv.scala)	A bag of per-JVM services (memory manager, RPC, BlockManager).
RDD[T] (core/.../rdd/RDD.scala)	Distributed collection abstraction.
Dependency[T] (core/.../Dependency.scala)	Lineage edge between RDDs (OneToOneDependency, ShuffleDependency, ...).
Partitioner (core/.../Partitioner.scala)	Maps keys to partition indices.
DAGScheduler (core/.../scheduler/DAGScheduler.scala)	Builds stages, submits TaskSets, handles failures.
TaskScheduler / TaskSchedulerImpl	Decides which executor runs which task.
SchedulerBackend (core/.../scheduler/SchedulerBackend.scala)	Pluggable cluster-manager interface.
Executor (core/.../executor/Executor.scala)	Runs tasks in worker JVMs.
BlockManager (core/.../storage/BlockManager.scala)	The local block store on each JVM.
ShuffleManager (core/.../shuffle/ShuffleManager.scala)	Pluggable shuffle implementation; default is SortShuffleManager.
MemoryManager (core/.../memory/MemoryManager.scala)	Splits heap between storage and execution; default is UnifiedMemoryManager.
RpcEnv (core/.../rpc/RpcEnv.scala)	Spark's Netty-based RPC abstraction.
MetricsSystem (core/.../metrics/MetricsSystem.scala)	Pluggable metrics with multiple sinks.
LiveListenerBus (core/.../scheduler/LiveListenerBus.scala)	Asynchronous event bus that powers the UI and metrics.

How a job runs

sequenceDiagram
    participant U as User code
    participant SC as SparkContext
    participant DAG as DAGScheduler
    participant TS as TaskSchedulerImpl
    participant SB as SchedulerBackend
    participant EX as Executor

    U->>SC: rdd.collect()
    SC->>DAG: submitJob(rdd, func)
    DAG->>DAG: getOrCreateShuffleMapStage chain
    DAG->>TS: submitTasks(TaskSet)
    TS->>SB: reviveOffers
    SB->>EX: launchTask(TaskDescription)
    EX->>EX: Task.runTask -> ShuffleMapTask / ResultTask
    EX-->>SC: ExecutorBackend.statusUpdate
    SC-->>U: result

The full algorithm includes barrier execution, push-based shuffle, decommissioning, and AQE-driven re-submission. See systems/scheduler.md.

Storage and shuffle

The default shuffle is sort-based. A ShuffleMapTask writes one file per mapper using SortShuffleWriter (core/.../shuffle/sort/) and an index file via IndexShuffleBlockResolver. Reducers fetch via ShuffleBlockFetcherIterator (core/.../storage/ShuffleBlockFetcherIterator.scala), which also drives push-based merging through PushBasedFetchHelper. The external shuffle service that serves blocks after executors die lives in common/network-shuffle/.

For more, see systems/shuffle.md and systems/storage.md.

Memory

UnifiedMemoryManager (core/.../memory/UnifiedMemoryManager.scala) splits the executor heap into a storage pool and an execution pool that can borrow from each other. Off-heap memory is allocated through common/unsafe.

See systems/memory.md.

RPC

The driver and executors talk over Netty using RpcEnv (core/.../rpc/RpcEnv.scala) and its default implementation NettyRpcEnv. Endpoints register with names; messages are serialized with Java or Kryo. The same RPC layer is used by the BlockManager master/slave protocol and the standalone Master/Worker registration.

See systems/rpc.md.

Web UI and status

The driver runs an embedded Jetty server (core/.../ui/) backed by AppStatusStore (core/.../status/AppStatusStore.scala), which listens to the live event stream and records jobs, stages, executors, and SQL plans. The same store is replayed by the History Server from event logs.

The kvstore behind AppStatusStore is common/kvstore, which can use either an in-memory or a RocksDB backend.

Cluster manager integration

Mode	Where
Local	core/.../scheduler/local/LocalSchedulerBackend.scala
Standalone	core/.../deploy/{master,worker,client}/
YARN	resource-managers/yarn/ (separate module)
Kubernetes	resource-managers/kubernetes/ (separate module)

All of them implement SchedulerBackend and register with TaskSchedulerImpl.

Integration points

• SQL. The DataFrame API (sql/core) uses SparkContext to submit jobs and reads/writes blocks via BlockManager.
• MLlib. Uses RDDs and the BarrierTaskContext for distributed training.
• GraphX. Uses RDDs.
• Streaming (Structured). Submits per-batch jobs through SparkContext and uses the state store, which is layered on top of BlockManager.
• Connect server. Embeds a SparkSession and forwards client requests into it.

Entry points for modification

• Adding a new public RDD method: edit core/.../rdd/RDD.scala and add to the Java bridge in core/.../api/java/JavaRDD.scala.
• Adding a scheduler hook: see SparkListener (core/.../scheduler/SparkListener.scala). Listeners run on the live bus and the replay bus.
• Adding a metric sink: implement org.apache.spark.metrics.sink.Sink and register it in metrics.properties. Existing sinks are in core/.../metrics/sink/.
• Adding a deploy plugin: implement SparkPlugin (core/.../api/plugin/SparkPlugin.scala) and configure spark.plugins. Plugins get notified on driver/executor start.