Architecture

Spark is a distributed engine made of a single driver that coordinates a cluster of executors. The driver hosts a SparkContext and a SparkSession. It compiles user programs into a DAG of stages, schedules tasks on executors, and tracks results. Executors run tasks, manage cached blocks, and report status back to the driver.

This page summarizes the major components and how they fit together. Each component links to a deeper dive in the rest of the wiki.

High-level component map

graph TD
    subgraph Driver
        SC[SparkContext]
        SS[SparkSession]
        DS[DAGScheduler]
        TS[TaskScheduler]
        BMM[BlockManagerMaster]
        UI[Web UI]
        SS --> SC
        SC --> DS
        DS --> TS
        SC --> BMM
        SC --> UI
    end

    subgraph ClusterManager[Cluster manager]
        SBE[SchedulerBackend]
        TS -.RPC.- SBE
    end

    subgraph Executor1[Executor]
        E1[Executor]
        BM1[BlockManager]
        SH1[ShuffleManager]
        E1 --- BM1
        E1 --- SH1
    end

    subgraph Executor2[Executor]
        E2[Executor]
        BM2[BlockManager]
        SH2[ShuffleManager]
        E2 --- BM2
        E2 --- SH2
    end

    SBE -->|launch tasks| E1
    SBE -->|launch tasks| E2
    BM1 -.heartbeat.- BMM
    BM2 -.heartbeat.- BMM
    BM1 <-->|fetch shuffle blocks| BM2

Key files for each box:

Component	Source
SparkContext	core/src/main/scala/org/apache/spark/SparkContext.scala
SparkSession	sql/core/src/main/scala/org/apache/spark/sql/classic/SparkSession.scala
DAGScheduler	core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
TaskScheduler impl	core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
BlockManager	core/src/main/scala/org/apache/spark/storage/BlockManager.scala
Executor	core/src/main/scala/org/apache/spark/executor/Executor.scala
Web UI	core/src/main/scala/org/apache/spark/ui/
RPC	core/src/main/scala/org/apache/spark/rpc/

See systems/scheduler.md, systems/storage.md, and systems/rpc.md for details.

Layered view of the codebase

graph TD
    User["User program (Scala / Java / Python / R / SQL)"]
    User --> SQLAPI["sql/api: DataFrame, Dataset, Column"]
    User --> CoreAPI["core: RDD, SparkContext"]
    SQLAPI --> Catalyst["sql/catalyst: parser, analyzer, optimizer"]
    Catalyst --> SQLExec["sql/core/execution: physical plan, AQE"]
    SQLExec --> CoreEngine["core: scheduler + storage + shuffle"]
    CoreAPI --> CoreEngine
    CoreEngine --> Common["common/{network,unsafe,kvstore,sketch}"]
    CoreEngine --> RM["resource-managers/{yarn,kubernetes}"]
    Connect["sql/connect/{server,client,common}"] --> SQLAPI

• API layer. Public types live in sql/api (DataFrame, Dataset, Column, Row, types) and in core (RDD, SparkContext, Partitioner).
• Catalyst. Parses SQL or DataFrame builder calls into a tree of LogicalPlan nodes, resolves them against a catalog, and rewrites the plan with a battery of rules. See modules/catalyst.md.
• SQL execution. Translates logical plans to SparkPlan (physical) and runs them on RDDs. Adaptive Query Execution (AQE) lives here. See modules/sql.md.
• Core engine. RDDs, the DAGScheduler, the TaskScheduler, the BlockManager, and the shuffle subsystem. See modules/core.md and the pages under systems/.
• Common low-level building blocks. Off-heap memory (common/unsafe), Netty-based RPC (common/network-common), the external shuffle service (common/network-shuffle), the local K/V store used by the history server (common/kvstore), and approximate-data sketches (common/sketch).
• Resource managers. Pluggable backends for YARN and Kubernetes live in resource-managers/. Standalone and local modes live in core/src/main/scala/org/apache/spark/deploy/.
• Connect. A gRPC server exposes Spark to thin clients. See modules/connect.md.

Job lifecycle

This is the canonical sequence for an action like df.count() or rdd.collect():

sequenceDiagram
    participant U as User code
    participant SC as SparkContext
    participant DAG as DAGScheduler
    participant TS as TaskSchedulerImpl
    participant SB as SchedulerBackend
    participant E as Executor
    participant BM as BlockManager (peer)

    U->>SC: action (collect / count / save)
    SC->>DAG: submitJob(rdd, func, partitions)
    DAG->>DAG: build stages from RDD lineage
    DAG->>TS: submitTasks(TaskSet) per stage
    TS->>SB: reviveOffers
    SB->>E: launchTask(TaskDescription)
    E->>E: run Task.runTask
    E-->>BM: read/write blocks (cache, shuffle)
    E-->>TS: statusUpdate(taskId, result)
    TS-->>DAG: taskEnded
    DAG-->>SC: jobSucceeded(result)
    SC-->>U: return value

The DAG is built from RDD Dependency objects (core/src/main/scala/org/apache/spark/Dependency.scala) and split at shuffle boundaries into ShuffleMapStage and ResultStage. The full algorithm lives in DAGScheduler.scala (~167 KB, the single largest file in core).

Memory and shuffle

Each executor partitions its memory between storage (cached blocks) and execution (sort, hash, aggregation, shuffle buffers) using a unified pool defined in core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala. Off-heap is supported through common/unsafe. Shuffle output is written by ShuffleWriter implementations in core/src/main/scala/org/apache/spark/shuffle/ and fetched by ShuffleBlockFetcherIterator (core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala).

Push-based shuffle and the external shuffle service live in common/network-shuffle. See systems/shuffle.md.

Spark SQL pipeline

graph LR
    A["SQL text or DataFrame builder"] --> B[ParserInterface]
    B --> C[Unresolved LogicalPlan]
    C --> D[Analyzer]
    D --> E[Resolved LogicalPlan]
    E --> F[Optimizer]
    F --> G[Optimized LogicalPlan]
    G --> H[SparkPlanner / Strategies]
    H --> I[SparkPlan]
    I --> J["AQE / WholeStageCodegen"]
    J --> K[RDD of InternalRow]
    K --> L[DAGScheduler]

Each stage is a tree-rewrite step over TreeNode (sql/catalyst/.../trees/TreeNode.scala). Code generation produces JVM bytecode at runtime via Janino (the Tungsten effort). AQE (sql/core/.../execution/adaptive/) re-optimizes plans after each shuffle stage based on real runtime statistics. See modules/catalyst.md and modules/sql.md.

Spark Connect

Spark Connect adds a gRPC server in front of the Spark driver so that lightweight clients (Python, Go, Swift, Rust, JVM) can issue queries without embedding the JVM. The protocol lives in sql/connect/common/src/main/protobuf/spark/connect/ (relations, expressions, commands, ml, pipelines, ...). The server side is in sql/connect/server/. See modules/connect.md.

Cluster managers and deploy modes

Spark supports four deploy modes, all of which plug into the same SchedulerBackend interface:

Mode	Source	Notes
Local	core/src/main/scala/org/apache/spark/scheduler/local/	Single JVM, useful for tests and notebooks
Standalone	core/src/main/scala/org/apache/spark/deploy/{master,worker}/	Built-in Master/Worker daemons
YARN	resource-managers/yarn/	Hadoop integration
Kubernetes	resource-managers/kubernetes/	Pod-based executors via the K8s API

Logging, metrics, and the UI

• Logging is centralized in common/utils/.../Logging.scala. Spark ships a structured-logging layer (common/utils/.../logging/) that emits MDC-tagged JSON when enabled.
• A pluggable metrics system (core/src/main/scala/org/apache/spark/metrics/) writes to JMX, Prometheus, Graphite, console, CSV, and Slf4j sinks.
• The Web UI is served by Jetty (core/src/main/scala/org/apache/spark/ui/) and is replayed offline by the History Server (core/src/main/scala/org/apache/spark/deploy/history/).