Glossary

A short reference for project-specific terms that appear repeatedly in the Spark codebase. When a term has a canonical implementation, the entry links to the file.

Engine concepts

• RDD (Resilient Distributed Dataset). The original distributed collection abstraction. Defined in core/src/main/scala/org/apache/spark/rdd/RDD.scala. Every higher-level API ultimately compiles down to an RDD DAG.
• DataFrame / Dataset. Strongly-typed (Dataset) and untyped (DataFrame) row-oriented abstractions backed by Catalyst. APIs are in sql/api and sql/core.
• Action vs transformation. Transformations are lazy (e.g., map, filter). Actions (e.g., count, collect, save) trigger the DAGScheduler.
• Partition. A unit of parallelism. Each task processes one partition.
• Stage. A set of tasks with no internal shuffle. Boundaries are shuffle dependencies. See core/src/main/scala/org/apache/spark/scheduler/Stage.scala.
• Job. A user-visible action. Decomposed into one or more stages.
• Task. A single unit of work, run on one partition by one executor. core/src/main/scala/org/apache/spark/scheduler/Task.scala.
• Lineage. The dependency graph that lets RDD partitions be recomputed on failure.
• Barrier execution. A mode where all tasks of a stage start together, used for ML/MPI-style workloads. See core/.../BarrierTaskContext.scala.

Driver and executors

• Driver. The JVM that hosts SparkContext/SparkSession and coordinates execution.
• Executor. A long-running JVM that executes tasks and caches blocks. core/.../executor/Executor.scala.
• SparkContext. Low-level entry point; owns the scheduler, RPC env, BlockManager, and UI. core/.../SparkContext.scala.
• SparkSession. SQL-aware entry point; wraps SparkContext and exposes the DataFrame API. sql/core/.../sql/classic/SparkSession.scala.
• SchedulerBackend. The pluggable interface a cluster manager implements to acquire and release executor resources. core/.../scheduler/SchedulerBackend.scala.

SQL and Catalyst

• Catalyst. Spark's tree-rewrite framework. Logical plans, expressions, and rules live in sql/catalyst/.
• TreeNode. The base of every plan/expression node. sql/catalyst/.../trees/TreeNode.scala.
• Analyzer. Resolves identifiers and types in an unresolved plan. sql/catalyst/.../analysis/Analyzer.scala.
• Optimizer. Rule-based logical-plan rewriter. sql/catalyst/.../optimizer/Optimizer.scala.
• SparkPlanner / Strategy. Picks physical operators for an optimized logical plan.
• SparkPlan. A physical operator that produces an RDD[InternalRow].
• AQE (Adaptive Query Execution). Re-optimizes the plan after each shuffle stage using runtime statistics. sql/core/.../execution/adaptive/.
• DSv2. The DataSource V2 connector API. sql/catalyst/.../connector/.
• Tungsten. The off-heap, code-generated execution layer. The unsafe row format lives in common/unsafe.

Storage and shuffle

• BlockManager. Per-JVM service that stores and serves blocks (cached RDDs, shuffle output, broadcast variables). core/.../storage/BlockManager.scala.
• BlockId. The 6+ subclasses that name blocks: RDDBlockId, ShuffleBlockId, BroadcastBlockId, etc.
• ShuffleManager. Strategy for writing and reading shuffle data. The default is SortShuffleManager. See core/.../shuffle/.
• External shuffle service. A long-running process on each node that serves shuffle blocks even after executors die. common/network-shuffle.
• Push-based shuffle. Shuffle blocks are pushed to mergers and assembled before the reducer pulls them. core/.../storage/PushBasedFetchHelper.scala.
• Fallback storage. Decommissioning executors copy their shuffle output to a configured remote location. core/.../storage/FallbackStorage.scala.

Memory

• UnifiedMemoryManager. Splits each executor heap into storage and execution pools, with bidirectional borrowing. core/.../memory/UnifiedMemoryManager.scala.
• MemoryConsumer. Anything that asks the manager for execution memory (sorters, hash maps, shuffle writers). common/unsafe.
• Off-heap memory. Allocated via sun.misc.Unsafe through common/unsafe. Disabled by default; enabled with spark.memory.offHeap.enabled.

Resource management and deploy

• Standalone mode. Spark's own Master/Worker daemons in core/.../deploy/.
• YARN. Hadoop's cluster manager. resource-managers/yarn/.
• Kubernetes. K8s pods as executors. resource-managers/kubernetes/.
• Dynamic allocation. Adds and removes executors based on pending tasks. core/.../ExecutorAllocationManager.scala.
• Decommissioning. Graceful shutdown of an executor that migrates its blocks before exit.
• Resource profile. A user-defined CPU/memory/GPU shape for a task or stage. core/.../resource/.

Streaming

• Structured Streaming. The DataFrame-based streaming API. sql/core/.../execution/streaming/.
• MicroBatchExecution / ContinuousExecution. The two execution modes.
• State store. Pluggable durable storage for stateful operators (HDFSBackedStateStoreProvider, RocksDBStateStoreProvider).
• Watermark. The latest event time used to decide when to emit late results.
• Output mode. Append, Update, Complete. See the streaming guide.
• DStream. The legacy streaming API (streaming/). Mostly maintained, not extended.

Spark Connect

• Connect. A gRPC layer that lets thin clients drive Spark. sql/connect/{client,common,server}.
• Relation / Plan / Command. The protobuf message types that describe a query. sql/connect/common/src/main/protobuf/spark/connect/.
• Artifact. A jar, file, or class transmitted from a Connect client to the server. sql/connect/.../artifact/ and core/.../JobArtifactSet.scala.

Tooling and process

• JIRA / SPARK-xxxxx. Every change is tied to a ticket on the Apache JIRA. PR titles must begin with [SPARK-xxxxx][COMPONENT].
• PR component tag. The uppercase last word of the JIRA component name, e.g., [CORE], [SQL], [STREAMING], [INFRA], [K8S], [CONNECT].
• Golden file tests. Snapshot-style SQL tests in sql/core/src/test/resources/sql-tests/. Driven by SQLQueryTestSuite. Never edited by hand.
• Codegen / WholeStageCodegen. The compilation of operator trees into a single Java method via Janino. sql/catalyst/.../expressions/codegen/ and sql/core/.../execution/WholeStageCodegenExec.scala.

Acronyms

• AQE - Adaptive Query Execution
• CBO - Cost-Based Optimizer
• DAG - Directed Acyclic Graph
• DSv1 / DSv2 - DataSource V1 / V2
• JDBC / ODBC - Java/Open Database Connectivity (Hive ThriftServer)
• K8s - Kubernetes
• PVC - Persistent Volume Claim (K8s storage)
• RPC - Remote Procedure Call
• TGT - Ticket-Granting Ticket (Kerberos)
• UDF / UDAF / UDTF - User-Defined Function / Aggregate / Table function