Memory

Each Spark JVM splits its heap (and optionally off-heap) into two pools managed by the Unified Memory Manager. Code outside the manager that allocates large amounts of memory must register as a MemoryConsumer so it can be told to spill when execution memory is under pressure.

Layout

graph TD
    Heap["JVM heap"] --> Reserved["Reserved (300 MB by default)"]
    Heap --> User["User memory (40% by default)"]
    Heap --> SparkMem["Spark memory (60% by default)"]
    SparkMem --> StoragePool["Storage pool"]
    SparkMem --> ExecPool["Execution pool"]
    StoragePool -. borrow .- ExecPool

    OffHeap["Off-heap (sun.misc.Unsafe / MemorySegment)"]
    OffHeap --> OffStorage["Off-heap storage pool"]
    OffHeap --> OffExec["Off-heap execution pool"]

Pool	What lives there
Reserved	Hard-coded headroom; never used by Spark.
User memory	User code, third-party libs, allocations not registered with Spark.
Storage	Cached RDD/DataFrame blocks, broadcast variables.
Execution	Sort buffers, hash maps for aggregations and joins, shuffle write buffers.

The two Spark pools can borrow from each other. Storage borrowing is bounded; execution can fully reclaim borrowed storage memory by evicting cached blocks.

Key files

Type	What it is
MemoryManager (core/.../memory/MemoryManager.scala)	Abstract base.
UnifiedMemoryManager (core/.../memory/UnifiedMemoryManager.scala)	Default. Implements the borrow rules.
StorageMemoryPool / ExecutionMemoryPool	The two pools.
MemoryStore (core/.../storage/memory/MemoryStore.scala)	The storage-pool consumer; stores BlockId-keyed blocks.
MemoryConsumer (common/unsafe/.../memory/MemoryConsumer.java)	The execution-pool consumer interface (sorters, hash maps).
TaskMemoryManager (common/unsafe/.../memory/TaskMemoryManager.java)	Per-task allocator that requests pages from the manager.
Platform (common/unsafe/.../Platform.java)	Off-heap allocation entry point.

Acquire / release flow

sequenceDiagram
    participant C as MemoryConsumer (e.g., UnsafeShuffleWriter)
    participant TMM as TaskMemoryManager
    participant UMM as UnifiedMemoryManager
    participant MS as MemoryStore

    C->>TMM: acquireExecutionMemory(numBytes)
    TMM->>UMM: acquireExecutionMemory
    alt Execution pool full
        UMM->>MS: evict storage blocks
        MS-->>UMM: bytes freed
    end
    UMM-->>TMM: granted
    TMM-->>C: page
    Note over C,TMM: Consumer writes records into the page
    C->>TMM: releaseExecutionMemory(numBytes)
    TMM->>UMM: releaseExecutionMemory

When the manager cannot fully grant a request, it asks other consumers to spill. Each MemoryConsumer implements spill(size: Long, trigger: MemoryConsumer). Sorters spill to disk; hash maps either spill or fall back to a sort-based path.

Off-heap

Set spark.memory.offHeap.enabled=true and spark.memory.offHeap.size=N. Off-heap is a separate pool with its own storage and execution sub-pools. Allocations go through Platform.allocateMemory (which uses Unsafe.allocateMemory on Java 8-16 and a MemorySegment-based path on Java 17+).

The unsafe row format flowing through whole-stage codegen is most efficient when off-heap is enabled because it avoids a copy when serializing to disk or to the network.

Tungsten and the unsafe row

The Tungsten effort introduced a binary row format (UnsafeRow) that is laid out in a single contiguous byte array (or off-heap region). Operators that have a "tungsten" code path (sort-merge join, aggregate, sort) operate on UnsafeRow directly without deserialization. The format and arithmetic helpers live in common/unsafe/.../UnsafeRow.java, UnsafeArrayData.java, UnsafeMapData.java.

Common knobs

• spark.executor.memory - JVM heap.
• spark.executor.memoryOverhead - extra non-heap memory budget (Netty, off-heap, native libs).
• spark.memory.fraction - what fraction of the JVM heap is "Spark memory" (default 0.6).
• spark.memory.storageFraction - the storage pool's share of Spark memory (default 0.5).
• spark.memory.offHeap.enabled, spark.memory.offHeap.size.

Integration points

• Used by every operator that allocates non-trivial memory.
• The BlockManager is the storage-pool consumer.
• BroadcastManager (core/.../broadcast/) interacts with the storage pool through BlockManager.
• AQE collects spill-statistics events from MemoryConsumer for re-optimization decisions.

Entry points for modification

• Add a memory consumer: extend MemoryConsumer, allocate pages from TaskMemoryManager, implement spill. Examples: UnsafeExternalSorter, BytesToBytesMap.
• Tune the manager: most knobs are in core/.../internal/config/package.scala under the memory section.
• Switch off-heap allocator: Platform selects the right path for the running JVM automatically; advanced users can plug in a custom allocator via Platform.allocateMemory overrides.