Systems

Cross-cutting subsystems that span the engine. Where modules follows the SBT/Maven module boundaries, this section follows the conceptual subsystems an engineer is likely to work on.

Pages

• scheduler - DAG scheduler, task scheduler, scheduler backends, dynamic allocation, decommissioning.
• storage - block manager, disk store, memory store, broadcast.
• shuffle - sort-based shuffle, push-based shuffle, the external shuffle service.
• memory - unified memory manager, off-heap, MemoryConsumer.
• rpc - Spark's Netty-based RPC layer.
• ui - the Web UI, the History Server, structured logging, metrics.

Each page is grounded in concrete files and points back to the module pages where the code formally lives.