resource-managers

The resource-managers/ directory holds out-of-core cluster-manager integrations. The in-tree built-in modes (local, standalone) live in core/; YARN and Kubernetes are big enough to warrant their own modules.

Modes overview

Mode	Source	Notes
Local	core/.../scheduler/local/	Single JVM, threads as executors. Useful for tests.
Standalone	core/.../deploy/{master,worker,client}/	Spark's own daemons. Two-process cluster manager.
YARN	resource-managers/yarn/	Hadoop's cluster manager.
Kubernetes	resource-managers/kubernetes/	K8s pods as executors.
Mesos	(removed in Spark 3.5)	No longer in the tree.

All four cluster-manager backends implement SchedulerBackend (core/.../scheduler/SchedulerBackend.scala) and CoarseGrainedSchedulerBackend.

YARN

resource-managers/yarn/
  src/main/scala/org/apache/spark/deploy/yarn/
    ApplicationMaster.scala       - the AM running inside YARN
    Client.scala                  - the spark-submit-side YARN client
    YarnAllocator.scala           - asks YARN for executor containers
    YarnRMClient.scala
    YarnSchedulerBackend.scala
    security/                      - Kerberos token providers
  src/main/scala/org/apache/spark/scheduler/cluster/
    YarnClusterManager.scala
    YarnClusterScheduler.scala
    YarnClusterSchedulerBackend.scala
    YarnClientSchedulerBackend.scala

Key flow:

1. Client.scala packages and submits the app to the YARN ResourceManager.
2. The RM launches ApplicationMaster.scala in a container.
3. The AM negotiates with YarnAllocator for executor containers.
4. Each container starts a CoarseGrainedExecutorBackend (defined in core/) that registers back with the driver.
5. Hadoop delegation tokens are managed by token providers under resource-managers/yarn/.../security/ and renewed periodically.

YARN is built with -Pyarn. There are no YARN-specific Python dependencies; this is JVM-only.

Kubernetes

resource-managers/kubernetes/
  core/                           - The K8s scheduler backend
    src/main/scala/org/apache/spark/deploy/k8s/
      Config.scala
      KubernetesUtils.scala
      submit/                     - spark-submit-side K8s client (KubernetesClientApplication)
      features/                    - per-pod features (volumes, env, secrets, ...)
      scheduler/cluster/k8s/      - KubernetesClusterSchedulerBackend, ExecutorPodsAllocator
  integration-tests/              - end-to-end tests on Minikube / kind

Key concepts:

• The driver runs as either a K8s pod ("cluster mode") or a local process ("client mode").
• ExecutorPodsAllocator maintains the desired executor count by creating pods.
• KubernetesFeatureConfigSteps contribute pieces of the pod spec (MountVolumesFeatureStep, EnvSecretsFeatureStep, LocalDirsFeatureStep, ...).
• Persistent volume claims can be reused across executor restarts via OnDemandVolumes.
• A K8s scheduler plugin can dynamically resize executors without restarting them.

Build with -Pkubernetes. The integration-tests subproject runs on Minikube or kind and is not part of the standard CI matrix.

Standalone (in core)

core/.../deploy/master/ is the Master daemon (an Apache 2 cluster manager). It tracks registered Workers and assigns executors to applications when they connect via StandaloneAppClient. Workers run on each cluster node and launch executors via org.apache.spark.deploy.worker.ExecutorRunner.

The standalone deploy mode is the simplest cluster setup: no Hadoop, no Kubernetes, just two Java processes per node.

Dynamic allocation

Cross-cutting work that all cluster-manager backends share:

• ExecutorAllocationManager (core/.../ExecutorAllocationManager.scala) decides when to request and release executors based on pending tasks and idleness.
• ExecutorAllocationClient is the interface backends implement (the YARN, K8s, and standalone backends all do).
• Decommissioning - graceful shutdown that migrates blocks - is implemented by BlockManagerDecommissioner (core/.../storage/BlockManagerDecommissioner.scala) and FallbackStorage (core/.../storage/FallbackStorage.scala).

Integration points

• All backends register with TaskSchedulerImpl and use CoarseGrainedSchedulerBackend as the base class for the network protocol with executors.
• Both YARN and K8s back-ends interact with BlockManager to migrate shuffle and cache blocks during graceful decommissioning.
• YARN reads Hadoop config via core/.../deploy/SparkHadoopUtil.scala. Kubernetes uses the fabric8 Kubernetes client (io.fabric8:kubernetes-client).

Entry points for modification

• Add a feature to the K8s pod spec: implement a new KubernetesFeatureConfigStep and register it in KubernetesDriverBuilder/KubernetesExecutorBuilder.
• Add a new YARN credential provider: implement ServiceCredentialProvider and register via META-INF/services/.
• Tweak dynamic allocation behavior: most knobs are in core/.../internal/config/DynamicAllocation.scala. Look at core/.../ExecutorAllocationManager.scala for the decision logic.