Debugging

Practical tips for narrowing down problems in the Spark codebase.

Read the logs

Spark's own log calls go through Logging (Scala) and Python's logging module. The Logging trait is in common/utils/src/main/scala/org/apache/spark/internal/Logging.scala. Spark also emits structured (MDC-tagged) JSON logs when spark.log.structuredLogging.enabled=true; the structured layer lives in common/utils/src/main/scala/org/apache/spark/internal/logging/.

Helpful log levels to crank up while debugging:

spark.executor.extraJavaOptions=-Dlog4j2.configurationFile=...
spark.driver.extraJavaOptions=-Dlog4j2.configurationFile=...

A custom log4j2.properties lives in conf/log4j2.properties.template. Copy and edit it. Frequently useful loggers:

• org.apache.spark.scheduler.DAGScheduler - stage construction, retries.
• org.apache.spark.scheduler.TaskSetManager - per-task placement and locality decisions.
• org.apache.spark.storage.BlockManager - block fetches, evictions, decommissioning.
• org.apache.spark.sql.execution.adaptive - AQE plan re-optimization.
• org.apache.spark.sql.connect - Connect protocol traces.

Use the Web UI

The driver Web UI (default http://<driver>:4040) exposes:

• Jobs - DAG visualization, stage retries, GC time.
• Stages - per-task metrics: shuffle read/write, spill, GC.
• Storage - cached RDDs and Datasets, with hit ratios.
• Executors - live JVM stats per executor.
• SQL - the query plan, including AQE re-optimizations.
• Streaming - micro-batch progress for Structured Streaming jobs.

Source: core/src/main/scala/org/apache/spark/ui/. The status data is also dumped to event logs (see below) and replayed by the History Server.

Event log replay

When spark.eventLog.enabled=true, the driver writes a JSON event log to spark.eventLog.dir. The History Server (./sbin/start-history-server.sh) replays these logs and serves the same UI for completed apps. Source for the writer is core/.../scheduler/EventLoggingListener.scala; the replayer is ReplayListenerBus.scala.

The History Server can use a RocksDB-backed kvstore (common/kvstore/) for very large event logs.

Heap dumps and JFR

Executors and the driver are JVMs, so the standard tools work:

jcmd <pid> GC.heap_dump /tmp/dump.hprof
jcmd <pid> JFR.start name=spark filename=/tmp/spark.jfr settings=profile duration=60s

Pass JVM options through spark.driver.extraJavaOptions / spark.executor.extraJavaOptions.

Python debugging

• python/pyspark/worker.py is the entry point of the Python worker. Errors from UDFs are re-raised on the driver after being pickled.
• The pyspark.errors module (python/pyspark/errors/) wraps JVM exceptions into Pythonic error classes.
• VS Code breakpoint support is wired up in python/run-with-vscode-breakpoints and python/conf_vscode/.
• viztracer profiling lives in python/run-with-viztracer and python/conf_viztracer/.

Common error patterns

• OutOfMemoryError: Java heap space in shuffle reads. Increase spark.executor.memory or shrink partitions. Check the Storage tab for cache pressure.
• OutOfMemoryError: Direct buffer memory. Netty (RPC and shuffle) uses off-heap. Bump spark.executor.memoryOverhead or -XX:MaxDirectMemorySize.
• FetchFailedException during shuffle. Indicates an executor died or the external shuffle service is unreachable. See core/.../scheduler/DAGScheduler.scala for the retry logic.
• AnalysisException: cannot resolve .... Catalyst's analyzer cannot resolve an identifier. Run df.explain(true) to see the unresolved plan.
• SparkException: Task not serializable. A closure captures a non-serializable object. Use org.apache.spark.util.ClosureCleaner (already invoked automatically) and check the enclosing class.

Reproducing CI failures

For PR failures, fetch the test annotations first - do not download the whole job log:

gh api repos/<owner>/spark/check-runs/<id>/annotations

For a flaky test, the standard practice is to reproduce locally with the same seed and -DfailFast=false. Many flaky tests are tracked on JIRA with a flaky-test label.

Useful entry points

Symptom	Where to look
Slow query	SQL UI, EXPLAIN, AQE under sql/core/.../execution/adaptive/
Skewed shuffle	Stages tab, spark.sql.adaptive.skewJoin
Lost executor	core/.../HeartbeatReceiver.scala, RM logs
Hanging job	DAGScheduler logs, thread dumps via Web UI Executors page
State store corruption (streaming)	sql/core/.../execution/streaming/state/