UI, logging, and metrics

This page covers the operational surface that engineers interact with at runtime: the Web UI, the History Server, structured logging, and the metrics system.

Web UI

The Spark Web UI is an embedded Jetty server reached on port 4040 of the driver (default). The code lives in core/src/main/scala/org/apache/spark/ui/.

core/.../ui/
  SparkUI.scala            - top-level builder; assembles tabs and the server
  WebUI.scala              - generic tabbed-page abstraction
  WebUIPage.scala          - one renderable page
  jobs/, stages/, storage/, executors/, environment/  - per-tab pages
  exec/, tools/            - additional pages
  static/                  - JS/CSS/images served alongside HTML
  filters/, security/      - auth filters, ACLs

The data backing each page comes from AppStatusStore (core/.../status/AppStatusStore.scala), which reads from the kvstore in common/kvstore/. AppStatusListener (core/.../status/AppStatusListener.scala) listens to the LiveListenerBus and maintains the kvstore in real time.

The SQL tab is a separate listener: SQLAppStatusListener (sql/core/.../execution/ui/SQLAppStatusListener.scala).

The Streaming tab is StreamingQueryStatusListener (sql/core/.../execution/streaming/ui/StreamingQueryStatusListener.scala).

History Server

When spark.eventLog.enabled=true, the driver writes a JSON event log to spark.eventLog.dir via EventLoggingListener (core/.../scheduler/EventLoggingListener.scala). The History Server (core/.../deploy/history/HistoryServer.scala and FsHistoryProvider.scala) reads those logs and replays them through ReplayListenerBus (core/.../scheduler/ReplayListenerBus.scala) into an AppStatusStore per app, then serves the same UI pages.

For very large event logs the History Server can use a RocksDB-backed kvstore; the choice is controlled by spark.history.store.path and the kvstore implementation in common/kvstore/.

Structured logging

Spark 3.5+ ships a structured-logging layer that emits MDC-tagged JSON when enabled. Source: common/utils/src/main/scala/org/apache/spark/internal/logging/.

• MDC - the message-context dictionary.
• LogEntry and LogKey - typed keys for context fields (operator, partition, stage id, job id, ...).
• Logging (common/utils/.../Logging.scala) - the trait classes mix in. It exposes logInfo, logWarning, logError overloads that accept MDC arguments.

Enabling: spark.log.structuredLogging.enabled=true. The output format is JSON with stable field names that downstream tooling (Splunk, Elasticsearch) can index.

Metrics

MetricsSystem (core/.../metrics/MetricsSystem.scala) is the pluggable metrics surface. It uses Dropwizard Metrics under the hood.

Sources (where metrics come from):

• BlockManagerSource
• DAGSchedulerSource
• ExecutorSource
• JvmSource
• ApplicationMasterSource (YARN)
• Per-streaming-query sources via StreamingQueryListener

Sinks (where metrics go):

• JmxSink, ConsoleSink, CsvSink, Slf4jSink
• GraphiteSink
• PrometheusServlet (built in)
• GangliaSink (in connector/spark-ganglia-lgpl/, separate artifact for license reasons)

Configuration is loaded from metrics.properties (or via spark.metrics.conf.*).

Event log to UI flow

graph LR
    Driver[Driver SparkContext] -->|SparkListenerEvent| Bus[LiveListenerBus]
    Bus --> ASL[AppStatusListener]
    Bus --> ELL[EventLoggingListener]
    ASL --> KV[AppStatusStore -> kvstore]
    KV --> UI[Web UI pages]
    ELL --> Log[Event log file]
    Log --> HS[History Server]
    HS --> RB[ReplayListenerBus]
    RB --> ASL2[AppStatusListener replay]
    ASL2 --> KV2[AppStatusStore]
    KV2 --> UI2[Replayed UI pages]

ACLs and authorization

spark.acls.enable flips on the UI ACL filter. ModifyAclsFilter and ViewAclsFilter restrict who can see and operate on the UI. Cluster managers contribute the user identity: YARN via the principal, K8s via the service account or OIDC.

Web UI as a debugging tool

For typical jobs, the most useful tabs are:

• SQL - shows the optimized and AQE-final plans, including operator-level metrics (rows, time, shuffle bytes).
• Stages - per-task timing, GC time, shuffle read/write, locality, peak execution memory.
• Executors - alive vs dead executors, peak memory, JVM thread dump links.
• Storage - cached RDDs/Datasets and their hit/miss ratios.

For Structured Streaming, the Structured Streaming tab shows micro-batch progress, input/output rates, and watermarks.

Integration points

• The same listener interface (SparkListener) is the public extension point for plugins that record their own UI tabs (e.g., the JDBC server tab in sql/hive-thriftserver/.../ui/).
• Structured logging fields show up in the JSON event log, so the History Server can filter and search by MDC keys.
• Metrics are independent of the listener bus and can be enabled even when event logging is off.

Entry points for modification

• Add a tab: implement WebUITab and WebUIPage, register through a SparkListener.onApplicationStart hook.
• Add a metric source: implement org.apache.spark.metrics.source.Source and register it via MetricsSystem.registerSource.
• Add a metric sink: implement org.apache.spark.metrics.sink.Sink and configure it in metrics.properties.
• Add a structured log key: define a new LogKey enum value in common/utils/.../logging/LogKey.scala and use MDC(LogKey.KEY -> value) at the call site.