sql

The Spark SQL family is the largest piece of the codebase. It comprises the SQL parser, the Catalyst optimizer, the DataFrame and Dataset APIs, the physical execution layer (including AQE and code generation), the Hive integration, the Connect server, and a growing set of data-source connectors.

This page covers sql/api and sql/core. Catalyst has its own page at catalyst.md, Hive integration at hive.md, Connect at connect.md, and Structured Streaming at streaming.md.

Purpose

• Parse SQL and DataFrame programs into Catalyst trees.
• Apply analysis, optimization, and code generation.
• Run the resulting physical plan on Spark Core RDDs.
• Provide a stable public API surface (DataFrame, Dataset, Column, Row, types) shared by both the classic Spark Session and the Spark Connect client.

Directory layout

sql/
  api/                                 - shared public API (DataType, Row, Column types)
    src/main/scala/org/apache/spark/sql/
      types/                           - DataTypes, schema
      ...
  catalyst/                            - tree rewrites, see modules/catalyst.md
  core/
    src/main/scala/org/apache/spark/sql/
      classic/SparkSession.scala       - the driver-side SQL entry point
      execution/                       - physical operators, codegen, IO, AQE, streaming
      execution/datasources/v1/        - V1 (legacy) data sources
      execution/datasources/v2/        - V2 (current) data source framework
      execution/adaptive/              - Adaptive Query Execution
      execution/streaming/             - Structured Streaming runtime
      execution/joins/                 - hash, broadcast, sort-merge, shuffled hash joins
      execution/window/                - window operators
      execution/aggregate/             - hash-based and sort-based aggregation
      execution/python/                - Python UDF execution
      execution/columnar/              - parquet vectorized reader, in-memory columnar caching
      execution/exchange/              - shuffles and broadcasts
      execution/datasources/jdbc/      - JDBC reader/writer
      jdbc/                            - JDBC dialects
      avro/                            - native Avro support (graduated from connector/)
      sources/                         - DataSource registration
      streaming/                       - DataStreamReader/Writer DSL
      scripting/                       - SQL scripting (PL/SQL-style control flow)
      util/                            - utility code
      classic/                         - the classic SparkSession implementation
      internal/                        - SQLConf, SessionState, BaseSessionStateBuilder
  hive/                                - see modules/hive.md
  hive-thriftserver/                   - JDBC server, see modules/hive.md
  connect/                             - see modules/connect.md
  pipelines/                           - declarative pipelines (Spark 4 era)

Key abstractions

Type / file	What it is
SparkSession (sql/core/.../sql/classic/SparkSession.scala)	Top-level entry point; owns SessionState.
Dataset[T] / DataFrame (sql/core/.../Dataset.scala)	Strongly-typed and untyped row collections.
LogicalPlan (sql/catalyst/.../plans/logical/LogicalPlan.scala)	The pre-execution tree.
Optimizer (sql/catalyst/.../optimizer/Optimizer.scala)	Rule-batch logical-plan rewriter.
SparkPlanner (sql/core/.../execution/SparkStrategies.scala)	Picks physical operators for a logical plan.
SparkPlan (sql/core/.../execution/SparkPlan.scala)	Physical operator base class.
WholeStageCodegenExec	Compiles a chain of operators into a single Java method.
AdaptiveSparkPlanExec (sql/core/.../execution/adaptive/)	Re-optimizes the plan after each shuffle stage.
DataSource (sql/core/.../execution/datasources/DataSource.scala)	The V1 data source resolver and registry.
DataSourceV2 (sql/catalyst/.../connector/)	The V2 connector SPI.
SQLConf (sql/core/.../internal/SQLConf.scala)	Per-session SQL configuration.
Catalog API (sql/api/.../catalog/)	User-facing catalog (databases, tables, functions).

SQL pipeline

graph LR
    A["SQL string or DataFrame builder"] --> B[SqlParser / DataFrame DSL]
    B --> C["Unresolved LogicalPlan"]
    C --> D[Analyzer]
    D --> E["Resolved LogicalPlan"]
    E --> F[Optimizer]
    F --> G["Optimized LogicalPlan"]
    G --> H[SparkPlanner / Strategies]
    H --> I[SparkPlan]
    I --> J{"AQE enabled?"}
    J -- yes --> K[AdaptiveSparkPlanExec]
    J -- no --> L[WholeStageCodegen]
    K --> L
    L --> M["RDD[InternalRow]"]
    M --> N[DAGScheduler]

For the rule-by-rule details of stages C->E and E->G, see catalyst.md.

Adaptive Query Execution

sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/ contains AQE. It runs the plan one stage at a time, gathers shuffle statistics (MapOutputStatistics, core/.../MapOutputStatistics.scala), and re-optimizes the unfinished suffix of the plan. Capabilities include:

• Coalescing post-shuffle partitions.
• Switching join strategies (e.g., sort-merge to broadcast) when the build side is small.
• Optimizing skewed joins by splitting the skewed partitions.

AQE is enabled by default in Spark 3.x and beyond.

Code generation (Tungsten)

WholeStageCodegenExec (sql/core/.../execution/WholeStageCodegenExec.scala) walks a chain of supported operators and stitches them into a single Java class compiled by Janino at runtime. Per-expression codegen is handled by Expression.doGenCode in sql/catalyst/.../expressions/codegen/.

The unsafe row format that flows through these stages lives in common/unsafe.

Data sources

Interface	Where
V1 DataSource	sql/core/.../execution/datasources/v1/
V2 Table, Scan, Write	sql/catalyst/.../connector/ and sql/core/.../execution/datasources/v2/
Parquet (vectorized)	sql/core/.../execution/datasources/parquet/
ORC	sql/core/.../execution/datasources/orc/
JSON	sql/core/.../execution/datasources/json/
CSV	sql/core/.../execution/datasources/csv/
Avro	sql/core/.../avro/
JDBC	sql/core/.../execution/datasources/jdbc/
Hive	sql/hive/
Kafka	connector/kafka-0-10-sql/
Protobuf	connector/protobuf/

The pluggable surface is V2; V1 remains for legacy compatibility. New built-in data sources land as V2 implementations.

SQL scripting

sql/core/src/main/scala/org/apache/spark/sql/scripting/ and the related sql/catalyst/.../trees/SqlScriptingContextManager.scala implement PL/SQL-style control flow over SQL statements (variables, IF/CASE, loops, exceptions). This is a Spark 4 feature.

Declarative pipelines

sql/pipelines/ (and the Python mirror in python/pyspark/pipelines/) is a Spark 4 feature for declarative ETL pipelines. The Connect protocol carries pipeline definitions in sql/connect/common/.../protobuf/spark/connect/pipelines.proto.

Integration points

• Submits jobs through SparkContext. Most physical operators end up calling mapPartitionsInternal on an RDD.
• Uses BlockManager for in-memory caching (Dataset.cache()), shuffle output, and broadcast joins.
• The Connect server (sql/connect/server/) translates protobuf into Catalyst commands and runs them in an embedded SparkSession.
• Hive integration is opt-in via enableHiveSupport() and pulls in sql/hive.

Entry points for modification

• Adding a SQL expression: extend Expression (sql/catalyst/.../expressions/Expression.scala), register it in FunctionRegistry, and add a golden-file test.
• Adding a join strategy: implement a new SparkPlan and a Strategy in sql/core/.../execution/SparkStrategies.scala.
• Adding a data source: implement the V2 Table, Scan, and (optionally) Write interfaces and register a DataSourceRegister in META-INF/services/.
• Adding a SQL config: declare it in SQLConf with a version annotation.