hive

Two source trees handle Hive integration:

• sql/hive/ - the Hive-backed catalog, Hive UDF/UDAF/UDTF support, Hive SerDes, Hive-compatible reading and writing.
• sql/hive-thriftserver/ - a JDBC/ODBC server (bin/spark-sql and the Thrift server) that exposes Spark over the Hive Thrift protocol.

Hive support is opt-in: build with -Phive (and -Phive-thriftserver) and call SparkSession.builder().enableHiveSupport().

Purpose

• Talk to a Hive metastore for catalog operations (databases, tables, partitions, columns).
• Read and write tables stored in formats Hive supports (Avro, ORC, Parquet, RCFile, custom SerDes).
• Allow Hive UDFs/UDAFs/UDTFs to be registered and called.
• Provide a JDBC/ODBC server that speaks HiveServer2's Thrift protocol so existing tooling (Tableau, DBeaver, Hive's beeline) can connect.

Directory layout

sql/hive/
  src/main/scala/org/apache/spark/sql/hive/
    HiveExternalCatalog.scala       - implementation of the ExternalCatalog interface
    HiveSessionStateBuilder.scala   - wires Hive resources into a SparkSession
    HiveSessionCatalog.scala        - SessionCatalog with Hive function support
    client/                          - HiveClient and shims for old Hive metastore wire versions
    execution/                       - Hive-specific physical operators (InsertIntoHive, ...)
    orc/                             - Hive's native ORC reader/writer
    HiveStrategies.scala             - injects Hive operators into the SparkPlanner
    HiveUtils.scala                  - configuration, classpath, isolated classloader
sql/hive-thriftserver/
  src/main/scala/org/apache/spark/sql/hive/thriftserver/
    HiveThriftServer2.scala          - main entry point for the JDBC server
    SparkSQLDriver.scala             - bridges Hive's Driver interface to SparkSession
    SparkSQLCLIDriver.scala          - the spark-sql CLI driver
    server/                           - SessionManager, OperationManager, ...
    ui/                              - the dedicated JDBC server tab in the Web UI

Key abstractions

Type	What it does
HiveExternalCatalog (sql/hive/.../HiveExternalCatalog.scala)	Backs Spark's ExternalCatalog with the Hive metastore.
HiveClient (sql/hive/.../client/HiveClient.scala)	Pluggable client for talking to the metastore.
IsolatedClientLoader (sql/hive/.../client/IsolatedClientLoader.scala)	Loads each Hive client version in its own classloader.
HiveSessionStateBuilder	Replaces the default builder when Hive support is on.
HiveStrategies	Adds Hive-specific physical-plan rules.
HiveThriftServer2	The JDBC/Thrift server entry point.
SparkSQLCLIDriver	The bin/spark-sql REPL.

Hive catalog wiring

graph LR
    SS[SparkSession] -->|enableHiveSupport| HSSB[HiveSessionStateBuilder]
    HSSB --> HSC[HiveSessionCatalog]
    HSC --> SC[SessionCatalog]
    HSC --> HEC[HiveExternalCatalog]
    HEC --> HC[HiveClient]
    HC -->|Thrift| MS[(Hive metastore)]
    HSSB --> HS[HiveStrategies]
    HS --> SP[SparkPlanner]

The IsolatedClientLoader is critical: Spark builds against multiple versions of the Hive metastore client (currently 2.x through 3.x) and isolates each in its own classloader so that their incompatible dependencies do not collide with Spark's own.

Hive UDFs

Built-in Hive UDFs are registered through HiveSessionResourceLoader and resolved by the HiveFunctionRegistry. Wrappers convert Hive's ObjectInspector types to Catalyst types in sql/hive/.../HiveInspectors.scala.

Hive SerDes and storage formats

Reads and writes use Hive's SerDe/InputFormat/OutputFormat machinery via HadoopTableReader and HiveFileFormat. ORC and Parquet have native fast paths in sql/core/.../execution/datasources/{orc,parquet}/; the Hive code path is used for less common formats and for compatibility.

Thrift server

bin/spark-sql and sbin/start-thriftserver.sh start HiveThriftServer2. It reuses Hive's SessionManager and OperationManager but swaps in SparkSQLDriver so SQL is compiled and executed via Catalyst rather than Hive's Tez/MapReduce engine.

A dedicated UI tab (sql/hive-thriftserver/.../ui/) shows session and operation history.

Integration points

• Pulls in the Hive metastore client jars at runtime.
• The Spark Connect server (sql/connect/server/) consumes Hive sessions through the same SessionHolder indirection.
• The kvstore behind the Web UI history records JDBC sessions and operations.

Entry points for modification

• Adding support for a new Hive metastore version: add a shim under sql/hive/.../client/ and wire it into IsolatedClientLoader.
• Tuning Hive command behavior: see HiveStrategies.scala and execution/InsertIntoHive*.
• Adding a new Thrift operation: implement an operation class in sql/hive-thriftserver/.../server/ and route it from SparkSQLOperationManager.