connectors

connector/ collects out-of-core data-source and integration modules that ship in the same repo but are released as separate artifacts. As of Spark 4 the Avro, Protobuf, and XML connectors have been promoted to the sql/core tree so that they ship with every Spark distribution; the rest remain optional artifacts.

Modules

Module	Source	What it does
spark-avro (legacy connector dir)	connector/avro/	Avro support; thin shim around the in-sql/core impl.
spark-protobuf	connector/protobuf/	Reads and writes Protobuf, including dynamic descriptors.
spark-sql-kafka-0-10	connector/kafka-0-10-sql/	Structured Streaming source/sink for Kafka 0.10+.
spark-streaming-kafka-0-10	connector/kafka-0-10/	Legacy DStream Kafka integration.
spark-token-provider-kafka-0-10	connector/kafka-0-10-token-provider/	Kafka delegation-token provider (Kerberos).
spark-streaming-kinesis-asl	connector/kinesis-asl/	DStream and Structured Streaming integration with AWS Kinesis.
spark-ganglia-lgpl	connector/spark-ganglia-lgpl/	Ganglia metrics sink (LGPL, hence its own artifact).
Profiler	connector/profiler/	Async-profiler integration that drops flamegraphs.
Docker integration tests	connector/docker-integration-tests/	JDBC tests against real databases via Docker.
Kafka assembly	connector/kafka-0-10-assembly/	Fat jar of the streaming Kafka connector.
Kinesis assembly	connector/kinesis-asl-assembly/	Fat jar of the Kinesis connector.

Kafka

connector/kafka-0-10-sql/ is the Structured Streaming Kafka source and sink. It parses Kafka offsets to and from the streaming offset log, supports per-topic-partition offset ranges, and integrates with the Kafka delegation-token system for Kerberos.

Key files:

• KafkaSourceProvider - registers kafka as a streaming source.
• KafkaMicroBatchStream - implements the V2 source.
• KafkaSourceRDD - per-batch RDD that reads from the Kafka brokers.
• KafkaWriteTask / KafkaContinuousReadSupport for sink and continuous mode.

Kinesis

connector/kinesis-asl/ (Apache Software License). Contains a DStream-based receiver (KinesisReceiver), a Structured Streaming source via KinesisDataStreamSource, and the checkpointing machinery built on the AWS Kinesis Client Library.

Protobuf

connector/protobuf/ adds from_protobuf and to_protobuf SQL functions plus a DataSource for files of length-prefixed protobuf records. It supports dynamic descriptor files and Confluent Schema Registry lookup.

Avro

The Avro connector is a thin shim. The actual implementation now lives in sql/core/.../avro/ and is built into Spark by default. The connector is kept around for users who pin to an older artifact id.

Profiler

connector/profiler/ integrates the async-profiler into Spark executors. It captures CPU and lock samples and writes flamegraphs to the configured output directory.

The plugin is wired in via SparkPlugin (core/.../api/plugin/SparkPlugin.scala).

Docker integration tests

connector/docker-integration-tests/ is excluded from the default test matrix. It runs end-to-end JDBC tests against real Postgres, MySQL, MSSQL, Oracle, and DB2 instances spun up in Docker. It is the canonical place to verify dialect-level behavior in sql/core/.../jdbc/dialect/.

Where each connector ships

Most connectors are shipped as separate Maven artifacts under org.apache.spark:spark-<name>_2.13. The Avro, Protobuf, and XML readers are also accessible through the built-in data source registry once the corresponding artifact is on the classpath. Some, such as spark-ganglia-lgpl, must be built explicitly because of license constraints.

Entry points for modification

• Bump a Kafka client version: edit connector/kafka-0-10-sql/pom.xml (and the legacy module's pom). Compatibility is checked with the docker integration tests where possible.
• Add a Protobuf descriptor source: edit connector/protobuf/.../sql/protobuf/utils/SchemaConverters.scala and add tests.
• Add a JDBC dialect-specific behavior: edit the dialect class in sql/core/.../jdbc/dialect/ (which is in the SQL module, not here) and add a docker test in connector/docker-integration-tests/ if you can reproduce against a real database.