sparkr

SparkR provides an R binding to Spark's DataFrame API. As of Spark 4 it is deprecated for end users but remains in the codebase for transitional support, backwards compatibility, and CI testing. Bug fixes are still accepted.

Purpose

• Expose Spark DataFrames to R via S4-style classes and dispatch.
• Bridge R UDFs (and the dapply, gapply family) into the Spark engine.
• Provide R wrappers for MLlib pipelines.

Directory layout

R/
  pkg/                              - the actual R package (CRAN-style)
    DESCRIPTION, NAMESPACE
    R/                              - R source
      sparkR.R                      - sparkR.session() entry point
      DataFrame.R, SQLContext.R, column.R, functions.R, types.R
      mllib_*.R                     - ML wrappers (one file per algorithm family)
      streaming.R                   - Structured Streaming wrappers
      worker/                       - the R worker run by executors for UDFs
    inst/                           - on-disk metadata, profiles
    tests/                          - testthat suites
  install-dev.sh                    - install the dev package into R
  run-tests.sh                      - run the testthat suite
  build/                            - build/release helpers
  WINDOWS.md                        - Windows-specific notes

The Scala-side R support is in core/.../api/r/ (RBackend, RBackendHandler, RRDD) and in mllib/.../ml/r/ for the ML wrappers.

Key abstractions

Type / file	What it is
sparkR.session (R/pkg/R/sparkR.R)	Entry point. Starts the JVM via org.apache.spark.deploy.RRunner.
SparkDataFrame (R/pkg/R/DataFrame.R)	S4 class wrapping a JVM Dataset.
dapply, gapply, dapplyCollect, gapplyCollect	Apply an R function to partitions or grouped partitions.
RBackend / RBackendHandler (core/.../api/r/)	The Scala-side gateway that the R process talks to.
RRDD (core/.../api/r/RRDD.scala)	An RDD whose computation is delegated to an R worker.
MLlib R wrappers (mllib/.../ml/r/*.scala)	One per algorithm; they wrap a Pipeline for R consumption.

How an R UDF runs

sequenceDiagram
    participant Drv as R driver
    participant JVM as RBackend (driver)
    participant E as Executor
    participant W as R worker

    Drv->>JVM: dapply(fn, schema)
    JVM->>JVM: capture closure, schema, R deps
    JVM->>E: launch task with serialized R function
    E->>W: spawn R process
    E->>W: send Arrow batches over socket
    W->>W: run fn
    W-->>E: Arrow batches back
    E-->>JVM: task result
    JVM-->>Drv: done

The same Arrow-IPC fast path used by PySpark is used by SparkR for dapply/gapply.

Streaming and ML

SparkR exposes Structured Streaming through read.stream, write.stream, and StreamingQuery. ML wrappers are auto-generated by Scala companion objects in mllib/.../ml/r/ and surfaced as R functions like spark.logit, spark.kmeans, spark.als, etc.

Integration points

• Talks to the JVM via RBackend (a custom RPC over a socket; not Py4J).
• The R worker is spawned per partition by the executor, similarly to PySpark.
• Tests run as part of dev/run-tests only when R changes are detected.

Entry points for modification

• Adding an R wrapper for an existing DataFrame method: edit R/pkg/R/DataFrame.R (or a more specific file) and update R/pkg/NAMESPACE.
• Adding an MLlib wrapper: edit the corresponding Scala wrapper in mllib/.../ml/r/ and the R-side wrapper in R/pkg/R/mllib_*.R. The Scala wrapper is the source of truth.
• Adjusting R worker behavior: edit R/pkg/R/worker/. The Scala-side machinery in core/.../api/r/ should rarely need changes.

Most contributions to SparkR today are bug fixes and CRAN-related housekeeping.