graphx

GraphX is Spark's library for graph processing on RDDs. It exposes a Graph[VD, ED] abstraction (vertices and edges as RDDs) and a Pregel-style API for iterative graph algorithms. It is in maintenance mode - bug fixes and dependency upgrades only - and is the smallest of the major modules.

Purpose

• Represent property graphs as a pair of RDDs (VertexRDD and EdgeRDD).
• Provide a fluent operator language for graph queries and transformations.
• Implement common algorithms (PageRank, Connected Components, Triangle Counting, Label Propagation, Strongly Connected Components, Shortest Paths, SVD++).

Directory layout

graphx/
  src/main/scala/org/apache/spark/graphx/
    Graph.scala               - the Graph abstraction
    VertexRDD.scala
    EdgeRDD.scala
    EdgeTriplet.scala
    Pregel.scala              - the Pregel API
    GraphOps.scala            - higher-level graph ops (degrees, joins, neighbors)
    PartitionStrategy.scala
    impl/                     - the actual implementations of Graph, VertexRDD, EdgeRDD
    lib/                      - canned algorithms
      PageRank.scala
      ConnectedComponents.scala
      TriangleCount.scala
      ShortestPaths.scala
      SVDPlusPlus.scala
      LabelPropagation.scala
      StronglyConnectedComponents.scala
    util/                     - GraphGenerators, BytecodeUtils
  pom.xml

There is no separate Python or Connect API. PySpark users typically use GraphFrames (an external project) for graph processing.

Key abstractions

Type	What it is
Graph[VD, ED] (graphx/.../Graph.scala)	A property graph parameterized by vertex and edge attribute types.
VertexRDD[VD] / EdgeRDD[ED]	Specialized RDDs that index vertices/edges for join performance.
EdgeTriplet[VD, ED]	An edge plus its two endpoint attributes.
Pregel	Iterative message-passing API.
PartitionStrategy	How edges are partitioned across executors.
GraphLoader	Loads a graph from an edge-list file.

Pregel API

Pregel-style algorithms in GraphX iterate by sending messages along edges. The signature is:

def pregel[A: ClassTag](initialMsg: A, maxIter: Int = Int.MaxValue,
                       activeDirection: EdgeDirection = EdgeDirection.Either)
                      (vprog: (VertexId, VD, A) => VD,
                       sendMsg: EdgeTriplet[VD, ED] => Iterator[(VertexId, A)],
                       mergeMsg: (A, A) => A): Graph[VD, ED]

PageRank in graphx/.../lib/PageRank.scala is a good example.

Integration points

• Built only on Spark Core (core/). It does not depend on sql/.
• Uses RDD lineage for fault tolerance during iterative computations.
• Persists intermediate vertex/edge RDDs at MEMORY_ONLY or MEMORY_AND_DISK to cap iteration cost.

Entry points for modification

• Adding a new algorithm: drop a file under graphx/.../lib/. Most algorithms are 50-200 lines.
• Tuning partitioning: implement a PartitionStrategy and pass it to Graph.partitionBy.
• Most other changes are bug fixes; do not invest in major new graph features here. The community-recommended path is GraphFrames for DataFrame-based graph queries.