Raft and HA

Vault uses HashiCorp's hashicorp/raft library for both the integrated storage backend and the leader-election mechanism it implies. Source: physical/raft/ (the storage adapter), vault/raft.go (1,582 lines, the runtime integration), vault/ha.go (the active/standby state machine), and vault/request_forwarding*.go for cross-node call forwarding.

Purpose

• Run Vault as its own consensus cluster (no Consul or etcd dependency).
• Provide HA: only one node serves writes at a time; standbys forward to the active node.
• Provide cluster-management commands: join, snapshot, restore, autopilot, peer removal.

Directory layout

vault/
├── raft.go                              # Glue between Core and the raft physical backend
├── ha.go                                # active/standby state machine
├── request_forwarding.go                # gRPC server that forwards from standby to active
├── request_forwarding_rpc.go            # gRPC client side
├── request_forwarding_service.proto     # Service definition
├── request_forwarding_service.pb.go     # Generated
├── request_forwarding_service_grpc.pb.go
├── cluster.go                           # Cluster TLS bootstrapping (used for forwarding)
└── cluster/                             # Cluster member helpers

physical/raft/                           # Storage adapter
├── raft.go                              # The Backend implementation
├── fsm.go                               # FSM that applies log entries to BoltDB
├── snapshot.go                          # Snapshot store
├── streamlayer.go                       # TLS multiplex over the cluster port
├── peers.go                             # Cluster membership
├── bolt_*.go                            # BoltDB tuning helpers
└── ...

sdk/helper/testcluster/                  # Helpers for spinning Raft clusters in tests

Key abstractions

Symbol	File	Description
RaftBackend	physical/raft/raft.go	Implements physical.Backend, HABackend, Transactional, and snapshot management.
Core.raftBootstrap / joinRaftCluster	vault/raft.go	Bring a node into the cluster (initial bootstrap or operator raft join).
Core.runStandby	vault/ha.go	The standby loop: try to acquire the leader lock; otherwise forward.
RequestForwardingHandler	vault/request_forwarding.go	The gRPC server hosted by every node so peers can forward requests in.
WrapHandlerFunc	vault/request_forwarding_rpc.go	The standby's call into the active node over gRPC.
AutoPilot	physical/raft/peers.go (and upstream raft autopilot)	Health-based peer management.

Cluster topology

graph LR
    subgraph Cluster
        N1[Node A<br/>Active] -- raft.write --> N1f[FSM + BoltDB]
        N2[Node B<br/>Standby] -- raft.append --> N2f[FSM + BoltDB]
        N3[Node C<br/>Standby] -- raft.append --> N3f[FSM + BoltDB]
        N2 -. forward request .-> N1
        N3 -. forward request .-> N1
        N1 -- raft replication --> N2 & N3
    end
    Client --> N1 & N2 & N3

Every node has a complete copy of the storage. Reads can be served by any node (subject to read consistency headers); writes are forwarded to the active.

Lifecycle

Bootstrap a single-node cluster:

vault server -config=raft.hcl    # storage "raft" { ... }
vault operator init -key-shares=… -key-threshold=…

Add nodes with vault operator raft join https://leader:8200. The first node initializes the Raft log; subsequent nodes catch up via streamlayer.go over the cluster port (typically 8201).

sequenceDiagram
    participant N as New node
    participant L as Leader
    N->>L: POST /sys/storage/raft/bootstrap (with TLS challenge)
    L-->>N: cluster info
    N->>L: open cluster port (TLS + ALPN)
    L-->>N: stream snapshot + tail log
    N->>N: apply to local BoltDB
    Note over N,L: Standby state until catch-up complete

Snapshots

vault operator raft snapshot save|restore|inspect is implemented in command/operator_raft_snapshot*.go and the HTTP layer in http/sys_raft.go. Snapshots are zstd-compressed BoltDB exports plus a trailing keyring marker so they can be restored to a fresh cluster.

Autopilot

Autopilot continuously evaluates peer health (recent heartbeats, log lag) and removes dead nodes from the configuration. CLI surface is under vault operator raft autopilot get-config|set-config|state. Implementation lives in the upstream hashicorp/raft-autopilot library, wired through physical/raft/peers.go.

Request forwarding

When a standby receives a write, Core.HandleRequest notices the local node isn't active and:

1. Wraps the request in a gRPC envelope (vault/request_forwarding.go).
2. Sends it to the active node over the cluster port.
3. The active node executes it, returns the response.
4. The standby relays the response to the client.

Idempotent reads can be served locally if the client sets X-Vault-No-Request-Forwarding.

Integration points

• Implements physical.Backend so Core treats it like any other storage backend.
• Holds the leader lock used by vault/ha.go for active election.
• Cluster TLS reuses the cluster_addr listener configured in the server config.
• Snapshots integrate with the recovery seal so the keyring travels with the backup.

Entry points for modification

• Tweak Raft tuning: physical/raft/raft.go's default RaftConfig and the BoltDB options in bolt_*.go.
• Add a new operator-raft CLI command: command/operator_raft*.go.
• Modify forwarding semantics: vault/request_forwarding*.go.