Snapshots and repositories

Purpose

The snapshot system provides point-in-time backup and restore of indices and cluster state to pluggable blob stores. The same blob format powers searchable snapshots, where indices are read directly from the repository without ever materializing the full data on disk.

Directory layout

server/src/main/java/org/elasticsearch/repositories/
├── Repository.java                      Pluggable repository SPI
├── RepositoriesService.java             Per-node registry
├── blobstore/                           Common blob-store-based repository implementation
│   ├── BlobStoreRepository.java         The big one
│   └── ...
├── fs/FsRepository.java                 Local filesystem
└── ...

server/src/main/java/org/elasticsearch/snapshots/
├── SnapshotsService.java                Master-side orchestration
├── RestoreService.java
├── SnapshotShardsService.java           Per-node shard snapshotting
└── ...

modules/repository-azure/, repository-gcs/, repository-s3/, repository-url/
plugins/repository-hdfs/

x-pack/plugin/searchable-snapshots/, snapshot-based-recoveries/, blob-cache/

Key abstractions

Type	File	Role
Repository	server/src/main/java/org/elasticsearch/repositories/Repository.java	Pluggable repository SPI
BlobStoreRepository	.../repositories/blobstore/BlobStoreRepository.java	Default implementation: shared shard files in a blob store
RepositoriesService	.../repositories/RepositoriesService.java	Per-node repository registry
SnapshotsService	server/src/main/java/org/elasticsearch/snapshots/SnapshotsService.java	Master-side workflow
SnapshotShardsService	.../snapshots/SnapshotShardsService.java	Per-node shard snapshot worker
RestoreService	.../snapshots/RestoreService.java	Master-side restore workflow
BlobCache	x-pack/plugin/blob-cache	Disk-backed cache for searchable-snapshot blobs

Snapshot creation

sequenceDiagram
  participant Master
  participant DataNode
  participant Repo as Repository (S3 / GCS / FS)
  Master->>Master: cluster state task: snapshot in progress
  Master->>DataNode: shard-level snapshot task per primary
  DataNode->>Repo: upload changed Lucene segment files
  DataNode->>Repo: upload shard-level snapshot index
  DataNode-->>Master: shard finalized
  Master->>Repo: upload top-level snapshot metadata

Shards reuse Lucene's segment-level granularity: only segments not yet present in the repository for this shard generation are uploaded. This makes incremental snapshots cheap.

The repository layout (under BlobStoreRepository) is:

<repo>/
├── index-<gen>                          Latest top-level repository metadata
├── index.latest                         Pointer file
├── snap-<uuid>.dat, meta-<uuid>.dat     Per-snapshot metadata
└── indices/<index-uuid>/
    └── <shard-id>/
        ├── index-<gen>                  Per-shard repository state
        ├── snap-<uuid>.dat              Shard snapshot description
        └── __<file-hash>                Reused Lucene segment blobs

The indirection via index-<gen> lets multiple snapshots share segment blobs.

Restore

Restores re-create the routing entries via RestoreService, then drive shard recovery from the repository — RecoverySource.SnapshotRecoverySource reads segment blobs back into the shard's data directory before it joins.

Searchable snapshots

For "frozen" tier indices, segments are not copied locally; instead the engine (SearchableSnapshotsEngine in x-pack/plugin/searchable-snapshots/) reads them on demand. Reads go through BlobCache (x-pack/plugin/blob-cache/), which maintains a per-node disk cache keyed by (blob, range). This is what makes the frozen tier viable on cheap object storage.

Snapshot Lifecycle Management (SLM)

X-Pack's SLM (x-pack/plugin/slm/) schedules snapshot creation and retention. SLM policies live in cluster state metadata; an SLM service on the master triggers snapshots via the regular API.

Repository plugins

Module	Backing store
modules/repository-azure	Azure Blob Storage
modules/repository-gcs	Google Cloud Storage
modules/repository-s3	Amazon S3 (and S3-compatible)
modules/repository-url	Read-only HTTP/HTTPS URL
plugins/repository-hdfs	HDFS

Each plugin implements Repository (almost always via BlobStoreRepository) and registers blob client setup, credential handling, and SDK options.

Snapshot-based recoveries

x-pack/plugin/snapshot-based-recoveries/ lets a replica recover from a snapshot rather than from the primary, dramatically reducing primary load during scale-out.

Stateless

x-pack/plugin/stateless/ reuses the repository plumbing but rather than periodic snapshots, it keeps the durable copy of every index continuously in object storage; nodes are caches that can be replaced at will. Stateless reuses BlobStoreRepository patterns plus a per-shard "object store engine" that bypasses local Lucene durability.

Entry points for modification

• New repository type? Implement Repository (typically via BlobStoreRepository) and contribute via RepositoryPlugin.
• New shard-snapshot strategy? Look at SnapshotShardsService.
• Tuning blob throughput? Per-repository chunk-size and concurrency settings under repositories.<type>.*.
• Searchable snapshot caches? xpack.searchable.snapshot.cache.* settings.