Backups

src/Backups/ implements BACKUP and RESTORE — declarative SQL statements that snapshot tables, databases, or the whole instance to a backup destination, and later recreate them from that snapshot.

Why a SQL surface?

Filesystem-level snapshots of a ClickHouse data directory are insufficient: the engine has cross-table state in Keeper, in materialized views, in dictionaries, and in access entities. A correct backup needs to capture the logical state — schemas plus part contents plus access plus configuration — and restore it consistently across replicas.

BACKUP was added in August 2021.

Surface

BACKUP DATABASE my_db TO S3('https://...', 'access_key', 'secret');
BACKUP TABLE my_db.events TO File('/var/backups/events.bk');
BACKUP ALL TO Disk('backups_disk', 'snapshot1.bk');
RESTORE DATABASE my_db FROM File('events.bk');

Backups can be stored on:

• A configured IDisk (local, S3, Azure, encrypted, web).
• An ad-hoc S3(...), AzureBlobStorage(...), File(...), Disk(...) destination.

Architecture

graph TD
    SQL[BACKUP statement] --> Coord[BackupsWorker<br/>BackupsWorker.cpp]
    Coord --> Entries[BackupEntryFromImmutableFile<br/>BackupEntryFromAppendOnlyFile<br/>BackupEntryFromMemory]
    Coord --> Writer[IBackupWriter<br/>BackupWriterDisk / S3 / File / WithChecksums]
    Writer --> Dest[(Destination disk / S3 / file)]
    Coord -.coord.-> Keeper[Keeper coordination<br/>BackupCoordination*]

Component	File
Worker (the orchestrator)	BackupsWorker.cpp
Backup definition (settings, name, destination)	BackupInfo.cpp, BackupSettings.cpp
Per-entry abstraction	IBackupEntry.h, BackupEntryFromImmutableFile.cpp, BackupEntryFromAppendOnlyFile.cpp, BackupEntryFromMemory.cpp, BackupEntryFromCallback.cpp
Reader/Writer over a destination	IBackupReader.h, IBackupWriter.h, BackupReaderDisk/File/S3.cpp, BackupWriterDisk/File/S3.cpp
Cluster coordination	BackupCoordinationLocal.cpp, BackupCoordinationRemote.cpp, BackupCoordinationStageSync.cpp
Restoration	RestorerFromBackup.cpp, RestoreCoordinationLocal/Remote.cpp
Concurrency	BackupsWorker.cpp, BackupConcurrencyCheck.cpp
Files-and-deduplication tracking	BackupCoordinationFileInfos.cpp, BackupFileInfo.cpp

Cluster-aware backups

BACKUP ... ON CLUSTER ... runs on every replica involved. BackupCoordinationRemote uses Keeper to:

• Elect a coordinator.
• Distribute file ownership (each replicated part is backed up exactly once).
• Sync stages (collect-files → write → finalize).
• Verify completion across replicas.

This is why a cluster-wide backup is consistent without freezing the cluster.

Incremental backups

BACKUP ... TO ... SETTINGS base_backup = ... uses an existing backup as the baseline. Only files that differ are written; the manifest records pointers into the base backup. RESTORE resolves them transparently.

What's included

BACKUP DATABASE includes:

• Table schemas (CREATE TABLE statements).
• All MergeTree parts (immutable on disk → cheap to back up by reference).
• Materialized views' state (the view's storage is a regular table).
• Dictionaries (or pointers to their config).
• (Optionally) access entities, ON CLUSTER DDL queue, named collections, user-defined SQL objects, settings profiles.

BACKUP ALL extends to instance-wide state.

Format

A backup is a directory tree (on whatever destination) with:

• .backup manifest — JSON with file list, hashes, ownership.
• A directory-per-database tree mirroring metadata/ and store/.
• Compressed .bin files for each part column (or pass-through references when the destination disk is the same as the source).

BackupCoordinationFileInfos deduplicates files within a backup (two parts that share an immutable file are stored once).

Diagnostics

• system.backups — recent backups and their state.
• system.backup_log — historical log.
• BACKUP <...> returns a backup_id that you can pass to system.backups.
• KILL BACKUP and KILL RESTORE for cancellation.

Settings

BackupSettings.cpp defines:

• compression_method, compression_level.
• password (for encrypted destinations).
• internal — for the cluster-coordination case.
• host_id, coordination_zk_path.
• read_from_filesystem_cache, s3_storage_class, s3_throttler_*.

Entry points for modification

• New backup destination → subclass IBackupReader / IBackupWriter and register in BackupFactory.
• New entry kind → subclass IBackupEntry.
• New settings → BackupSettings.cpp.

Related pages

• MergeTree — supplier of immutable parts.
• IO and disks — destination disks.
• Coordination — cluster orchestration.