Jobs

pkg/jobs/ is CockroachDB's framework for long-running, cluster-wide, resumable work. Schema changes, backups, restores, imports, changefeeds, TTL deletes, IMPORT, INSPECT, history retention, and cross-cluster replication all run as jobs.

Purpose

Provide a single, durable, observable pattern for asynchronous work that:

• Survives node restarts (state is persisted in system.jobs and system.job_info).
• Reports progress and can be paused/canceled by PAUSE JOB/CANCEL JOB.
• Coordinates which node currently runs the job via pkg/sql/sqlliveness/ claims.
• Emits structured logs and metrics.

Files

pkg/jobs/
├── jobs.go                  // Job + Resumer interface (~32 KB)
├── registry.go              // Registry: claim, run, resume, cancel (~73 KB)
├── adopt.go                 // adopting unclaimed jobs (~23 KB)
├── update.go                // mutating job state
├── progress.go              // progress reporting
├── job_info_storage.go      // payload + progress in system.job_info
├── job_scheduler.go         // recurring schedules (system.scheduled_jobs)
├── scheduled_job.go         // ScheduledJob type
├── metrics.go               // jobs metrics (~20 KB)
├── jobspb/                  // proto types — Payload, Progress, Details
├── jobsauth/                // permission checks
├── jobsprofiler/            // execution detail profiling
├── jobsprotectedts/         // protected timestamps for backups/restores
├── jobfrontier/             // streaming "frontier" (used by CDC, backup)
├── ingeststopped/           // ingest-stop checkpoints
├── joberror/                // standard error types
└── metricspoller/           // background metric polling

Key types

Type	File	Description
Registry	pkg/jobs/registry.go	One per node. Owns adoption, running, cancelation.
Job	pkg/jobs/jobs.go	A handle to a single job row.
Resumer	pkg/jobs/jobs.go	Interface that job implementors satisfy.
Payload	pkg/jobs/jobspb/jobs.proto	Static fields: type, description, owner.
Progress	pkg/jobs/jobspb/jobs.proto	Mutable fields: fraction, high water, error.
ScheduledJob	pkg/jobs/scheduled_job.go	Recurring job created from CREATE SCHEDULE.
Scheduler	pkg/jobs/job_scheduler.go	Service that triggers due schedules.

Lifecycle

stateDiagram-v2
  [*] --> pending
  pending --> running: claim by Registry
  running --> paused: PAUSE JOB
  paused --> running: RESUME JOB
  running --> reverting: cancel / failure
  running --> succeeded
  reverting --> failed
  reverting --> canceled
  succeeded --> [*]
  failed --> [*]
  canceled --> [*]

The transitions are persisted as status updates in system.jobs.

Persistence

Two system tables back jobs:

• system.jobs — small per-job row with id, status, claim info, owner.
• system.job_info — large blob storage for Payload, Progress, and per-resumer "execution details" (logs, traces, profiles).

pkg/jobs/job_info_storage.go is the API for read/write. Profiling artifacts (heap, CPU, goroutine dumps) collected during a job land here too via pkg/jobs/jobsprofiler/.

How a job runs

1. SQL statement creates a job row: Registry.CreateAdoptableJobWithTxn writes a row in PENDING.
2. Each node's Registry.Adopt claims unclaimed jobs through a SQL session lease.
3. The matching Resumer (registered via RegisterConstructor) is constructed.
4. Resumer.Resume(ctx, execCtx) runs to completion, periodically updating progress.
5. On error, Resumer.OnFailOrCancel runs to clean up.
6. The row's status transitions to SUCCEEDED, FAILED, or CANCELED.

If the running node dies, another node re-adopts the job — the resumer must therefore be idempotent.

Resumers in the tree

Resumer	Owner	Code
Backup, Restore	disaster-recovery	pkg/backup/
Changefeed	cdc	pkg/ccl/changefeedccl/
Schema change	sql-foundations	pkg/sql/schema_changer.go, declarative version under pkg/sql/schemachanger/scjob/
TTL	sql-queries	pkg/sql/ttl/
Inspect	sql-queries	pkg/sql/inspect/
GC	sql-foundations	pkg/sql/gcjob/
Import	sql-foundations	pkg/sql/importer/
Logical/Physical replication	disaster-recovery	pkg/crosscluster/
Key-visualizer	obs	pkg/keyvisualizer/
SQL stats compaction	obs	pkg/sql/sqlstats/persistedsqlstats/
History retention	sql-queries	pkg/sql/history_retention_job.go

Schedules

pkg/jobs/job_scheduler.go and pkg/jobs/scheduled_job.go implement CREATE SCHEDULE for recurring backup, changefeed, etc. A scheduler service wakes on the cron expression, creates a child job, and records the run in system.scheduled_jobs.

Observability

Cluster-internal SQL views surface every job's state:

SELECT * FROM crdb_internal.jobs WHERE status='running';
SELECT * FROM [SHOW JOBS];
SELECT * FROM crdb_internal.cluster_queries;

The DB Console has a Jobs page that uses the same data.

Related pages

• SQL — most jobs are created from SQL.
• features/backup-restore, features/changefeed, features/cross-cluster-replication — major job-implemented features.
• Cluster version — migrations also run as jobs.