Fail points

A fail point is a named, runtime-controllable hook in the server. Tests use them to inject delays, errors, or take a fixed branch through a code path. Without them many integration tests would be unable to deterministically reproduce race conditions or rare error paths. The user-facing reference is docs/fail_points.md.

Purpose

A typical fail point looks like this in code:

#include "mongo/util/fail_point.h"

MONGO_FAIL_POINT_DEFINE(rsSyncApplyStop);

void applyOplogBatch(...) {
    rsSyncApplyStop.executeIf(
        [&](const BSONObj& data) {
            sleepuntilSignal();
        },
        [](const BSONObj& data) {
            // predicate: only fire if data matches
            return true;
        });
    // ... real work ...
}

Tests enable it via:

db.adminCommand({ configureFailPoint: 'rsSyncApplyStop', mode: 'alwaysOn' });

When the test wants to release the hook:

db.adminCommand({ configureFailPoint: 'rsSyncApplyStop', mode: 'off' });

Modes

configureFailPoint accepts:

• "alwaysOn" — every check fires.
• "off" — the hook is disabled (default).
• { activationProbability: 0.5 } — probabilistic firing.
• { times: N } — fire N times, then disable.
• { skip: N, times: M } — skip the first N, then fire M times.

The optional data payload is a BSON document passed to the registered closure.

Hook styles

The framework supports several hook styles:

• shouldFail() — fastest path. Returns true/false; the caller branches.
• pauseWhileSet() — blocks while the fail point is active. Used to pause work mid-flight.
• executeIf(...) — runs a closure with the configured data, optionally gated by a predicate.
• scoped() / scopedIf(...) — RAII handle that exposes the data while the hook is active.

The full API is in src/mongo/util/fail_point.h.

Discovery

The set of registered fail points is exposed via:

db.adminCommand({ configureFailPoint: '*', mode: 'off' }); // not actually a wildcard, just illustrative
db.adminCommand({ getFailPointInfo: 1 });

The list comes from the MONGO_FAIL_POINT_DEFINE registrations across the binary and is populated at startup.

Where they're used

Fail points are scattered throughout the codebase, especially in:

• src/mongo/db/repl/ — to pause oplog application, force election state changes, drop heartbeats.
• src/mongo/db/index_builds/ — to pause two-phase index builds at well-known points.
• src/mongo/db/s/ — to pause migrations, range deletions, resharding state transitions.
• src/mongo/s/transaction_router.cpp — to inject errors into commit/abort paths.
• src/mongo/db/storage/ — to inject WiredTiger errors and write-conflicts.

A grep for MONGO_FAIL_POINT_DEFINE finds hundreds of registrations.

Production safety

Fail points are compiled out in release builds by default — they're enabled when the binary is built with the --use-diagnostic-latches / debug-mode toggles, which is the case for all CI and Evergreen builds but not for shipped binaries. Some "runtime" fail points are always present in shipping binaries because they implement test-only commands or are gated behind admin-only flags; those are explicit and reviewed.

Key source files

File	Purpose
src/mongo/util/fail_point.h	The FailPoint class and macros.
src/mongo/util/fail_point_registry.h	The global registry.
src/mongo/util/fail_point_service.cpp	configureFailPoint / getFailPointInfo command glue.

Integration points

• Tests (jstests/) routinely use fail points to coordinate with the server.
• Diagnostic commands — configureFailPoint and waitForFailPoint are exposed as admin commands when test commands are enabled (--enableTestCommands=1).
• The replication and sharding subsystems treat fail points as a documented part of their test API; many fail points in those modules have stable names that tests rely on.

Entry points for modification

Adding a fail point is a single MONGO_FAIL_POINT_DEFINE(name) plus a name.shouldFail() (or richer hook) at the desired location. The name is the externally visible identifier — pick something unique and descriptive. Removing a fail point requires checking that no jstests/ invocation still references it; the simplest way is rg "configureFailPoint.*\\bMyFailPoint\\b" jstests/.