Modularity

The MongoDB codebase has, for most of its life, allowed any file to use any other file's symbols. With ~9,000 C++ files spread across src/mongo/, this freedom has gradually become a maintainability tax: subsystems reach into each other's internals in ways that block refactors, increase compile times, and make the API surface effectively unbounded.

The modularity initiative documented in docs/modularity.md is the team's response. It defines:

• Modules — named groups of files declared in modules_poc/modules.yaml.
• API visibility levels — OPEN, PUBLIC, PRIVATE, FILE_PRIVATE, NEEDS_REPLACEMENT, USE_REPLACEMENT(...), plus parent/sibling variants — applied via attribute macros from src/mongo/util/modules.h.
• Tooling under modules_poc/ that scans the codebase and rejects cross-module access to anything that isn't PUBLIC.

This page summarizes how the system is organized and how to work with it.

Why this is hard for MongoDB specifically

A typical project enforces module boundaries by leaning on the language: separate compilation units, public headers in one directory, internal headers in another. MongoDB's structure makes that difficult:

• No directory-level public/internal split. Almost every directory mixes "public-ish" headers with implementation detail, often by historical accident.
• Heavy use of templates and macros. Templates instantiate at the call site, breaking simple "header X is private" rules.
• Cross-cutting concerns. OperationContext, Status, and BSON are touched by every module; their genuine public API needs to be carefully separated from their internals.
• Module assignment isn't team ownership. A team may legitimately own files in many modules (e.g. a query team owns code in query, pipeline, exec, and commands).

The modules_poc tooling addresses these by working off Clang's AST: it sees what was actually used at compile time, not just what was textually included.

How a module is defined

Modules live in modules_poc/modules.yaml. Each module entry lists glob patterns; if multiple globs match, the most specific wins. Sample structure:

- name: replication
  globs:
    - src/mongo/db/repl/**
    - src/mongo/db/op_observer/**
- name: sharding
  globs:
    - src/mongo/db/s/**
    - src/mongo/s/**

The current set covers roughly 30 modules — replication, sharding, query, exec, storage, transport, executor, util, base, idl, and so on. The full list is in modules.yaml and grows over time.

Visibility levels

Visibility is set with attributes from src/mongo/util/modules.h:

Macro	Meaning
MONGO_MOD_OPEN	Public for use and inheritance from anywhere.
MONGO_MOD_PUBLIC	Public for use; cannot be subclassed across modules.
MONGO_MOD_PRIVATE (default)	Only usable from the same module.
MONGO_MOD_PARENT_PRIVATE	Like PRIVATE, but visible to the parent module's submodules.
MONGO_MOD_FILE_PRIVATE	Only usable from the same "file family" (header + cpp + tests).
MONGO_MOD_NEEDS_REPLACEMENT	Public temporarily; the API should be removed or replaced.
MONGO_MOD_USE_REPLACEMENT(name)	Public, but callers should switch to name instead.
MONGO_MOD_PUB_FOR_TECHNICAL_REASONS	Public only because the language demands it (e.g. macros expanding into details).

Including mongo/util/modules.h in a header switches its default visibility from "anything goes" to PRIVATE, making it explicit which APIs are intended for outside consumers.

Tooling

Under modules_poc/:

• merge_decls.py — runs the scanner across the codebase via Bazel (--config=mod-scanner) and writes merged_decls.json. The scanner is the source of truth for "this declaration was used from this file, with this visibility."
• browse.py — TUI that displays declarations grouped by module and file, sorted by number of unmarked declarations. Used to label public APIs.
• private_headers.py — bulk-marks headers that have no detected external usages as fully private.
• mod_diff.py — generates a PR comment summarizing the API visibility changes in the current branch.

The full workflow is described in docs/modularity.md.

Workflow

A typical PR that's labeling APIs:

1. buildscripts/poetry_sync.sh
2. modules_poc/merge_decls.py — refresh the scan.
3. modules_poc/private_headers.py --module=YOUR_MODULE -n — preview which headers can be auto-marked private.
4. Open modules_poc/browse.py, look at the remaining unmarked headers, and label each declaration with the appropriate MONGO_MOD_* macro.
5. modules_poc/mod_diff.py — generate a per-PR summary of the changes.
6. Open the PR with the mod_diff.py output included as a code-block comment for reviewers.

Limitations and caveats

The full list is in docs/modularity.md. Highlights:

• Templates and dependent types may report usages that the tool can't fully resolve — these manifest as "missing" usages of dependent type members.
• Macros are expanded before scanning, so anything emitted from a macro is attributed to the expansion site.
• using declarations can confuse usage attribution; sometimes a usage appears against the base class rather than the using class.
• _forTest functions default to FILE_PRIVATE and need explicit marking if intended for outside test files.

These limitations mean the tool is conservative — it reports false negatives (missed usages) more often than false positives (spurious violations).

Status

The effort is ongoing. The progress is tracked module-by-module via a jq query against merged_decls.json (the example in docs/modularity.md prints "marked / total" percentages per module). The bulk of the labeling has happened in query, replication, and sharding so far; some smaller modules are essentially complete; cross-cutting modules like util and base will likely take longest because their consumers are everywhere.

Why this matters for contributors

• Private APIs can be reorganized freely. If you own a module, anything you marked PRIVATE is yours to change without ceremony.
• Public APIs are a contract. Changes to PUBLIC declarations are breaking — they need to go through the deprecation flow described in docs/modularity.md.
• Cross-module access requires a PUBLIC API. If you find yourself needing a PRIVATE header from another module, the right answer is usually to expose the slice of API you need (with that module's owner's blessing) or move the file. "Just include it" is no longer the answer.

Key source files

File	Purpose
docs/modularity.md	The canonical design document.
src/mongo/util/modules.h	The visibility macros and their semantics.
modules_poc/modules.yaml	Module-to-files assignment.
modules_poc/merge_decls.py	Scanner driver.
modules_poc/browse.py	TUI for labeling.
modules_poc/private_headers.py	Bulk-mark de-facto-private headers.
modules_poc/mod_diff.py	PR comment generator.