Storage

The storage layer turns disk blocks into memory pages and back. It sits below the access methods and above the operating system. Source: src/backend/storage/.

Purpose

Provide a small set of primitives that the rest of the backend relies on:

• A buffer manager that caches 8 KB pages in shared memory, pins them on behalf of callers, and writes dirty ones back lazily.
• A storage manager (smgr) that virtualizes file access: reading, writing, extending, and truncating relations through a pluggable interface (currently only one implementation, md, for "magnetic disk").
• A file manager (fd) that pools OS file descriptors so the backend can have more open files than the kernel limit.
• Lock managers — both heavyweight (deadlock-detecting, used at the SQL level) and lightweight (LWLock, used inside the engine).
• An AIO (asynchronous I/O) subsystem (src/backend/storage/aio/) for issuing reads and writes without blocking.
• Shared memory and IPC — segment creation, latches, condition variables.

Directory layout

src/backend/storage/
├── aio/         # async I/O abstraction (worker, io_uring, posix_aio)
├── buffer/      # shared and local buffer pool
├── file/        # virtual file descriptors, fsync queue
├── freespace/   # FSM (free-space map)
├── ipc/         # shmem segments, latches, procarray, sinvaladt
├── large_object/# large-object storage (pg_largeobject)
├── lmgr/        # heavyweight + lightweight locks, deadlock detection
├── page/        # page header utilities, item-id manipulation
├── smgr/        # storage manager (md)
└── sync/        # background sync queue

Key abstractions

Type	File	Role
BufferDesc	src/include/storage/buf_internals.h	Per-buffer descriptor in shared memory: tag, refcount, dirty bit, usage count.
Buffer	src/include/storage/buf.h	A 1-based integer index into the shared buffer array.
BufferTag	src/include/storage/buf_internals.h	The (relation, fork, block) triple identifying what's in a buffer.
SMgrRelation	src/include/storage/smgr.h	Storage-manager handle for a relation.
RelFileLocator	src/include/storage/relfilelocator.h	The (db, tablespace, relfilenode) triple identifying a physical file.
LWLock	src/include/storage/lwlock.h	Lightweight latch protecting shared-memory data structures.
LOCK / LOCALLOCK	src/include/storage/lock.h	Heavyweight lock entries.
PGPROC	src/include/storage/proc.h	Per-backend shared state in the proc array.
Latch	src/include/storage/latch.h	Cross-process condition variable.

Buffer manager

The default buffer pool is shared_buffers 8 KB pages, allocated in shared memory at postmaster start. Source: src/backend/storage/buffer/bufmgr.c and freelist.c.

The lifecycle of a page read:

sequenceDiagram
    participant AM as Access method
    participant BM as bufmgr (ReadBuffer)
    participant SM as smgr (md)
    participant OS as Kernel page cache

    AM->>BM: ReadBufferExtended(rel, fork, blk)
    BM->>BM: BufTableLookup() in shared hash
    alt hit
        BM->>BM: PinBuffer + UsageCount++
        BM-->>AM: Buffer
    else miss
        BM->>BM: StrategyGetBuffer (clock sweep)
        opt victim is dirty
            BM->>SM: smgrwrite()
            SM->>OS: pwrite() + fsync queue
        end
        BM->>SM: smgrread()
        SM->>OS: pread()
        OS-->>SM: page bytes
        SM-->>BM: filled buffer
        BM->>BM: BufTableInsert
        BM-->>AM: Buffer
    end

Eviction uses a clock sweep (StrategyGetBuffer in freelist.c): each buffer has a usage_count, decremented as the sweep hand passes; a buffer with usage_count == 0 and refcount 0 is the victim.

Pinning, locking, and dirty bits

A Buffer returned by ReadBuffer is pinned — the caller is responsible for ReleaseBuffer (or letting the resource owner release it on transaction end). To read or modify the page contents, the caller takes a content lock (LockBuffer(buf, BUFFER_LOCK_SHARE) or BUFFER_LOCK_EXCLUSIVE).

Marking a buffer dirty: MarkBufferDirty(buf). Writes are deferred — the buffer manager does not flush on dirty. The background writer (src/backend/postmaster/bgwriter.c) and the checkpointer move pages to disk over time.

Local buffers

Temporary tables live in local buffers (per-backend, not shared). Source: src/backend/storage/buffer/localbuf.c. Same API but no locking and no WAL.

Storage manager (smgr)

smgr abstracts the actual disk file. Currently the only implementation is md (magnetic disk), in src/backend/storage/smgr/md.c. It maps a RelFileLocator plus a fork number (main/fsm/vm/init) to one or more 1 GB segment files: base/<dbid>/<relfilenode>, base/<dbid>/<relfilenode>.1, etc.

The smgr API is small but central:

• smgrread, smgrextend, smgrwrite, smgrnblocks
• smgrtruncate, smgrcreate, smgrunlink
• smgrdosyncs, smgrimmedsync — used by the checkpointer

The forks:

Fork	Purpose
Main	The actual heap or index data.
FSM	Free-space map, tracks per-page free space for INSERT placement. src/backend/storage/freespace/freespace.c.
VM	Visibility map, tracks "all-visible" and "all-frozen" pages. src/backend/access/heap/visibilitymap.c.
Init	Init fork for unlogged relations: an empty image used to reset the relation after a crash.

Async I/O

src/backend/storage/aio/ is a relatively new subsystem that lets the backend issue reads and writes asynchronously rather than synchronously through pread/pwrite. It has multiple methods configurable via io_method:

• worker — pool of helper backends.
• io_uring — Linux native (when --with-liburing).
• (Historical / experimental: posix_aio.)

The layered API exposes pgaio_io_* operations; the buffer manager calls into it for prefetching and large reads. This subsystem is under active development and is the substrate for upcoming features like read-ahead in sequential scans and parallel index builds.

File descriptor pool

The OS limits how many files a process can have open at once. PostgreSQL works around this with vfd (virtual file descriptors), source: src/backend/storage/file/fd.c. The backend can "open" arbitrarily many files; the vfd layer LRU-evicts and reopens them transparently. Always use OpenTransientFile, BasicOpenFile, or PathNameOpenFile rather than open() directly.

Lock managers

Two distinct systems:

Lightweight locks (LWLocks)

Source: src/backend/storage/lmgr/lwlock.c. Spin-wait + semaphore-backed, shared/exclusive. Protect shared-memory data structures. Examples: BufFreelistLock, ProcArrayLock, XidGenLock, per-buffer BufferDescriptor.content_lock.

LWLockAcquire(&lock, LW_EXCLUSIVE);
/* critical section */
LWLockRelease(&lock);

LWLocks have no deadlock detection. Cycles are programming errors; the project relies on a documented lock-ordering convention (see comments in src/include/storage/lwlock.h). Acquiring multiple LWLocks always follows tranche-id order.

Heavyweight locks

Source: src/backend/storage/lmgr/lock.c, proc.c. Hash-table-backed; one entry per locked object. Used at the SQL level: relation locks, row locks, advisory locks, virtual-XID locks. Modes form a partial order (e.g., AccessShareLock, RowExclusiveLock, ShareLock, ExclusiveLock, AccessExclusiveLock for relations).

If a backend cannot acquire a heavy lock immediately, it sleeps in ProcSleep. After deadlock_timeout (default 1s), the deadlock detector (src/backend/storage/lmgr/deadlock.c) walks the wait-for graph; if a cycle exists, one transaction is aborted to break it.

ProcArray

src/backend/storage/ipc/procarray.c maintains the array of PGPROC entries — one per active backend. The proc array is the single most-contended data structure in the engine, because every snapshot computation reads it. It stores each backend's:

• Backend ID and PID
• Database OID
• XID and xmin (oldest XID still visible to this backend)
• vacuumFlags (e.g., this backend is a vacuum)
• Latches and condition variables for IPC

Visibility-related operations live here: GetSnapshotData builds a snapshot by snapshotting the proc array; TransactionIdIsInProgress consults it.

Latches

Cross-process condition variables. Source: src/backend/storage/ipc/latch.c. A backend WaitLatches; another process SetLatches it via signal/socket. Used pervasively for: client wait, replication wait, autovacuum signaling, parallel worker coordination.

WaitEventSet is the multi-fd version; it can wait on a latch, multiple sockets, and a postmaster-death pipe at once. Most "the backend is idle" loops in the codebase end up in WaitEventSetWait.

Entry points for modification

• Adding a buffer-pool feature (e.g., a new prefetch hint): start in src/backend/storage/buffer/bufmgr.c. The prefetch entry point is PrefetchBuffer.
• Adding a new storage manager: implement the f_smgr interface and register it in smgrsw in src/backend/storage/smgr/smgr.c.
• Adding a new LWLock: declare it in src/backend/storage/lmgr/lwlock.c (BuiltinTrancheNames array) or via LWLockNewTrancheId for extensions.
• Adding a new heavy-lock mode: rare, but lives in src/include/storage/lock.h (LockMethodData).

For the layer above this — heap, indexes, table AM API — see Access methods. For how this is integrated into MVCC, see MVCC and transactions. For how dirty pages reach disk safely, see WAL and recovery.