searcher (grep-searcher)

Active contributors: Andrew Gallant

Purpose

crates/searcher (published as grep-searcher) is ripgrep's line-oriented search engine. It owns the loop that:

1. Reads bytes from a source (file, stdin, or memory map).
2. Hands chunks to a Matcher to find matches.
3. Tracks line numbers, context lines, binary detection, and encoding.
4. Pushes results into a Sink (typically a printer).

The Matcher is engine-agnostic; the Sink is output-agnostic. This crate sits between them and does the line-oriented bookkeeping.

Directory layout

crates/searcher/src/
├── lib.rs                 # Public API surface (re-exports + crate docs)
├── line_buffer.rs         # Streaming line buffer for non-mmap mode (~35K)
├── lines.rs               # LineIter, LineStep — slicing byte buffers into lines
├── sink.rs                # The Sink trait + SinkMatch / SinkContext / SinkFinish
├── macros.rs              # Internal macros
├── testutil.rs            # SearcherTester: matrix tester used by this crate and grep-printer
└── searcher/
    ├── mod.rs             # Searcher / SearcherBuilder / Encoding / BinaryDetection (~41K)
    ├── core.rs            # The actual search loop (~23K)
    ├── glue.rs            # Wires Searcher + Matcher + Sink + LineBuffer (~50K)
    └── mmap.rs            # MmapChoice and the mmap vs. read decision

Key abstractions

Type	File	Description
Searcher	crates/searcher/src/searcher/mod.rs	The main type. Configured by SearcherBuilder. Has methods like search_path, search_file, search_reader, search_slice.
SearcherBuilder	crates/searcher/src/searcher/mod.rs	Builder for Searcher. Controls line terminator, multi-line, encoding, binary detection, mmap choice, line numbers, before/after context, and stop-on-NUL.
Sink	crates/searcher/src/sink.rs	Trait that receives match callbacks. matched, context, context_break, binary_data, begin, finish.
SinkMatch / SinkContext / SinkFinish	crates/searcher/src/sink.rs	The data passed to Sink callbacks: the matching/contextual lines, byte offsets, line numbers.
BinaryDetection	crates/searcher/src/searcher/mod.rs	None, Quit(byte), or Convert(byte) — what to do when a forbidden byte is seen.
Encoding	crates/searcher/src/searcher/mod.rs	Wraps an encoding_rs::Encoding for transcoding non-UTF-8 input.
MmapChoice	crates/searcher/src/searcher/mmap.rs	Auto, Always, Never decision for whether to use a memory map.
LineBuffer	crates/searcher/src/line_buffer.rs	Streaming buffer used in non-mmap mode. Reads chunks, finds line boundaries, hands aligned slices to the search loop.
LineIter / LineStep	crates/searcher/src/lines.rs	Iterators for slicing a &[u8] into lines respecting the configured terminator.

How it works

sequenceDiagram
    participant Caller
    participant Builder as SearcherBuilder
    participant Searcher
    participant Source as Reader / Mmap
    participant Matcher
    participant Sink

    Caller->>Builder: configure (multi_line, encoding, binary, ...)
    Builder->>Searcher: build()
    Caller->>Searcher: search_path(matcher, path, sink)
    Searcher->>Source: open / mmap
    loop for each chunk
        Source->>Searcher: bytes
        Searcher->>Matcher: find_iter_at(chunk)
        Matcher->>Searcher: matches
        Searcher->>Sink: matched(SinkMatch) / context(SinkContext)
    end
    Searcher->>Sink: finish(SinkFinish)

The "chunk" here depends on mode:

• mmap: the entire file is mapped, then searcher/glue.rs runs the loop over the whole slice in one call to searcher/core.rs.
• buffered: line_buffer.rs reads in fixed-size chunks (default 8 KiB), maintains an alignment invariant ("the buffer always ends at a line boundary"), and feeds aligned slices to the same core loop.

Multi-line mode

When multi_line(true) is set on SearcherBuilder, the searcher disables line-by-line slicing and gives the matcher the whole haystack at once. This is what -U/--multiline does. It is mutually exclusive with chunked buffered reads — multi-line mode forces mmap (or a single read into a Vec<u8> if mmap is disabled).

Binary detection

BinaryDetection::quit(b) is the default for ripgrep when searching files. The searcher peeks at the first ~8 KiB of the haystack; if b (typically NUL) appears, the rest of the file is skipped and the sink's binary_data callback fires.

BinaryDetection::convert(b) instead replaces the byte with a line terminator and continues searching. ripgrep uses this for stdin under some conditions.

BinaryDetection::None disables the check (--text).

Encoding

If Encoding::new(...) is set, the source is read through an encoding_rs decoder that emits UTF-8 bytes. Heuristic UTF-16 detection (BOM-based) is built in. This is what makes rg --encoding utf-16le work.

Public examples

crates/searcher/examples/search-stdin.rs is a minimal program showing how to search stdin with a regex and print matching lines. The crate's lib-level docs in lib.rs show a fuller example using sinks::UTF8.

Test harness

testutil.rs defines SearcherTester, a matrix-style harness that runs the same query through every relevant configuration (mmap on/off, multi-line on/off, line numbers on/off, ...) and compares against a single expected output. Both this crate's tests and grep-printer's tests use it.

Integration points

• Used by: crates/core/search.rs::SearchWorker, every printer in grep-printer.
• Uses: grep-matcher (for the Matcher interface), bstr (byte string utilities), encoding_rs (transcoding), memchr (fast byte scanning), memmap2 (memory mapping).
• Re-exported by: grep::searcher.

Entry points for modification

• New SearcherBuilder option: add a config field, plumb through Searcher, surface as a CLI flag in crates/core/flags/defs.rs if user-facing.
• Performance changes to the core loop: searcher/core.rs and searcher/glue.rs. These files are written for the optimiser; benchmark before merging.
• New sink behaviour: usually means changes in grep-printer, not here. Only modify the Sink trait if the new behaviour cannot be expressed with existing callbacks.

For more on what plugs into a searcher, see matcher and printer.