regex (grep-regex)

Active contributors: Andrew Gallant

Purpose

crates/regex (published as grep-regex) is the default Matcher implementation for ripgrep. It wraps Rust's regex crate and regex-syntax AST utilities, adding line-oriented and inner-literal optimisations specific to grep-style search.

This is the engine that runs unless -P/--pcre2 is set.

Directory layout

crates/regex/src/
├── lib.rs              # Public API (six lines of re-exports + module declarations)
├── matcher.rs          # RegexMatcher and RegexMatcherBuilder (~25K)
├── config.rs           # Compile-time options (case-folding, multi-line, Unicode, ...)
├── ast.rs              # regex-syntax AST helpers used by literal extraction
├── ban.rs              # Forbidding certain features in non-multi-line search (anchored to lines)
├── error.rs            # RegexError + ErrorKind
├── literal.rs          # Inner literal extraction (~38K) - the biggest single source of correctness bugs historically
├── non_matching.rs     # Detecting bytes a regex can never match (used to bypass irrelevant haystack chunks)
└── strip.rs            # Stripping line-anchors that grep already enforces

Key abstractions

Type	File	Description
RegexMatcher	crates/regex/src/matcher.rs	Implementation of the Matcher trait. The struct that ripgrep's searcher hands haystack bytes to.
RegexMatcherBuilder	crates/regex/src/matcher.rs	Builder controlling case sensitivity, Unicode, multi-line, line terminator, swap-greed, and more.
RegexCaptures	crates/regex/src/matcher.rs	The Captures impl.
Config	crates/regex/src/config.rs	Internal struct holding all compile-time options.
ConfiguredHIR	crates/regex/src/config.rs	A regex-syntax HIR plus its config; intermediate representation between user input and a compiled Regex.
LiteralSets	crates/regex/src/literal.rs	Inner literal extraction: finding required substrings in a regex so the search can use a fast literal scanner (memchr / aho-corasick) before invoking the full DFA.

How it works

graph LR
    Pattern["pattern: &str"] --> Builder[RegexMatcherBuilder::build_many]
    Builder --> AST["regex-syntax AST"]
    AST --> Strip["strip line anchors"]
    Strip --> Literal["literal extraction"]
    Literal --> Compile["regex::Regex compile"]
    Compile --> Matcher[RegexMatcher]
    Matcher --> Searcher

When the binary asks for a regex matcher (in crates/core/flags/hiargs.rs::matcher), it constructs a RegexMatcherBuilder, populates it from CLI flags (-i, -S, --multiline, -w, --null-data, etc.), and calls build_many with the patterns. Internally:

1. Each pattern is parsed into a regex-syntax AST.
2. strip.rs strips redundant ^/$ anchors when ripgrep is doing line-oriented search (otherwise the regex engine would re-check what the searcher already enforces).
3. literal.rs walks the AST looking for required literal substrings. If found, it builds an Aho-Corasick automaton or a memchr scanner. The Searcher will use the literal scanner to skip over uninteresting parts of the haystack.
4. non_matching.rs derives the set of bytes the regex can never match, used to bail out early on chunks that obviously contain no match.
5. The cleaned HIR is compiled into a regex::Regex.

The Matcher trait methods (find_at, find_iter_at, etc.) delegate to the compiled Regex, but with the line-oriented assumptions baked in.

Inner-literal extraction

literal.rs is the source of more historical bugs than any other file in the workspace; the changelog cites several BUG #... entries against it. Examples:

• 14.1.1 (Sep 2024): (?i:e.x|ex) failed to match e-x. The bug was in grep-regex's literal extraction picking the wrong required substring.
• 15.0.0 (Oct 2025): rare panic for some classes of large regexes on large haystacks (BUG #3135).

The extractor's job is to find a substring that must appear in any match, so the Searcher can quickly scan with memchr or Aho-Corasick. When the extractor over-reaches (claims a literal is required when it isn't), matches are missed. When it under-reaches (claims no literals are required when one is), performance degrades. Getting this right under the full Unicode-and-look-around-free regex grammar is hard.

If you change literal.rs, add regression tests in both this crate's #[cfg(test)] mod tests and tests/regression.rs. The test files have a stable convention of naming regression tests with the issue number.

Integration points

• Implemented by: nothing. This crate is itself the implementation.
• Used by: crates/core/flags/hiargs.rs::matcher (the binary's matcher constructor), grep::regex (re-export through the facade).
• Depends on: regex, regex-syntax, regex-automata, aho-corasick, bstr.

Entry points for modification

• New regex flag (e.g., a new --style of regex semantics): add a method to RegexMatcherBuilder in matcher.rs, plumb the option through Config, and update the corresponding CLI flag in crates/core/flags/defs.rs.
• Bug in inner-literal extraction: edit literal.rs. Add a regression test against the known-bad regex.
• Performance regressions: re-run benchsuite/ against the canonical corpora before and after.

For the Matcher contract this crate satisfies, see matcher.