Decompression and preprocessors

ripgrep can search inside compressed files (-z/--search-zip) and through arbitrary preprocessors (--pre). Both use the same subprocess-spawning machinery in crates/cli.

-z/--search-zip

When -z is set, ripgrep matches each file's path against a built-in extension table. If a match is found, the file is piped through the corresponding helper command and the helper's stdout is fed to the searcher as if it were the file's contents.

Supported formats and default helpers (registered in crates/cli/src/decompress.rs::DecompressionMatcher::default):

Extensions	Helper
.gz, .tgz	gzip -d
.bz2, .tbz2	bzip2 -d
.xz, .txz	xz -d
.lz4	lz4 -d -q -c
.lzma	xz --format=lzma -d
.zst, .zstd	zstd -q -d -c
.br	brotli -d

The helper must exist on $PATH. ripgrep caches $PATH lookups; a missing helper on the first encounter prints a one-time warning and the file is skipped.

--pre and --pre-glob

--pre PROGRAM registers a preprocessor that runs against every file. Optionally restricted with --pre-glob:

rg --pre myextract --pre-glob '*.pdf' --pre-glob '*.docx' 'pattern' .

The preprocessor receives the file path on its argv (and stdin = the file's bytes). Its stdout is fed to the searcher as the haystack. This is how text-extraction toolchains (pdftotext, unzip -p, git log) plug into ripgrep.

How it works

graph TD
    Walker["walker yields path P"] --> Pick{path classifier}
    Pick -->|"matches --pre-glob"| Pre["--pre command(P)"]
    Pick -->|"-z and ext recognised"| Decomp[decompression helper P]
    Pick -->|otherwise| Direct[std::fs::File::open P]
    Pre --> CmdReader[CommandReader]
    Decomp --> CmdReader
    CmdReader --> Searcher
    Direct --> Searcher

Both paths land in crates/cli/src/process.rs::CommandReader, which:

• Spawns the subprocess with Command::new(...).stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::piped()).
• Streams the input file into the subprocess's stdin.
• Hands the subprocess's stdout to the searcher as a Read impl.
• On EOF, waits for the subprocess and surfaces any non-zero exit + stderr as an error message attached to the file path.

This is more careful than ad-hoc Command::output() because:

• Errors are user-formatted ("error: /usr/bin/gzip exited with status 1: stderr: ..."), not lost.
• A broken pipe on the searcher side terminates the helper cleanly.
• stderr from the helper is captured rather than leaked to the user's terminal.

Performance: the Windows fast path

A 15.0.0 perf fix (PERF #2111) avoids resolving helper binaries on Windows when -z is not used. The default-table lookup was previously eager; it is now lazy. This shaved a measurable startup cost on Windows directory scans.

Reference: file & function pointers

What	Where
DecompressionMatcher and the default table	crates/cli/src/decompress.rs
DecompressionReader	crates/cli/src/decompress.rs
CommandReader (general subprocess wrapper)	crates/cli/src/process.rs
--search-zip flag	crates/core/flags/defs.rs
--pre flag	crates/core/flags/defs.rs
--pre-glob flag	crates/core/flags/defs.rs
Where the binary picks decompression vs. direct read	crates/core/search.rs

For the package-level view see packages: cli.

Limitations

• Only one helper per file. If the file is .tar.gz, -z decompresses the gzip but does not also untar.
• The helper has to read from stdin or an argv path. Helpers that need a tty or do random-access seeks won't work.
• --pre and -z are mutually exclusive on a given file: if --pre-glob matches, the preprocessor wins; otherwise -z matchers run.