cli (grep-cli)

Active contributors: Andrew Gallant

Purpose

crates/cli (published as grep-cli) is a grab-bag of utilities that command-line search applications need but aren't part of the core search engine. It deliberately has no central type; each module solves one small portable problem.

The crate is reused by ripgrep itself, but is also intended for downstream tools.

Directory layout

crates/cli/src/
├── lib.rs           # Re-exports + crate docs
├── decompress.rs    # DecompressionMatcher: detecting compressed files and piping them through helpers (~20K)
├── escape.rs        # escape() / unescape() for byte sequences containing arbitrary bytes
├── hostname.rs      # hostname(): a portable hostname() implementation for terminal hyperlinks
├── human.rs         # ParseSizeError / parse_human_readable_size for "100M", "2.5G", etc.
├── pattern.rs       # InvalidPatternError + helpers for the -e and --file flag
├── process.rs       # CommandReader / CommandError for spawning subprocess preprocessors and decompressors
└── wtr.rs           # StandardStream wrapper around termcolor with line-buffered or block-buffered modes

Key abstractions

Type	File	Description
StandardStream	crates/cli/src/wtr.rs	A termcolor::WriteColor wrapper. stdout() returns line-buffered when stdout is a tty, block-buffered otherwise.
BufferWriter	crates/cli/src/wtr.rs	Multi-thread-safe buffered writer used by search_parallel to keep concurrent writes from interleaving.
is_readable_stdin	crates/cli/src/lib.rs	Cross-platform "is stdin connected to something readable, or is it a tty?"
escape / escape_os / unescape / unescape_os	crates/cli/src/escape.rs	Convert between byte sequences and \xFF-style escape strings.
parse_human_readable_size	crates/cli/src/human.rs	"100K" → 102400, used by --max-filesize.
DecompressionMatcher / DecompressionReader	crates/cli/src/decompress.rs	When -z/--search-zip is set, look up the right helper command (gzip -d, xz -d, ...) for a file extension or magic bytes and pipe through it.
CommandReader	crates/cli/src/process.rs	A Read impl that runs an arbitrary subprocess and reads its stdout. Used both by decompression and by --pre.
hostname()	crates/cli/src/hostname.rs	Returns the host name as a String, used to fill in {host} in hyperlink format strings.
pattern.rs		Helpers for reading patterns from a file (-f) or from -e, including BOM stripping and per-line splitting.

Decompression

DecompressionMatcher recognises files by extension (.gz, .bz2, .xz, .lz4, .zst, .br, ...) and maps each to a command. The mapping is configurable: ripgrep registers defaults like ["gzip", "-d", "-c"] for .gz but the user can override via --pre-glob/--pre. The module's design also lets users disable decompression for specific globs.

DecompressionReader wraps a CommandReader. The subprocess is given the file path on its command line; ripgrep reads the decompressed output as if it were the file's content.

Process spawning

process.rs deliberately handles a few concerns most ad-hoc subprocess code gets wrong:

• The subprocess's stderr is captured so that errors propagate cleanly into ripgrep's error messages instead of leaking onto the user's terminal.
• Broken-pipe is treated as graceful termination so quitting --max-count early doesn't blow up.
• Helper-binary lookups are cached: looking up gzip in $PATH happens once per program run, not once per file.

How it works

graph TD
    Main["rg invocation"] --> WTR["wtr::stdout()"]
    Main --> CmdLine["pattern.rs / escape.rs / human.rs"]
    Main --> Walker["walker yields a path"]
    Walker -->|"if -z and ext matches"| Decompress["DecompressionReader"]
    Walker -->|"if --pre"| Pre["CommandReader"]
    Walker -->|otherwise| File["std::fs::File"]
    Decompress --> CommandReader
    Pre --> CommandReader
    CommandReader --> Searcher
    File --> Searcher

Integration points

• Used by: crates/core/flags/hiargs.rs (for stdout, decompression, hostname, escape), crates/core/flags/defs.rs (for parse_human_readable_size), the -z and --pre paths in crates/core/search.rs.
• Re-exported by: grep::cli.
• Depends on: bstr, globset, log, termcolor.

Entry points for modification

• New decompression format: register the extension and helper command in crates/cli/src/decompress.rs::DecompressionMatcher::default.
• New hyperlink template variable: add a placeholder field in crates/printer/src/hyperlink/ and (if it depends on a system call) plumb the value through grep-cli.
• Subprocess error reporting: process.rs. Error paths there feed directly into the user-visible "stderr from /usr/bin/gzip:" messages.