Open-Source Wikis

/

fd

/

fd

/

Architecture

sharkdp/fd

Architecture

fd is a single-binary Rust application. It has no daemon, no server, and no plugin system. The whole program is one process that parses command line options, walks the filesystem in parallel, applies a chain of filters to each entry, and either prints matches or runs commands against them.

This page describes the major components in src/ and how a search request flows through them.

High-level flow

graph TD
    user[User shell] -->|argv| main[fn main / fn run<br/>src/main.rs]
    main -->|parse| cli[Opts<br/>src/cli.rs]
    main -->|build patterns| regex_helper[regex_helper.rs]
    main -->|construct| config[Config<br/>src/config.rs]
    main -->|scan paths, regexps, config| walk[walk::scan<br/>src/walk.rs]

    walk -->|WalkBuilder + WalkParallel| ignore_crate[ignore crate]
    walk -->|spawn senders| senders[Worker threads<br/>spawn_senders]
    senders -->|Batch of WorkerResult| chan[crossbeam-channel]
    chan -->|receiver| receive[receive / ReceiverBuffer]

    senders -->|filter| filters[Filters:<br/>FileTypes, extensions,<br/>SizeFilter, TimeFilter,<br/>OwnerFilter, ignore_contain]

    receive -->|no command| stdout[Print to stdout<br/>src/output.rs]
    receive -->|--exec / --exec-batch| exec[CommandSet::execute<br/>src/exec/mod.rs]
    exec --> children[Child processes via argmax::Command]

Components

Module File Responsibility
Entry point src/main.rs fn main and fn run. Parses Opts, builds regular expressions, constructs Config, calls walk::scan. Also picks tikv-jemallocator as the global allocator on supported targets.
CLI definition src/cli.rs The Opts struct (clap Parser), all flag definitions, the custom Exec parser for -x/-X, enums (FileType, ColorWhen, HyperlinkWhen, StripCwdWhen), and helpers like default_num_threads.
Configuration src/config.rs The Config struct that bundles every effective setting after CLI parsing and smart-case detection. Read-only at search time.
Regex helpers src/regex_helper.rs pattern_has_uppercase_char (smart case detection) and pattern_matches_strings_with_leading_dot (the "you probably need --hidden" hint).
Walker src/walk.rs The parallel walker. Wraps ignore::WalkBuilder/WalkParallel, spawns sender threads that filter entries, and runs a ReceiverBuffer that buffers/streams output or hands results to the exec subsystem.
Directory entry src/dir_entry.rs DirEntry wraps either a normal ignore::DirEntry or a broken symlink path, lazily caches metadata and lscolors::Style.
File-type filtering src/filetypes.rs The FileTypes struct used for -t/--type.
Filters src/filter/ SizeFilter (-S), TimeFilter (--changed-within/--changed-before), OwnerFilter (-o, Unix only).
Format templates src/fmt/ The FormatTemplate and Token types used for both --format output and for --exec placeholder expansion ({}, {/}, {//}, {.}, {/.}).
Command execution src/exec/ CommandSet (one-by-one and batch modes), CommandTemplate, CommandBuilder (uses argmax::Command to stay below ARG_MAX), and the job/batch receiver functions.
Output src/output.rs Path printing with optional colors, hyperlinks, custom path separators, and trailing slashes for directories.
Hyperlinks src/hyperlink.rs OSC 8 file:// URL construction with percent encoding.
Filesystem helpers src/filesystem.rs Cross-platform helpers: absolute paths, is_existing_directory, file-type predicates, OS-string-to-bytes conversion, and default_path_separator for MSYS environments.
Exit codes src/exit_codes.rs The ExitCode enum (Success, HasResults, GeneralError, KilledBySigint) with merge_exitcodes and a Unix re-raise of SIGINT.
Errors src/error.rs A single print_error helper that writes [fd error]: ... to stderr.
  1. Parse arguments. Opts::parse() (clap derive) builds the option struct. If --gen-completions is set, print_completions writes a clap-generated completion script and exits.
  2. Resolve search paths. set_working_dir honours --base-directory, then opts.search_paths() collects positional paths or defaults to the current directory.
  3. Build regular expressions. Each pattern (the main one plus any --and expressions) is converted by build_pattern_regex. Globs are translated through globset::GlobBuilder; --fixed-strings runs through regex::escape. The actual Regex objects are built later, after smart-case detection, by build_regex.
  4. Construct Config. construct_config derives effective settings: smart case sensitivity (via pattern_has_uppercase_char), path separator, time/size/owner constraints, color decision (terminal detection plus NO_COLOR), LsColors (env or the bundled DEFAULT_LS_COLORS), hyperlink mode, file-type set, extensions (compiled into a RegexSet), --format template, and the CommandSet for --exec/--exec-batch/--list-details.
  5. Sanity checks. ensure_search_pattern_is_not_a_path rejects patterns like /etc/passwd (without --full-path) and ensure_use_hidden_option_for_leading_dot_pattern warns when a pattern only matches dotfiles but --hidden was not passed.
  6. Walk. walk::scan constructs a WorkerState, builds an ignore::WalkParallel walker that respects ignore files and --exclude, spawns one sender closure per worker thread (spawn_senders), and runs a receiver in the same thread scope.
  7. Filter and forward. Each sender filters an entry through: ignore_contain (skip directory if it contains a marker file), depth limits, the conjunction of pattern regexes against either the basename or the full path, the extension RegexSet, FileTypes, owner constraints, size constraints, and time constraints. Surviving entries are batched into a Batch and pushed onto a bounded crossbeam-channel.
  8. Receive. ReceiverBuffer starts in Buffering mode, accumulating entries until either the buffer hits 1000 items or max_buffer_time (default 100 ms) elapses. It then sorts the buffered batch and switches to Streaming, writing each subsequent entry directly to stdout.
  9. Or execute. When --exec/--exec-batch/--list-details is set, the receiver path goes through exec::job (one-by-one in a thread pool) or exec::batch (single argv built up via argmax::Command::args_would_fit).
  10. Exit. ExitCode::exit returns 0 for success, 1 for general errors (or --quiet with no match), 1 from the OR-reduced exit codes of child commands, or 130 after re-raising SIGINT on Unix.

Concurrency model

  • Sender side. WalkParallel::run (from the ignore crate) creates config.threads worker threads. Each worker runs the closure built in spawn_senders. The number of threads defaults to std::thread::available_parallelism() clamped at 64 (default_num_threads in src/cli.rs).
  • Channel. Senders push Batch values onto a bounded channel of capacity 2 * threads, defined in WorkerState::scan. Each Batch holds up to 256 results, or 1 result when running in per-result --exec mode with multiple threads (so batches are spread across exec workers).
  • Receiver side. Without --exec, a single receiver thread runs ReceiverBuffer::process. With --exec and OneByOne mode, a thread-scoped pool of config.threads worker threads each consume the channel and run one command per result. With --exec-batch, a single thread collects all results, then CommandBuilder flushes them in chunks that fit inside ARG_MAX.
  • Cancellation. A shared quit_flag: Arc<AtomicBool> is set by the receiver when it stops (e.g., --max-results reached or stdout broken pipe). A separate interrupt_flag is set by a ctrlc handler; pressing Ctrl-C twice exits immediately, otherwise the walker drains gracefully and the process re-raises SIGINT on Unix to preserve the right exit status.

Global allocator

On most targets, fd uses tikv-jemallocator as the #[global_allocator] (see src/main.rs). Jemalloc is excluded on Windows, Android, macOS, FreeBSD, OpenBSD, illumos, 32-bit musl, and riscv64. The choice is made for performance (see PR #481 and the comment in Cargo.toml).

External crates that do most of the heavy lifting

Crate Used for
clap (with derive, wrap_help, cargo) Command-line parsing and help generation.
ignore Parallel directory walking that respects .gitignore, .ignore, .fdignore, and a global ignore file.
globset Translating --glob patterns into regular expressions.
regex and regex-syntax Pattern matching against file names and AST inspection for smart-case and leading-dot detection.
crossbeam-channel Bounded channels between senders and the receiver.
lscolors and nu-ansi-term LS_COLORS parsing and ANSI styling.
argmax A Command wrapper that knows the platform ARG_MAX so batch exec never overflows.
jiff Date and duration parsing for --changed-within / --changed-before (replaced chrono and humantime in 10.3.0).
etcetera Locating the global ignore file in the platform's config directory.
normpath Path normalisation, especially on Windows.
nix (Unix), libc, faccess Unix-specific syscalls for owner filtering, signal handling, hostnames, and executable detection.
ctrlc Cross-platform SIGINT handler.

For a full dependency list see Cargo.toml and reference/dependencies.

Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.

Architecture – fd wiki | Factory