Reader and chunk list

Purpose

The reader pulls input items into fzf and the chunk list stores them in a way that the matcher can scan without holding any locks. Together they let fzf start streaming results before all input has been read.

Directory layout

src/
├── reader.go         # Reader: stdin, child command, walker, channel
├── reader_test.go
├── chunklist.go      # Chunk + ChunkList; bounded-size append-only storage
├── chunklist_test.go
├── cache.go          # ChunkCache: caches MatchResults keyed on (chunks, pattern)
├── cache_test.go
├── item.go           # Item type (line + ANSI colors + index)
├── ansi.go           # ANSI escape-sequence parser
└── tokenizer.go      # Field tokenizer for --nth / --with-nth / --accept-nth

Key abstractions

Symbol	File	Description
Reader	src/reader.go	Reads from one of: chan string, child command, file system walker, stdin.
Reader.ReadSource(...)	src/reader.go	Top-level entry from core.go. Decides which source to read from.
Reader.feed(io.Reader)	src/reader.go	Common buffered read loop that splits on newline (or NUL with --read0).
Reader.readFiles(...)	src/reader.go	Built-in file system walker; uses github.com/charlievieth/fastwalk.
Chunk, ChunkList	src/chunklist.go	Append-only collection of Item arrays of size chunkSize.
ChunkList.Snapshot(tail)	src/chunklist.go	Returns an immutable snapshot of the current chunks; supports --tail=N truncation.
ChunkList.ForEachItem(fn, done)	src/chunklist.go	Iterates all items under the lock; used for change-with-nth.
ChunkCache	src/cache.go	Caches MatchResults by chunk pointer + pattern; clears retired chunks.
Item	src/item.go	A single input line with parsed text, ANSI colors, original bytes (when --with-nth), and index.

How it works

graph TD
    SRC{Input source}
    SRC -- channel --> CH[readChannel]
    SRC -- FZF_DEFAULT_COMMAND<br/>or initial reload --> CMD[readFromCommand]
    SRC -- TTY stdin --> WALK[readFiles<br/>fastwalk]
    SRC -- piped stdin --> STDIN[readFromStdin]
    CH --> PUSH[chunkList.Push]
    CMD --> FEED[feed]
    WALK --> PUSH
    STDIN --> FEED
    FEED --> PUSH
    PUSH --> CHUNKS[ChunkList<br/>append-only]
    CHUNKS --> SNAP[Snapshot]
    SNAP --> MATCH[Matcher.scan]

Source selection

ReadSource selects an input source based on what was passed in:

1. If inputChan != nil (programmatic embedding), drain it.
2. Else if initCmd is set (initial reload from --bind=start:reload(...) or similar), run that command.
3. Else if stdin is a TTY:
   • If FZF_DEFAULT_COMMAND is set, run that.
   • Otherwise call readFiles with the configured walker roots.
4. Else (piped stdin), call readFromStdin.

Walker safety

readFiles uses fastwalk.Walk for parallel directory traversal. It also implements two safety nets:

• Hidden / ignored directories are skipped without descending.
• Symlink loops that point to a walker root or its ancestor are skipped (the comment in reader.go calls out the WSL/MSYS case where Z: is a symlink to /).

opts.follow, opts.hidden, opts.file, opts.dir, and --walker-skip all flow through readFiles.

Event polling

Reader.startEventPoller runs an internal goroutine that watches a per-Reader atomic event flag. When new items have been pushed, it sets EvtReadNew on the shared EventBox so the coordinator can wake up. The poll interval grows from readerPollIntervalMin to readerPollIntervalMax (in src/constants.go) when no new items arrive — this is the "adaptive throttle" that keeps idle CPU low.

Chunked storage

ChunkList.Push appends an item to the last chunk, allocating a new fixed-size chunk when the previous one is full. Snapshot(tail) returns a copy of the chunk slice (cheap; len(chunks) is small) with the first/last chunks duplicated so the matcher can read them while the reader continues to mutate the live ones.

Snapshot also implements --tail=N: it walks back from the most recent chunk, retains exactly N items, retires the older chunks via ChunkCache.retire, and rewrites the head chunk when the tail boundary cuts through it.

Restart semantics

reload and reload-sync actions call core.go's inner restart closure, which:

1. Calls reader.terminate() to kill the current command.
2. Clears the chunk list and bumps inputRevision.major.
3. Spawns the new command via reader.restart, which goes through the same feed path.

The denylist and pattern caches are cleared when not running in useSnapshot mode.

Integration points

• From core: ReadSource is called once at startup, then restart is called for every reload / reload-sync.
• To matcher: the matcher reads the snapshot returned by ChunkList.Snapshot directly; no message passing needed.
• To EventBox: the reader sets EvtReadNew and EvtReadFin; the coordinator polls these.
• To options: opts.WalkerRoot, opts.WalkerOpts, opts.WalkerSkip, opts.ReadZero, opts.Filter all configure reader behavior.

Entry points for modification

• Add a new input source: branch ReadSource and call feed (for byte streams) or pusher (for already-split items). Don't forget to update EvtReadFin semantics — the coordinator depends on getting exactly one fin event.
• Tune chunk size: chunkSize is in src/constants.go. Changing it has ripple effects on snapshot copy cost and matcher partition granularity. Measure with make bench.
• Walker behavior: add a flag in src/options.go, plumb it into walkerOpts and the fn in readFiles. Symlink and hidden-handling logic is already structured so new exclusions slot in.
• Tail handling: if you change the semantics of --tail, update both ChunkList.Snapshot (storage) and the head-line / header-line accounting in core.go.