tailwindcss-oxide (the scanner)

Active contributors: Robin, Philipp, Jordan

Purpose

crates/oxide/ walks a project, reads source files, extracts every Tailwind class candidate, and returns them as deduplicated strings. It is the engine the JavaScript adapters call into through @tailwindcss/oxide (the NAPI binding in crates/node/).

Directory layout

crates/oxide/
├── Cargo.toml
├── src/
│   ├── lib.rs                # Public re-exports
│   ├── main.rs               # Tiny CLI for local profiling
│   ├── cursor.rs             # Cursor over &[u8]
│   ├── fast_skip.rs          # Per-file early-exit heuristic
│   ├── glob.rs               # Glob handling + optimization
│   ├── paths.rs              # Path helpers
│   ├── throughput.rs         # Bench harness
│   ├── extractor/
│   │   ├── mod.rs                            # Extractor entry point
│   │   ├── machine.rs                        # State-machine trait
│   │   ├── boundary.rs                       # Candidate boundary detection
│   │   ├── bracket_stack.rs                  # Bracket balance helper
│   │   ├── candidate_machine.rs              # Top-level candidate driver
│   │   ├── named_utility_machine.rs          # `bg-red-500`-style names
│   │   ├── named_variant_machine.rs          # `hover:`-style prefixes
│   │   ├── modifier_machine.rs               # `/50`, `/[…]`
│   │   ├── arbitrary_value_machine.rs        # `[#fff]` style
│   │   ├── arbitrary_property_machine.rs     # `[--var:1]`
│   │   ├── arbitrary_variable_machine.rs     # `(--var)`
│   │   ├── css_variable_machine.rs           # `var(--foo)` references
│   │   ├── string_machine.rs                 # `"…"`, `'…'`
│   │   ├── utility_machine.rs                # Generic utility
│   │   ├── variant_machine.rs                # Generic variant
│   │   └── pre_processors/                   # Per-language tweaks
│   ├── scanner/
│   │   ├── mod.rs                            # Scanner driver
│   │   ├── auto_source_detection.rs          # Default scan rules
│   │   ├── detect_sources.rs                 # Resolve globs to file paths
│   │   ├── init_tracing.rs                   # `tracing` setup
│   │   ├── sources.rs                        # SourceEntry plumbing
│   │   └── fixtures/                         # Tests
│   └── fixtures/                             # Tests
├── fuzz/                                     # cargo-fuzz harness
└── tests/
    └── scanner.rs                            # ~1,800 line integration tests

Key abstractions

Type / function	File	Description
Scanner	crates/oxide/src/scanner/mod.rs	The high-level scanner: walker + extractor + dedup + mtime cache.
PublicSourceEntry	crates/oxide/src/scanner/sources.rs	The user-facing { base, pattern, negated } source entry.
Sources	crates/oxide/src/scanner/sources.rs	Internal source-set after normalization.
Extractor	crates/oxide/src/extractor/mod.rs	Runs the chosen pre-processor and state machines over a byte slice.
Extracted	crates/oxide/src/extractor/mod.rs	Output enum: Candidate(span) or CssVariable(span).
Cursor<'a>	crates/oxide/src/cursor.rs	Position-aware view of &[u8].
BracketStack	crates/oxide/src/extractor/bracket_stack.rs	Tracks () [] {} balance during extraction.
Boundary	crates/oxide/src/extractor/boundary.rs	Decides where a candidate ends.
optimize_patterns	crates/oxide/src/glob.rs	Combines and deduplicates glob patterns for the walker.
BINARY_EXTENSIONS_GLOB	crates/oxide/src/scanner/auto_source_detection.rs	The default ignore list for binary file extensions.
ChangedContent	crates/oxide/src/scanner/mod.rs	File(path, ext) or Content(text, ext) — the unit of work.

How a scan works

graph TD
    A[Scanner constructed with sources] --> B[Walker via WalkBuilder]
    B -->|.gitignore + .tailwindcssignore + binary excludes| C[List of candidate files]
    C --> D[Per file: read bytes]
    D --> E[fast_skip.rs heuristic]
    E -->|skip| F[Discard file]
    E -->|keep| G[Pre-processor for extension]
    G --> H[Extractor::extract]
    H --> I[State-machine driver]
    I --> J[Per-shape machine]
    J --> K[Boundary trim]
    K --> L[Push to candidates set]
    L --> M[Return deduped Vec String]

Scanner::scan(&mut self) -> Vec<String> does the full sweep over disk. Scanner::scan_content(input) runs the extractor over an in-memory string (used by Vite/Webpack/PostCSS when they have already loaded a file's contents).

Scanner::get_candidates_with_positions(input) returns each candidate plus its byte position, used by the Tailwind IntelliSense LSP and similar tooling.

The state-machine extractor

Each shape Tailwind needs to recognize has its own state machine. crates/oxide/src/extractor/machine.rs defines the trait:

pub trait Machine {
    fn next(&mut self, c: &mut Cursor<'_>) -> MachineState;
}

Machines return one of Idle, Active, Done(span), etc. The driver in extractor/mod.rs keeps a list of active machines, advances them character-by-character, and emits whichever one completes first. This is how nested shapes work — a hover:[--x] candidate has a named_variant_machine driving an inner arbitrary_property_machine, both sharing one cursor.

The shape machines are intentionally small (200–800 lines each) so contributors can fix one without touching the rest of the extractor.

Pre-processors

Some source languages embed Tailwind classes in syntax that the extractor would otherwise mis-handle:

Pre-processor	Input
haml.rs	Haml templates
pug.rs	Pug indentation-based syntax
razor.rs	Razor / .cshtml
ruby.rs	ERB-style Ruby templates
slim.rs	Slim
svelte.rs	Svelte's class: attribute syntax
vue.rs	Vue templates

Pre-processors transform the byte slice (typically replacing template syntax with neutral whitespace) before the extractor runs.

Auto source detection

When no @source is provided, the scanner falls back to "scan everything that looks like source code." auto_source_detection.rs defines:

• A glob pattern for binary extensions that should always be ignored.
• An additional list of well-known build artifact directories (dist/, target/, node_modules/, etc.).
• Rules for how .gitignore interacts with @source (explicit @source paths override .gitignore).

detect_sources.rs resolves user-provided glob patterns into concrete WalkBuilder arguments.

Caching and incremental scans

The Scanner struct keeps:

• files: FxHashSet<PathBuf> — every file seen.
• dirs: FxHashSet<PathBuf> — every directory walked.
• mtimes: FxHashMap<PathBuf, SystemTime> — populated only after the first scan completes (to keep the initial build fast).
• has_scanned_once: bool — gates the mtime cache.
• candidates: FxHashSet<String> — accumulated candidates across scans.

On the second and subsequent scans, files whose mtime hasn't changed are skipped entirely. New files and modified files are re-extracted; their previously-emitted candidates remain in the set.

Concurrency

The walker uses Rayon for parallelism. WalkBuilder produces an iterator the scanner runs in parallel, accumulating into shared Mutex-guarded sets.

Integration points

• Exported via crates/node/ as the @tailwindcss/oxide npm package.
• lib.rs re-exports Scanner, ChangedContent, GlobEntry, and PublicSourceEntry for external Rust consumers.
• Depends on bstr, globwalk, fast-glob, bexpand, walkdir, dunce, regex, rayon, fxhash, tracing, tracing-subscriber, classification-macros, and the vendored ignore crate.

Entry points for modification

• Add a new shape (e.g. a new arbitrary-value form): create a new *_machine.rs under extractor/, implement Machine, register it in the driver in extractor/mod.rs. Add tests at the bottom of the file.
• Change boundary rules: edit boundary.rs. The boundary character classes are defined as const arrays.
• Add a pre-processor for a new language: create a file under extractor/pre_processors/ and register it by extension in extractor/mod.rs.
• Change auto-source-detection behavior: edit scanner/auto_source_detection.rs. The glob lists and the ignore overrides live there.
• Tune mtime caching: edit Scanner::scan in scanner/mod.rs. The has_scanned_once gate is the key invariant.

For the published-package side of this crate see crates/node.