detect

Active contributors: Zachary Rice, Richard Gomez

Purpose

detect/ is the heart of Gitleaks. It owns the Detector type, the per-fragment scan loop, allowlist evaluation, baseline filtering, the recursive-decoding loop, and composite-rule (multi-part) handling.

Directory layout

detect/
├── detect.go          # Detector type, DetectSource/DetectContext/Detect, rule loop, allowlist eval
├── baseline.go        # Load + filter against an existing JSON report
├── codec/             # Decoder for base64/hex/percent (separate page)
├── files.go           # Deprecated wrapper over sources.Files (kept for v8 API compat)
├── git.go             # Deprecated wrapper for legacy callers
├── location.go        # Byte-offset → line/column resolver
├── reader.go          # io.Reader → Detector helper
└── utils.go           # Shannon entropy, SCM link generation, terminal printing

Key abstractions

Symbol	File	Description
Detector	detect/detect.go	Holds compiled config, prefilter trie, semaphore, accumulated findings
NewDetector(cfg) / NewDetectorContext(ctx, cfg)	detect/detect.go	Construct a detector
NewDetectorDefaultConfig()	detect/detect.go	Detector preloaded with the embedded default config
Detector.DetectSource(ctx, source)	detect/detect.go	Scan every fragment a source yields, return findings
Detector.DetectContext(ctx, fragment)	detect/detect.go	The main per-fragment scan loop
Detector.detectRule(...)	detect/detect.go	Apply one config.Rule to one fragment
Detector.processRequiredRules(...)	detect/detect.go	Composite-rule logic
Detector.AddFinding(f)	detect/detect.go	Compute fingerprint, apply .gitleaksignore/baseline filters, append
Detector.AddGitleaksIgnore(path)	detect/detect.go	Load a .gitleaksignore file
Detector.AddBaseline(path, source)	detect/baseline.go	Load + remember a JSON baseline
IsNew(finding, redact, baseline)	detect/baseline.go	Field-level equality check used to filter baseline-known findings
shannonEntropy(s)	detect/utils.go	Per-secret entropy computation
createScmLink(remote, finding)	detect/utils.go	Build a clickable URL into the offending line on GitHub/GitLab/etc.
filter(findings, redact)	detect/utils.go	Dedupe overlapping generic vs specific rules and apply redaction
printFinding(f, noColor)	detect/utils.go	Verbose-mode pretty printer
Fragment (deprecated alias)	detect/detect.go	Type alias for sources.Fragment; v9 removes

How it works

The scan pipeline funnels every input through DetectContext. The high-level lifecycle:

sequenceDiagram
    participant Caller as cmd/git.go
    participant Det as Detector
    participant Src as sources.Source
    participant Yield as DetectContext

    Caller->>Det: DetectSource(ctx, source)
    Det->>Src: source.Fragments(ctx, yield)
    loop per fragment
        Src->>Det: yield(fragment, err)
        Det->>Yield: DetectContext(ctx, fragment)
        Yield->>Det: []report.Finding
        Det->>Det: AddFinding(each)
    end
    Det-->>Caller: []report.Finding, err

The main loop in DetectContext

The function in detect/detect.go does this for every fragment:

1. Skip the fragment if its file path matches Config.Path or baselinePath (so the config and baseline files don't trigger themselves).
2. Skip if any global allowlist matches the path/commit.
3. Build a keyword presence map by running the global Aho-Corasick prefilter over the fragment's lowercased content. The trie was built once at Detector construction time from every keyword across every rule.
4. For each rule:
   • If the rule has zero keywords, run it unconditionally.
   • Otherwise, run it only if at least one of its keywords is in the presence map.
5. After all rules have run on the current text, optionally decode encoded segments (base64/hex/percent) and recurse — up to MaxDecodeDepth passes. Decoded values are spliced in place; rule findings against decoded segments are tagged with decoded:<encoding> and decode-depth:<depth> (see features/decoding).
6. Pass the final findings through filter(), which dedupes a generic-rule finding when a more specific rule matched the same line+secret, and applies redaction.

detectRule

For a single rule on a single fragment (detect/detect.go):

1. Honor SkipReport (used by composite rules' auxiliaries).
2. Run rule allowlist commit/path checks; return early if the rule is allowlisted for this commit/path.
3. If Path is set:
   • Path-only rule (no Regex): match the path; produce a file detected: finding.
   • Path + Regex: skip if the path doesn't match.
4. Skip if MaxTargetMegaBytes is set and the fragment exceeds it.
5. Run Regex.FindAllStringIndex over the (possibly decoded) text.
6. For each match:
   • Compute line/column via location() (detect/location.go).
   • Apply encoded-segment overlap logic (codec.SegmentsWithDecodedOverlap, codec.AdjustMatchIndex).
   • Build a report.Finding.
   • Honor inline gitleaks:allow markers (unless IgnoreGitleaksAllow).
   • Extract the secret from the configured SecretGroup (or the first non-empty capture group).
   • Compute Shannon entropy and apply the rule's Entropy threshold if set.
   • Run the rule's content allowlists (regex, stopwords) via checkFindingAllowed.
7. If the rule has RequiredRules, hand the primary findings to processRequiredRules.

Allowlist evaluation

Two helpers split the work:

• checkCommitOrPathAllowed runs at fragment intake, before any regex evaluation. It checks the commit SHA and the file path; with MatchCondition=AND, regex/stopword conditions on the same allowlist are skipped here and re-evaluated later.
• checkFindingAllowed runs against an extracted finding. It checks the rule's content allowlists (regex against match / line / secret, plus stopwords). For AND allowlists, it also re-evaluates commit/path so the full conjunction is checked.

Composite (required) rules

processRequiredRules (detect/detect.go) runs once per primary rule that has RequiredRules. It:

1. Collects findings for every required rule against the same fragment, marking them as InheritedFromFinding to prevent infinite recursion.
2. For each primary finding, checks withinProximity against the collected required findings using the rule's WithinLines / WithinColumns constraints.
3. Drops any primary finding that doesn't have at least one required finding for every required rule.
4. Attaches the satisfied required findings to the surviving primary finding via Finding.AddRequiredFindings.

See features/composite-rules for the user-facing semantics.

Fingerprints, .gitleaksignore, and baseline

AddFinding does three filters in order:

1. Compute the global fingerprint <file>:<rule-id>:<line> and (if commit is present) the commit fingerprint <commit>:<file>:<rule-id>:<line>. Skip if either matches the loaded .gitleaksignore set.
2. If a baseline was loaded, run IsNew(finding, redact, baseline) and skip if equal to a baseline entry.
3. Append under findingMutex. If Verbose is on, pretty-print to stdout.

Integration points

• Receives: sources.Fragment from sources/. The sources.Source interface is the only intake.
• Reads: config.Config (rules, allowlists, ordered rule list, keyword set).
• Writes: appends to internal findings slice; the caller drains it via Detector.Findings() or DetectSource's return value.
• Calls into codec/: codec.NewDecoder, codec.SegmentsWithDecodedOverlap, codec.AdjustMatchIndex, codec.Tags, codec.CurrentLine.
• Calls into cmd/scm/: createScmLink switches on scm.Platform to render finding URLs.
• Concurrency: exposes Detector.Sema (a *semgroup.Group with limit 40). Sources schedule fragment work onto it.

Entry points for modification

• Adding a new per-fragment filter or transformation: insert it into DetectContext before the rule loop or between rules and the decode pass.
• Changing how a finding is built: detectRule is the place. Be careful with loc boundaries when currentRaw differs from fragment.Raw due to decoding.
• Changing fingerprint format: AddFinding defines both fingerprint shapes. Be aware that changing the format silently invalidates every user's baseline and .gitleaksignore; the existing code carefully pre-computes both shapes for compatibility.
• Changing concurrency: the sema is constructed in NewDetectorContext with hardcoded limit 40. If you need a different value, parameterize it.
• Adding a new finding output channel (e.g., a streaming hook): currently the only path is findings slice + verbose stdout. A streaming callback would slot in next to AddFinding.