sources

Active contributors: Zachary Rice, Richard Gomez

Purpose

sources/ is responsible for producing the Fragments that the detector scans. Three concrete implementations cover the three scan modes: git history, filesystem trees, and single readers (which is also where archive extraction lives).

Directory layout

sources/
├── source.go      # Source interface + FragmentsFunc callback type
├── fragment.go    # Fragment struct
├── git.go         # Git: parses `git log -p` / `git diff` output via go-gitdiff
├── files.go       # Files: filepath.WalkDir + per-file enqueue
├── file.go        # File: single io.Reader with archive recursion
└── common.go      # Helpers: shouldSkipPath, isArchive, OS detection, peek-boundary read

Key abstractions

Symbol	File	Description
Source	sources/source.go	The single-method interface every source implements
FragmentsFunc	sources/source.go	Callback signature: func(Fragment, error) error
Fragment	sources/fragment.go	What's fed to the detector: raw content + path + commit info
Git	sources/git.go	Source for git repos; embeds a *GitCmd, *Config, *RemoteInfo, semaphore
GitCmd	sources/git.go	Wrapper around the running git log -p / git diff process
NewGitLogCmd / NewGitLogCmdContext	sources/git.go	Spawn git log -p -U0 --full-history --all --diff-filter=tuxdb
NewGitDiffCmd / NewGitDiffCmdContext	sources/git.go	Spawn git diff -U0 --no-ext-diff (with --staged if requested)
RemoteInfo	sources/git.go	Platform + Url for finding-link generation
NewRemoteInfoContext(ctx, platform, source)	sources/git.go	Detects remote URL, parses host, picks an SCM platform
Files	sources/files.go	Source for directory trees
File	sources/file.go	Source for a single io.Reader; handles archives
CommitInfo	sources/git.go	Author/email/date/message attached to git fragments

How it works

Git source

The git scanner streams git log -p -U0 ... output through gitleaks/go-gitdiff, which emits a *gitdiff.File per changed file. For each file:

graph TD
    Cmd["git log -p / git diff"] -->|stdout| GD[go-gitdiff parser]
    GD -->|*gitdiff.File| Loop[Git.Fragments loop]
    Loop -->|delete| Drop1[skip]
    Loop -->|binary,not archive| Drop2[skip]
    Loop -->|binary,archive| Blob[git cat-file blob]
    Blob -->|io.Reader| FileSrc[File source recursion]
    Loop -->|text| Frags[per-TextFragment yield]
    Frags --> Yield
    FileSrc --> Yield

Key behaviors:

• Only additions are scanned, by virtue of -p -U0 and gitdiff.OpAdd. Deletions and unchanged context lines are not yielded.
• Binary files are skipped unless they look like archives (extension/MIME match through isArchive). Archive blobs are streamed in via git cat-file blob and fed back through the File source, so archive recursion shares one code path between git and dir modes.
• Commit allowlists are checked early, on the gitdiff PatchHeader.SHA, before fragments are yielded.
• Concurrency: each gitdiff file is enqueued onto Git.Sema. The default cap is 40 (set by Detector.Sema in detect/).
• --log-opts: arbitrary git log flags can be appended. The code warns if any element is wrapped in single or double quotes (a common shell-quoting bug; see issue #1153).

Files source

sources/files.go walks a directory tree with filepath.WalkDir. For each file:

• Skip if it's empty.
• Skip if MaxFileSize is set and the file exceeds it.
• Resolve symlinks if FollowSymlinks is true; otherwise skip them.
• Apply shouldSkipPath (config global allowlist).
• Open the file, build a File source for it, and run file.Fragments.

The work is dispatched through the same Sema so concurrency is bounded the same way as for git.

A DirectoryTargets helper at the top of the file is marked deprecated; it predates the Files source and exists for v8 API stability.

File source

sources/file.go is the workhorse for any single io.Reader. It:

1. Tries to identify the content as an archive via mholt/archives (archives.Identify).
2. If it's an archive and archiveDepth+1 ≤ MaxArchiveDepth, extract or decompress and recurse into each inner file. Inner paths are joined with ! (InnerPathSeparator). Archive depth is incremented on each recursion.
3. Otherwise, read the underlying reader in 100KB chunks (defaultBufferSize). Each chunk is extended forward to a "safe boundary" (whitespace) by readUntilSafeBoundary to avoid splitting a secret across two fragments.
4. Detect binary content via h2non/filetype on the first chunk; if the MIME is application/*, skip the rest of the file.
5. Yield each chunk as a Fragment with StartLine set to the running newline count.

On Windows, File.Fragments populates both FilePath (with /) and WindowsFilePath (with \) so allowlists keyed on either separator continue to work.

Fragment

sources/fragment.go defines the data the detector receives:

type Fragment struct {
    Raw                  string         // text content
    Bytes                []byte         // optional byte view
    FilePath             string         // forward-slash path
    SymlinkFile          string
    WindowsFilePath      string         // back-slash path on Windows; "" elsewhere
    CommitSHA            string         // legacy; v9 plans to use CommitInfo.SHA only
    StartLine            int
    CommitInfo           *CommitInfo    // git mode only
    InheritedFromFinding bool           // marker for composite-rule recursion
}

Remote info and SCM platforms

NewRemoteInfoContext runs git ls-remote --get-url to find the origin URL, normalizes SSH-style URLs (git@host:owner/repo) into HTTPS, and asks platformFromHost to map the hostname to one of:

• github.com → scm.GitHubPlatform
• gitlab.com → scm.GitLabPlatform
• dev.azure.com / visualstudio.com → scm.AzureDevOpsPlatform
• gitea.com / code.forgejo.org / codeberg.org → scm.GiteaPlatform
• bitbucket.org → scm.BitbucketPlatform

If the user passed --platform, that wins. The result is later consumed by createScmLink in detect/utils.go to build Finding.Link.

Integration points

• detect/ drives every source through Detector.DetectSource(ctx, source).
• cmd/ instantiates the right source based on the subcommand: Git for gitleaks git, Files for gitleaks dir, File for gitleaks stdin and gitleaks git --pre-commit.
• config/ is consulted by shouldSkipPath (global path allowlist) before files are opened.
• go-gitdiff is the parser for git log -p output; archive identification uses mholt/archives; binary detection uses h2non/filetype.

Entry points for modification

• Adding a new source (e.g., a network blob store): implement the Source interface. Yield Fragments with FilePath set; if you have file-like content, consider building a File source per blob and calling file.Fragments to inherit archive handling for free.
• Changing chunking: defaultBufferSize and readUntilSafeBoundary (sources/common.go peek-boundary logic) control how large each fragment is. Larger fragments mean fewer prefilter passes but more memory.
• Changing archive support: archive detection is delegated to archives.Identify; supported formats are whatever mholt/archives recognizes. The recursion depth is MaxArchiveDepth and is enforced inside File.Fragments.
• Changing git invocation: NewGitLogCmdContext builds the argv. The --diff-filter=tuxdb choice is deliberate (it includes added/modified/copied/typechange files but excludes pure deletes); changing it without care can affect performance and findings.