codec

Active contributors: bplaxco, Zachary Rice

Purpose

detect/codec/ finds encoded substrings in a fragment, decodes them in place, and tracks the mapping between decoded text and the original encoded location so that findings inside decoded segments can still report accurate file/line positions and tag themselves with the encoding kind.

Directory layout

detect/codec/
├── decoder.go        # Decoder type, Decode() (one pass), findEncodedSegments
├── encodings.go      # registered encodings (percent, unicode, hex, base64) + combined regex
├── segment.go        # EncodedSegment + overlap/merging logic
├── start_end.go      # startEnd helper struct used for ranges
├── ascii.go          # ASCII printability filter
├── percent.go        # percent decoder
├── unicode.go        # unicode escape decoder
├── hex.go            # hex decoder
└── base64.go         # base64 decoder

Key abstractions

Symbol	File	Description
Decoder	decoder.go	Stateful decoder with a memoization map to avoid decoding the same span twice
Decoder.Decode(data, predecessors)	decoder.go	One decoding pass; returns the in-place rewritten string and the new segment list
EncodedSegment	segment.go	A region of original text, its decoded location, the predecessors that produced it, and an encodingKind bitmask
encoding	encodings.go	A registered encoding: kind, regex pattern, decode function, precedence
encodingKind	encodings.go	A bitmask (powers of two) so a segment can record multiple encodings when nested
findEncodingMatches	encodings.go	Runs the combined regex once, attaches each match to its encoding, drops overlapping lower-precedence matches
SegmentsWithDecodedOverlap, AdjustMatchIndex, Tags, CurrentLine	segment.go	Helpers used by detect.detectRule to translate finding offsets back to the original text

How it works

The decoder runs as a loop inside Detector.DetectContext:

graph TD
    Frag[Fragment.Raw] --> Pass1[pass 1: find encoded segments]
    Pass1 --> Splice1[splice decoded values in place<br/>build EncodedSegments]
    Splice1 --> Rules1[run rules on the spliced text]
    Rules1 --> Check{depth < MaxDecodeDepth<br/>and new segments?}
    Check -->|yes| Pass2[pass 2: find segments<br/>passing previous as predecessors]
    Pass2 --> Splice2
    Check -->|no| End[exit loop]

Each pass:

1. findEncodingMatches(data) runs a single combined regex (built in encodings.go's init()) with one named capture group per encoding. The match list is sorted into overlap groups; lower-precedence matches that overlap higher-precedence ones are dropped so a base64-looking substring inside a percent-encoded string won't be double-decoded.
2. Each surviving match is decoded; the resulting EncodedSegment records:
   • original — the byte range in the very first (un-decoded) text
   • encoded — the byte range in the current pass's text
   • decoded — where the decoded value lands after the splice
   • encodings — bitmask of every encoding seen at this site or its predecessors (so nested base64-of-percent gets both bits set)
   • depth — incremented from any predecessor's depth, so the decode-depth:N tag can be attached at the end
3. Decode writes the spliced output by walking the original text and, for each segment, copying everything before the segment, then writing decodedValue, then advancing past segment.encoded.end.

Decoded values are memoized in Decoder.decodedMap so identical encoded substrings don't get re-decoded across passes.

Supported encodings

| Encoding | Pattern | Notes | | -------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------ | | percent | %[0-9A-Fa-f]{2}(?:.*%[0-9A-Fa-f]{2})? | URL-style %XX. Highest precedence because it's partial — only the encoded bytes are decoded, surrounding text is left alone. | | unicode | (?:U\+[a-fA-F0-9]{4}(?:\s | $))+or\\u[a-fA-F0-9]{4}+ | Unicode code points in U+xxxx or \uxxxx form | | hex | [0-9A-Fa-f]{32,} | At least 32 hex chars in a row to avoid matching short identifiers | | base64 | [\w\/+-]{16,}={0,2} | At least 16 chars; standard or URL-safe alphabet |

The order matters: encodings is declared in encodings.go and the higher items get a higher precedence at init() time.

Linking findings back to the original

When detectRule finds a match in currentRaw (the decoded text), it asks the codec helpers:

• SegmentsWithDecodedOverlap(encodedSegments, start, end) — return the segments that overlap this match. If empty, the match is entirely outside any decoded region and offsets are already correct.
• AdjustMatchIndex(segments, [start, end]) — translate the decoded-text offsets back to original-text offsets so Finding.StartLine/EndLine are computed against the user's actual file content.
• Tags(segments) — produces decoded:base64, decode-depth:2, etc., which are appended to Finding.Tags.
• CurrentLine(segments, currentRaw) — returns the decoded line for allowlist regex evaluation.

The recursion limit

Detector.MaxDecodeDepth (CLI flag --max-decode-depth) caps the number of passes. The loop in DetectContext increments currentDecodeDepth at the end of each pass and breaks when it exceeds the cap or when no new encoded segments are found. The default flag value is 5; setting it to 0 effectively disables decoding for that scan.

Integration points

• detect/ is the only consumer. Detector.DetectContext constructs a *Decoder, calls Decode per pass, and uses SegmentsWithDecodedOverlap / AdjustMatchIndex / Tags / CurrentLine from detectRule.
• regexp/ compiles the combined encoding regex; the codec doesn't touch the standard library regexp directly.
• No external dependencies beyond the project's own packages — base64, hex, percent, and unicode decoding are all hand-rolled in *.go files in this directory.

Entry points for modification

• Adding a new encoding: append a new *encoding to the encodings slice in encodings.go. Choose a kind value that's the next power of two (so the bitmask stays additive) and add a name to encodingNames. Implement a decode(string) string function. The combined regex is rebuilt at init() time from the slice, so nothing else needs to change. Add a kind constant and update encodingKind.String/kinds.
• Tuning precedence: precedence is assigned in init() from the slice order (higher index = lower precedence). Reorder encodings to change which encoding wins on overlap.
• Changing the recursion semantics: the loop is in Detector.DetectContext, not here. The codec is stateless across passes except for the memoization map.
• Adding tags: Tags(segments) constructs the decoded:<encoding> and decode-depth:<depth> tags. Adjust there if you want to add a new finding-level marker.