Debugging

When something goes wrong in Go itself — a compiler crash, a runtime panic, a flaky test — the project leans on a rich set of in-tree diagnostic knobs. This page is a runbook of the most useful ones.

Crashes and panics in user code

A Go program that panics prints a goroutine traceback. Useful environment knobs:

Variable	Effect
GOTRACEBACK=none	No traceback on crash, only message
GOTRACEBACK=single	Default; traceback for the crashing goroutine
GOTRACEBACK=all	Traceback for every user goroutine
GOTRACEBACK=system	Add runtime goroutines
GOTRACEBACK=crash	Traceback all + abort to produce a core dump
GOTRACEBACK=wer	Windows error reporting integration

Documented in doc/godebug.md and implemented in src/runtime/runtime1.go (the gotraceback parser).

Garbage collector tracing

Set GODEBUG to enable per-cycle GC logs:

GODEBUG=gctrace=1 ./yourprogram

Output is one line per GC cycle written to stderr. Format documented in src/runtime/extern.go and doc/godebug.md. Useful companions:

• GODEBUG=gccheckmark=1 — verify mark phase by re-running it stop-the-world.
• GODEBUG=gcpacertrace=1 — pacer decisions per cycle.
• GODEBUG=allocfreetrace=1 — very verbose; logs every allocation and free.
• GODEBUG=scheddetail=1,schedtrace=1000 — scheduler dumps every 1s.
• GODEBUG=schedtrace=100 — short scheduler line every 100ms.

Memory leaks

go test -memprofile=mem.out ./pkg
go tool pprof mem.out

For long-running programs, expose net/http/pprof and read /debug/pprof/heap. The implementation is in src/runtime/pprof/ and src/net/http/pprof/.

To find specifically which goroutines are stuck:

curl http://localhost:port/debug/pprof/goroutine?debug=2

Recent additions in src/runtime/goroutineleakprofile_test.go and supporting code surface goroutine leaks more directly.

Tracer

The Go execution tracer captures detailed runtime events.

import "runtime/trace"

f, _ := os.Create("trace.out")
trace.Start(f)
defer trace.Stop()

Then:

go tool trace trace.out

The backend lives in src/runtime/trace.go, src/runtime/traceruntime.go, and friends. The viewer is cmd/trace.

Compiler diagnostics

When changing or investigating the compiler (src/cmd/compile/):

Flag	What it shows
-gcflags=-m	Inlining and escape analysis decisions for one package
-gcflags='all=-m=2'	Verbose, applied to all packages
-gcflags=-S	Print final assembly
-gcflags=-l -N	Disable inlining and optimizations (better debugger experience)
-gcflags='-d=ssa/check_bce/debug=1'	Print bounds-check elimination decisions
-gcflags=-d=help	List all -d debug knobs
-gcflags=-W	Print compiler IR after typecheck
GOSSAFUNC=funcName go build	Generate ssa.html showing every SSA pass for funcName
-gcflags=-h	Crash compiler on first error (useful for debugging)
-gcflags='-d=checkptr=2'	Extra unsafe.Pointer validation

The ssa.html output (from GOSSAFUNC=...) is the single most useful tool when debugging the SSA passes; it shows every block and every value at every pass.

Linker diagnostics

go build -ldflags='-v=2'           # verbose linking
go build -ldflags='-X=...'         # set string variable
go tool nm binary                  # symbols
go tool objdump binary             # disassembly
go tool addr2line binary           # PC → file:line

Linker source: src/cmd/link/. Most -ld debug flags are listed in cmd/link/doc.go.

delve and source-level debugging

dlv (delve) lives outside this repo (github.com/go-delve/delve) but is the standard interactive debugger. It works on optimized Go binaries via DWARF info, which the linker emits by default. To make stepping easier:

go build -gcflags="all=-N -l"  # disable optimization + inlining

Reproducing a flaky test

The standard idioms:

go test -count=100 -run TestFoo ./pkg          # repeat
go test -race -count=100 -run TestFoo ./pkg    # under race detector
GOMAXPROCS=1 go test -count=100 ...            # serialize goroutines
GOMAXPROCS=8 go test -count=100 ...            # contend more

For deeper investigation, the tracer (above) plus runtime.Callers and runtime.GC() checkpoints are usually enough.

Reproducing a TryBot failure

1. Note the OS/arch the trybot reported (e.g., windows-amd64).
2. Use gomote to spin up a remote builder of the same kind:

   gomote create windows-amd64
   gomote push <id>
   gomote run <id> go/src/run.bash

3. Or replicate locally with GOOS=... GOARCH=... on a cross-compile-only toolchain.

gomote is part of golang.org/x/build/cmd/gomote. Access requires a Google-internal account for some builders.

Useful runtime functions to know about

When debugging from inside Go code:

• runtime.Caller(n) / runtime.Callers — capture call-site PCs.
• runtime.Stack(buf, all) — capture a goroutine traceback into a buffer.
• runtime.GC() — force a GC cycle.
• runtime.GOMAXPROCS(0) — read current GOMAXPROCS.
• runtime.NumGoroutine() — count of live Gs.
• runtime.SetCPUProfileRate, runtime.SetBlockProfileRate, runtime.SetMutexProfileFraction — enable various profilers programmatically.

Documented in src/runtime/extern.go and src/runtime/debug/.

Where to read next

• Testing — running tests and benchmarks.
• Components → Runtime — what scheduler and GC events mean.
• Components → Compiler — what each -gcflags knob controls.
• doc/godebug.md (in-tree) — full list of GODEBUG settings.