Debugging

Practical notes for debugging a MinIO server locally and in the field.

Logs

MinIO writes structured logs through internal/logger/. Output destinations are configured under [logger] in the server config (or MINIO_LOGGER_* env vars):

• Console. The default for foreground ./minio server runs.
• File rotation. When --log-dir (or MINIO_LOG_DIR) is set in cmd/server-main.go, the server rotates its own log file with size and gzip controls (--log-size, --log-compress).
• Webhook target. Defined in internal/config/notify/notify_webhook.go and dispatched via internal/event/target/.
• Audit log. Separate from the operational log. Audit events run through cmd/admin-handlers.go and end up in internal/logger/audit/.

Set MINIO_LOG_LEVEL to control verbosity. Look for MINIO_TRACE if you want raw HTTP wire logs.

mc admin for live diagnostics

mc admin trace local                  # streaming HTTP+admin traces
mc admin trace local --verbose        # request/response bodies
mc admin trace local --call all       # internal calls (storage REST, peer REST, locks)
mc admin info local                   # cluster overview
mc admin profile start local --type cpu,heap
mc admin profile stop local           # pulls the pprof archive
mc support inspect local              # collects logs, configs, and pprof for support

The HTTP routes those drive are wired under /minio/admin/v3 in cmd/admin-router.go. Tracing is implemented in cmd/http-tracer.go.

Profiling

The server exposes pprof at /minio/v2/debug/pprof/* and lets mc admin profile start/stop time-bounded profiles. Internally:

• cmd/admin-handlers.go mounts the pprof handlers behind the admin auth check.
• github.com/felixge/fgprof is used for the additional fgprof endpoint.

Sample workflow:

mc admin profile start local --type cpu,heap,goroutines
# reproduce the issue
mc admin profile stop local           # writes profile.zip
go tool pprof -http=:8081 profile.zip

Common errors

A few error patterns and their canonical fixes:

Symptom	Likely cause	Fix
Drive offline / Read-quorum	A peer is unreachable; the affected set has lost too many drives.	mc admin info to identify the drive; ensure the peer host is up; check firewall and DNS.
Drive failed: format.json missing	A drive is brand new or its format file got deleted.	The new-disks heal worker (cmd/background-newdisks-heal-ops.go) will reformat and rebuild it. Or run mc admin heal -r.
Object size mismatch / bitrot	Silent disk corruption.	mc admin heal on the bucket; consider replacing the drive.
Replication failures for a target	Network or credentials to the target endpoint changed; target queue is stuck.	mc admin replicate ls or mc admin replicate status; restart the target.
XMinioStorageFull	All drives in the affected set are over the high-water mark.	Add a new server pool with mc admin pool (after writing the new endpoints to config).

Inspecting xl.meta

The mc admin inspect local/bucket/key command (and its server side at cmd/admin-handlers.go) can pull a tarball with the per-drive xl.meta files for an object. To decode locally use the fmt-gen tool baked into cmd/fmt-gen.go or dump with msgp against the type defined in cmd/xl-storage-format-v2.go.

Cluster health checks

• /minio/health/live — process is up.
• /minio/health/ready — all server pools have quorum and are accepting writes.
• /minio/health/cluster — same as ready but harder.

Implementations in cmd/healthcheck-handler.go and cmd/healthcheck-router.go.

Distributed lock troubleshooting

internal/dsync/ exposes a debug HTTP path (/minio/lock/v1/...) that returns lock holders. mc admin trace --call lock shows acquire/release events. The globalLockTimeout and DSync heartbeat are set in cmd/globals.go.

When all else fails

The mc support inspect collector packages logs, current config, and a default profile set. Open an issue (or for AIStor customers, a support case) with the resulting bundle. Also see docs/debugging/ for low-level diagnostic scripts.