Metrics, audit, and tracing

MinIO exposes three classes of observability: Prometheus metrics (two generations side by side), structured audit logs, and live request tracing.

Purpose

• Let operators see what the cluster is doing in real time and over time.
• Provide audit trails for compliance.
• Help debug production issues with mc admin trace without restarting the server.

Metrics

Two endpoints

Endpoint	Style	File entry point
/minio/metrics/v2	One giant collector, AWS-CW-flavoured names.	cmd/metrics-v2.go
/minio/metrics/v3/...	Grouped collectors per concern.	cmd/metrics-v3*.go

The v3 endpoints are organised under sub-paths:

Path	File
/minio/metrics/v3/api	cmd/metrics-v3-api.go
/minio/metrics/v3/audit	cmd/metrics-v3-audit.go
/minio/metrics/v3/cache	cmd/metrics-v3-cache.go
/minio/metrics/v3/cluster/config	cmd/metrics-v3-cluster-config.go
/minio/metrics/v3/cluster/erasure-set	cmd/metrics-v3-cluster-erasure-set.go
/minio/metrics/v3/cluster/health	cmd/metrics-v3-cluster-health.go
/minio/metrics/v3/cluster/iam	cmd/metrics-v3-cluster-iam.go
/minio/metrics/v3/cluster/notification	cmd/metrics-v3-cluster-notification.go
/minio/metrics/v3/cluster/usage	cmd/metrics-v3-cluster-usage.go
/minio/metrics/v3/ilm	cmd/metrics-v3-ilm.go
/minio/metrics/v3/logger/webhook	cmd/metrics-v3-logger-webhook.go
/minio/metrics/v3/replication	cmd/metrics-v3-replication.go
/minio/metrics/v3/bucket/replication	cmd/metrics-v3-bucket-replication.go
/minio/metrics/v3/scanner	cmd/metrics-v3-scanner.go
/minio/metrics/v3/system/cpu	cmd/metrics-v3-system-cpu.go
/minio/metrics/v3/system/drive	cmd/metrics-v3-system-drive.go
/minio/metrics/v3/system/memory	cmd/metrics-v3-system-memory.go
/minio/metrics/v3/system/network	cmd/metrics-v3-system-network.go
/minio/metrics/v3/system/process	cmd/metrics-v3-system-process.go

The v3 router is mounted in cmd/metrics-router.go. The shared types and registry are in cmd/metrics-v3.go, cmd/metrics-v3-types.go, and cmd/metrics-v3-handler.go. v2 and v3 coexist; v2 is retained for clients that scrape the older endpoint.

Real-time and resource metrics

• cmd/metrics-realtime.go powers mc admin metrics (a streaming subset).
• cmd/metrics-resource.go and cmd/os-instrumented.go capture per-process resource usage (gopsutil).

How metrics are produced

Most production code does not call Prometheus directly. Counters and gauges live as fields on the relevant subsystem (e.g. bucketReplicationStats, scannerStats); collectors read those fields when scraped. This keeps the hot path lock-free.

Audit

Audit events are separate from operational logs. Each request produces one audit record describing the auth identity, the API verb, latency, response status, and the target bucket/object.

• Source side: internal/logger/audit/.
• Targets: webhook (default), Kafka, others — share infrastructure with notifications via internal/event/target/.
• Record schema: defined in internal/logger/audit/; user docs at docs/auditlog/.

MINIO_AUDIT_* environment variables enable and configure audit; see Configuration.

Logger

internal/logger/ is the operational log facade:

• Console (default, structured).
• File rotation (set --log-dir / MINIO_LOG_DIR).
• Webhook target.
• HTTP target (push to syslog-equivalent).

Helpers most code uses:

• logger.Info / logger.Error / logger.Fatal.
• logger.LogIf(ctx, err) — non-fatal, logs only if err != nil.
• logger.AuditLog — dispatched to the audit pipeline.

Tracing

cmd/http-tracer.go wraps the request lifecycle with a tracer that pushes events into an in-memory pubsub. cmd/admin-handlers.go exposes /minio/admin/v3/trace as a server-sent stream — that's what mc admin trace consumes.

Trace categories include:

• s3 — every S3 verb.
• internal — admin, KMS, STS.
• lock — DSync acquire/release.
• os — xl-storage system calls (instrumented in cmd/os-instrumented.go).
• replication, healing, scanner.

mc admin trace --call all -v streams everything; flags filter by category.

Health

cmd/healthcheck-handler.go exposes:

• /minio/health/live — process is up.
• /minio/health/ready — write-quorum is up.
• /minio/health/cluster — stricter readiness for distributed setups.

Integration points

• Counters live with their owning subsystem; collectors read them.
• Audit records flow through internal/logger/audit/ to internal/event/target/.
• Tracing uses internal/pubsub/ to deliver events without buffering on disk.

Entry points for modification

• New metric. Add a counter to the owning subsystem, expose it via the relevant metrics-v3-<area>.go collector. Mirror in metrics-v2.go if the metric should remain on the v2 endpoint.
• New audit field. Update the schema in internal/logger/audit/, regenerate any msgp tests, document under docs/auditlog/.
• New trace category. Extend cmd/http-tracer.go and the trace event type in madmin-go.

See Configuration for the matching env vars.