How to monitor

Logging

Where logs go

The agent and operator log to stdout/stderr in structured JSON or key-value form (configurable). They are picked up by whatever logging stack the cluster uses (Fluent Bit, Loki, Stackdriver, ...).

Conventions

• Built on log/slog via pkg/logging/.
• Each subsystem keeps a logger annotated with subsys=<name>. Subsystem names are constants in pkg/logging/logfields/.
• Common fields: endpoint, identity, ipAddr, podName, policyKey, error.

Log levels

• --debug — turns on debug-level for the agent.
• --debug-verbose — feature-area knobs (policy, kvstore, flow, datapath, ...).
• Per-subsystem verbosity is controllable at runtime via the agent REST API (/v1/config).

Useful queries

kubectl -n kube-system logs ds/cilium | grep "level=ERROR"
kubectl -n kube-system logs ds/cilium | grep "subsys=daemon" | grep "Started"
kubectl -n kube-system logs ds/cilium | grep "endpoint=12345"

Metrics

Endpoints

• cilium-agent exposes Prometheus metrics on port 9962 (configurable). Path: /metrics.
• cilium-operator exposes on 9963.
• hubble-relay exposes on 9966.
• Hubble's per-flow metrics are emitted at 9965.

Metric families

• Agent self-health: cilium_agent_*, cilium_controllers_*, cilium_endpoint_*, cilium_identity_*.
• Policy: cilium_policy_* counts of rules, computations, regenerations.
• K8s clients: cilium_k8s_client_* request rates and latencies.
• kvstore: cilium_kvstore_* operations, errors, quorum.
• Datapath: cilium_drop_count_total, cilium_forward_count_total, cilium_proxy_*.
• NAT / conntrack: cilium_nat_*, cilium_ct_* map sizes and pressure.
• Hubble flow metrics: hubble_*_total, hubble_*_seconds. See features/hubble-observability.md.
• Custom from features: BGP, LB-IPAM, encryption, FQDN.

Dashboards

The community publishes Grafana dashboards under install/kubernetes/cilium/files/:

• Agent dashboard.
• Operator dashboard.
• Hubble dashboard.
• DNS dashboard.

Adding a metric

pkg/metrics/ is the registry. Each cell that produces metrics defines them as Prometheus collectors and registers them through cell.Provide.

Tracing

Cilium does not export distributed traces by default. The Hubble flow stream provides per-flow telemetry that approximates traces — flows include drop reasons, latency for L7 requests, and trace points labelled in the dataplane (trace_reason, trace_observation_point).

For request-level tracing, use Envoy with the standard tracing extensions (Jaeger, Zipkin, OpenTelemetry). Configure them through CiliumEnvoyConfig.

Alerting

There is no built-in alerting layer; cluster operators wire alerts off the metrics above. Recommended alerts:

Alert	Signal
Agent unhealthy	cilium_agent_health_status{type="error"} > 0
Endpoint regeneration storm	rate(cilium_endpoint_regenerations_total[5m]) > X
Drop spike	rate(cilium_drop_count_total[5m]) exceeds baseline.
Policy compute slow	cilium_policy_regeneration_time_stats_seconds p99 high.
kvstore unhealthy	cilium_kvstore_quorum_errors_total > 0 over 5m.
BGP session flap	cilium_bgp_session_state change rate.
Conntrack near full	cilium_ct_entries_* approaching map size.
Hubble Relay unreachable	up{job="hubble-relay"} == 0.

The Documentation/operations/troubleshooting/ tree has more.

How cilium status ties it together

cilium-cli status aggregates:

• DaemonSet readiness from the K8s API.
• Per-pod agent health from the REST API.
• Cluster Mesh peer state.
• Hubble enablement and Relay reachability.
• BGP, encryption, IPAM stats.

It is the first command to run when triaging.

Key source files

File	Purpose
pkg/metrics/	Metric registry.
pkg/logging/	Logger setup.
pkg/status/	Status reporter consumed by cilium status.
pkg/hubble/metrics/	Hubble Prometheus exporters.
install/kubernetes/cilium/files/	Grafana dashboards.