Hubble observability

Active contributors: rolinh, glibsm, michi-covalent, gandro

Purpose

Hubble is Cilium's flow observability layer. It produces structured flow events for every connection the dataplane sees, with rich identity / Kubernetes / L7 metadata, and exposes them via gRPC, Prometheus, and a UI.

What you get

• Per-flow gRPC stream — hubble observe shows a real-time tail of flows, with filters (--pod, --namespace, --verdict, --type).
• Service map — the Hubble UI renders an identity-to-identity service graph derived from flow logs.
• Drop reasons — every drop carries a structured reason (policy denied, missing route, conntrack full, FQDN unmatched, ...).
• Prometheus metrics — flow counters, latency histograms, DNS / HTTP / Kafka / gRPC dimensions.
• Kubernetes events — drops can also be emitted as K8s events via pkg/hubble/dropeventemitter/.
• Flow exporters — long-term archival to disk / S3 / fluent-bit via pkg/hubble/exporters/.
• Recorder — pcap capture of selected flows via pkg/hubble/recorder/.

Layered components

See apps/hubble.md for the binaries. The functional layers are:

graph LR
    BPF[bpf/ programs<br/>perf_event_output]
    Mon[pkg/monitor]
    Parser[pkg/hubble/parser]
    Buf[pkg/hubble/container<br/>ring buffer]
    Obs[pkg/hubble/observer<br/>gRPC]
    Relay[hubble-relay]
    UI[Hubble UI]
    CLI[hubble CLI]
    Metrics[pkg/hubble/metrics]
    Prom[Prometheus]

    BPF --> Mon
    Mon --> Parser
    Parser --> Buf
    Buf --> Obs
    Obs --> Relay
    Relay --> UI
    Relay --> CLI
    Parser --> Metrics
    Metrics --> Prom

Flow content

Each pb.Flow (defined in api/v1/flow/flow.proto) carries:

• IP 5-tuple plus protocol.
• Source and destination identity (numeric + labels).
• Source and destination Kubernetes pod and namespace.
• Verdict (FORWARDED, DROPPED, AUDIT, ERROR).
• Drop reason (when applicable).
• L7 metadata (HTTP method/path/status, gRPC method, Kafka topic, DNS query/response).
• Trace context (encrypted, NAT'd, redirected, mocked).
• Time, summary, sock-level data.

Parsers per flow class live under pkg/hubble/parser/:

• threefour/ — L3/L4.
• seven/ — L7 (HTTP, Kafka, gRPC, DNS).
• tracing/ — datapath trace events (used for drop annotation).
• sock/ — socket-level (sockops) events.

Filters and querying

pkg/hubble/filters/ defines composable FlowPredicates. The CLI translates --protocol tcp --pod foo --verdict DROPPED into a server-side filter that runs against each flow as it streams.

Metrics

pkg/hubble/metrics/ ships several optional exporters, enabled by listing them in Helm values (hubble.metrics.enabled):

• dns — DNS query / response counts and latency.
• drop — drop counts by reason and identity pair.
• tcp — TCP retransmits, RST, half-open.
• flow — generic flow counts.
• flows-to-world — egress to outside-cluster destinations.
• http — request rate, latency, response status.
• port-distribution — port histogram.
• icmp — ICMP counts.
• kafka — Kafka topic stats.
• policy — verdict counts by policy direction.

Integration points

• Identity: every flow is tagged with source and destination identity labels via pkg/identity/cache/.
• Kubernetes: pod and namespace come from the agent's pod cache.
• Envoy: L7 access logs flow into Hubble parsers (pkg/proxy/accesslog/).
• Drop events: pkg/hubble/dropeventemitter/ writes K8s events for select drops.

Entry points for modification

• New parser type: add a sub-package under pkg/hubble/parser/ and register it in the parser dispatch.
• New metric: under pkg/hubble/metrics/<name>/.
• New filter dimension: pkg/hubble/filters/.
• New flow field: edit api/v1/flow/flow.proto, regenerate, update parsers.

Key source files

File	Purpose
api/v1/flow/flow.proto	Flow message.
pkg/hubble/parser/parser.go	Top-level parser.
pkg/hubble/observer/server.go	Per-node gRPC server.
pkg/hubble/relay/server.go	Cluster-wide aggregator.
pkg/hubble/metrics/	Prometheus exporters.
pkg/hubble/exporters/	Flow log archival.

See apps/hubble.md for the binaries that surface this data.