Hubble and hubble-relay

Active contributors: rolinh, glibsm, michi-covalent, gandro

Purpose

Hubble is Cilium's flow observability layer. It comes in three pieces:

• Hubble inside the agent (pkg/hubble/) — attaches to the agent's monitor ring buffer, parses raw datapath events into structured flows, exposes a per-node gRPC API on the local Unix socket.
• hubble-relay (hubble-relay/, pkg/hubble/relay/) — cluster-wide gRPC aggregator. Connects to every Cilium agent's gRPC endpoint and proxies a unified Observer, Peer, and Flow view to clients.
• hubble CLI (hubble/, pkg/hubble/) — a CLI tool that talks gRPC to either an agent or Relay and renders flows interactively.

There is also a separate UI repo (cilium/hubble-ui) that consumes Relay's gRPC API.

Directory layout

hubble/
├── main.go               # CLI entry point
├── cmd/                  # cobra commands (observe, status, ...)
└── pkg/                  # CLI helpers

hubble-relay/
├── main.go               # entry point: hive.New(cmd.Relay)
└── cmd/                  # relay command

The bulk of the implementation lives under pkg/hubble/:

Subpackage	Purpose
pkg/hubble/observer/	The per-node gRPC Observer server.
pkg/hubble/parser/	Decodes raw monitor events into flow protobufs.
pkg/hubble/relay/	Relay-specific code (peer manager, fan-out).
pkg/hubble/peer/	Peer service that lets Relay discover agents.
pkg/hubble/recorder/	pcap-style flow recorder.
pkg/hubble/metrics/	Prometheus exporters for flow counters.
pkg/hubble/exporters/	Optional flow log exporters (file, JSON).
pkg/hubble/dropeventemitter/	Emits Kubernetes events on policy drops.
pkg/hubble/server/	Common gRPC server scaffolding.

How it works

sequenceDiagram
    participant BPF as BPF dataplane
    participant Agent as cilium-agent
    participant HObs as pkg/hubble/observer
    participant Relay as hubble-relay
    participant CLI as hubble CLI
    BPF->>Agent: monitor events (perf ring buffer)
    Agent->>HObs: forward events
    HObs->>HObs: parse + cache (ring buffer)
    Relay->>HObs: gRPC GetFlows stream
    HObs-->>Relay: stream of pb.Flow
    CLI->>Relay: GetFlows
    Relay-->>CLI: aggregated flows

Each agent maintains an in-memory ring buffer of recent flows (pkg/hubble/container/). Relay maintains a long-lived gRPC stream to every agent in the cluster — the peer manager (pkg/hubble/peer/) keeps the peer list in sync.

Flows include rich metadata: identity labels of source and destination, Kubernetes pod and namespace, traffic verdict (forwarded, dropped, audit), drop reason, L7 metadata when applicable. Parsers under pkg/hubble/parser/ decode each subtype: tracing/, threefour/ (L3/L4), seven/ (L7), sock/ (socket).

Integration points

• Monitor: pkg/monitor/agent/ produces the events Hubble parses.
• Identity: every flow is annotated with the source/destination identity using the agent's identity cache.
• Metrics: Hubble metrics are emitted via the agent's /metrics endpoint and a separate :9965 listener (configurable).
• Exporters: flow logs can be written to disk or stdout; UI consumes Relay's gRPC.

Entry points for modification

• New flow fields: edit api/v1/flow/flow.proto, run make -C api proto, then update parsers under pkg/hubble/parser/.
• New filters: extend pkg/hubble/filters/ (each filter is a FlowPredicate).
• New CLI commands: add under hubble/cmd/.
• New metrics: add under pkg/hubble/metrics/<name>/.

Key source files

File	Purpose
hubble/main.go	CLI entry.
hubble-relay/main.go	Relay entry.
pkg/hubble/observer/server.go	gRPC Observer service.
pkg/hubble/parser/parser.go	Top-level event dispatcher.
pkg/hubble/relay/server.go	Relay gRPC server.
pkg/hubble/peer/manager.go	Peer discovery for Relay.

See features/hubble-observability.md for the user-facing capabilities.