pkg/proxy and pkg/envoy

Active contributors: jrajahalme, sayboras, mhofstetter, jrfastab, aanm

Purpose

pkg/proxy/ and pkg/envoy/ cooperate to provide L7 enforcement and traffic management:

• pkg/proxy/ is the per-endpoint redirect runtime. It picks proxy ports, programs kernel TPROXY rules, registers redirects for endpoints, and parses Envoy access logs back into Hubble flow events.
• pkg/envoy/ is the Envoy supervisor: bootstrap config rendering, xDS gRPC server, listener / cluster / secret management, custom filter integration.

The two are split because the redirect lifecycle (per-endpoint, fast) lives close to the endpoint manager, while the Envoy fleet management (supervisor, xDS, CRD reconciliation) lives in its own subsystem.

Layout

pkg/proxy/
├── proxy.go               # Proxy interface + dispatch
├── envoyproxy.go          # Envoy redirect registration
├── dnsproxy.go            # bridge to FQDN proxy
├── kafka.go               # Kafka redirect
├── accesslog/             # parse Envoy access logs into flow events
├── endpoint/              # per-endpoint proxy state
├── logger/                # access log writer
├── healthz/               # proxy health
└── ...

pkg/envoy/
├── envoy.go               # supervisor (start child, watch logs, restart)
├── xds_server.go          # xDS gRPC server
├── ciliumenvoyconfig.go   # apply CEC -> xDS resources
├── listener.go            # build per-policy listeners
├── route.go / cluster.go  # xDS resource builders
├── secrets/               # SDS for TLS
├── bootstrap.go           # render envoy bootstrap.yaml
├── xds/                   # xDS protocol helpers (vendored extras)
└── ...

pkg/ciliumenvoyconfig/
└── manager.go             # CEC + CCEC reconciler

Key abstractions

Type	File	Role
Proxy	pkg/proxy/proxy.go	Per-node coordinator across L7 backends.
EndpointInfo	pkg/proxy/endpoint/	Per-endpoint redirect state.
EnvoyProxy	pkg/proxy/envoyproxy.go	Maintains per-endpoint Envoy listeners via xDS.
Envoy	pkg/envoy/envoy.go	Supervises the local Envoy process.
XDSServer	pkg/envoy/xds_server.go	gRPC server for xDS.
CECManager	pkg/ciliumenvoyconfig/manager.go	CRD reconciler.

Per-endpoint redirect

sequenceDiagram
    participant Pol as policy compilation
    participant EP as Endpoint
    participant Proxy as pkg/proxy
    participant Envoy as Envoy via xDS
    Pol->>EP: SelectorPolicy with L7 redirect on port 8080
    EP->>Proxy: register redirect (id, port, parser)
    Proxy->>Proxy: pick proxy port (e.g., 14001)
    Proxy->>Envoy: xDS Listener on 14001
    EP->>EP: write policy map: redirect 8080 -> 14001

The dataplane redirects matching connections to the chosen proxy port via TPROXY (programmed via pkg/proxy/iptables.go or socketlb equivalents). Envoy enforces L7, writes an access log, and the flow returns to the destination via the standard datapath.

Access log parsing

Envoy is configured with a Cilium access log filter that emits a per-request log line containing source/destination identity, L7 metadata, and the policy verdict. pkg/proxy/accesslog/ reads those lines and feeds them into Hubble parsers, so HTTP / gRPC / Kafka flows in hubble observe show method, path, status, latency.

SDS (TLS material)

For TLS termination (Gateway API), keys and certs are streamed to Envoy via SDS. pkg/envoy/secrets/ watches Kubernetes Secrets identified by annotations and pushes them through xDS. This avoids writing keys to disk on the node.

Custom Envoy distribution

The Cilium-flavoured Envoy build lives at cilium/proxy (a fork of upstream Envoy). It includes Cilium-specific filter chains (the cilium.l7policy filter, mTLS bridge). Container images for the embedded Envoy (cilium-envoy) are produced from images/cilium-envoy/.

Integration points

• Endpoint manager: registers / unregisters redirects.
• Policy: L7 rules are turned into proxy redirect setups.
• CRDs: CiliumEnvoyConfig / CiliumClusterwideEnvoyConfig for direct Envoy control.
• Hubble: consumes access logs.

Entry points for modification

• New L7 protocol: implement a parser + filter, add a redirect kind in pkg/proxy/, wire xDS resources in pkg/envoy/.
• New CEC feature: extend pkg/ciliumenvoyconfig/manager.go and the CRD type.
• New SDS source: pkg/envoy/secrets/.

Key source files

File	Purpose
pkg/proxy/proxy.go	Per-node proxy coordinator.
pkg/proxy/envoyproxy.go	Envoy redirects.
pkg/envoy/envoy.go	Supervisor.
pkg/envoy/xds_server.go	xDS server.
pkg/envoy/listener.go	Listener builder.
pkg/ciliumenvoyconfig/manager.go	CEC reconciler.

See systems/envoy-l7-proxy.md for the architecture overview.