Envoy and the L7 proxy

Active contributors: jrajahalme, sayboras, mhofstetter, jrfastab, aanm

Purpose

Cilium uses Envoy for L7 enforcement. When a network policy requires HTTP method, gRPC method, Kafka topic, or TLS-aware processing, the dataplane redirects the connection to Envoy. Envoy then makes the L7 verdict (allow/deny/transform) and either forwards the request upstream or rejects it.

Envoy also serves as the Kubernetes Gateway API data plane. Cilium implements GatewayClass cilium, watches Gateway and HTTPRoute resources, and translates them into Envoy listeners and clusters via xDS.

There is no sidecar — one Envoy process runs per node, shared by every endpoint that needs L7 redirection on that node. Envoy is configured live via a control-plane xDS server inside the agent.

Directory layout

pkg/envoy/
├── envoy.go                    # the supervisor that runs envoy as a child process
├── xds_server.go               # xDS gRPC server
├── resource_watcher.go         # watch for CiliumEnvoyConfig changes
├── ciliumenvoyconfig.go        # apply CEC to xDS
├── listener.go                 # listener generation for L7 policy
├── route.go / cluster.go       # xDS resource builders
├── secrets/                    # SDS for TLS materials
├── bootstrap.go                # generate Envoy bootstrap.yaml
└── ...

pkg/proxy/
├── proxy.go                    # selects appropriate proxy (envoy / DNS / kafka)
├── envoyproxy.go               # registration of L7 redirects with envoy
├── accesslog/                  # parses envoy access log into Hubble flow
└── ...

pkg/ciliumenvoyconfig/
├── manager.go                  # high-level CRD-driven controller
└── ...

api/v1/envoy/                   # Cilium's xDS extensions (filter config protos)

How a policy redirects to Envoy

sequenceDiagram
    participant Pod as Source pod
    participant TC as bpf_lxc.c
    participant Pol as policy map
    participant Envoy as Envoy proxy
    participant Agent as cilium-agent (xDS)
    participant Dest as Destination pod

    Pod->>TC: HTTP request
    TC->>Pol: lookup (identity, port, proto)
    Pol-->>TC: redirect to proxy port N
    TC->>Envoy: forward (TPROXY)
    Envoy->>Agent: xDS bootstrap (Cilium filter)
    Agent-->>Envoy: stream listeners + routes
    Envoy->>Envoy: apply L7 policy (HTTP method, path, etc.)
    alt allowed
        Envoy->>Dest: forward upstream
    else denied
        Envoy-->>Pod: 403 / RST
    end

The redirect is set up as follows:

1. The agent compiles policy and sees a rule that requires L7 awareness.
2. It picks a per-endpoint proxy port and updates the policy map: traffic matching that L4 5-tuple is redirected via TPROXY to the proxy port.
3. It tells Envoy via xDS to listen on that port and apply the L7 rule.
4. The kernel TPROXY rule (programmed via pkg/proxy/iptables.go or socket-LB equivalents) hands the connection to Envoy.
5. Envoy makes the L7 decision and writes an access log entry that the agent ingests for Hubble flow visibility.

Embedded vs distroless Envoy

Cilium has historically embedded Envoy as a child process of the agent. More recent releases ship the cilium-envoy distroless image and run Envoy as a separate DaemonSet pod. The agent talks xDS to the local Envoy regardless of whether it runs in-process or as a sibling pod.

The Envoy build itself comes from the cilium/proxy repo (a fork of envoyproxy/envoy with Cilium filters). The matching version is pinned in images/cilium-envoy/.

Custom CEC and Gateway API

Two CRDs let users program Envoy directly:

• CiliumEnvoyConfig (CEC) — namespaced, attaches an arbitrary Envoy configuration to selected services.
• CiliumClusterwideEnvoyConfig (CCEC) — cluster-scoped variant.

The Gateway API integration uses these internally: pkg/gateway-api/ (or successor packages) translates Gateway and HTTPRoute into one or more CEC resources, which then apply via the standard pkg/ciliumenvoyconfig/ reconciler.

SDS (Secret Discovery Service)

TLS materials (private keys, certificates, CA bundles) are streamed to Envoy via SDS. Cilium watches Kubernetes Secrets matching annotated names and pushes them through pkg/envoy/secrets/. This avoids writing keys to disk on each node.

Integration points

• Agent: acts as xDS server.
• Datapath: sets up TPROXY redirect via BPF socketlb or iptables, depending on mode.
• Hubble: parses Envoy access logs into flow events with L7 metadata.
• Identity: Envoy filter chain receives the source identity via the Cilium filter config so it can enforce identity-aware L7 policy.

Entry points for modification

• New L7 policy semantics: extend api/v1/envoy/cilium_filter.proto, regenerate, then update the filter on the proxy side (cilium/proxy repo) and the agent xDS plumbing in pkg/envoy/.
• New Gateway API features: extend the translator under pkg/gateway-api/ and the consequent CiliumEnvoyConfig shape.
• New listener types: see pkg/envoy/listener.go and pkg/envoy/route.go.

Key source files

File	Purpose
pkg/envoy/envoy.go	Envoy supervisor.
pkg/envoy/xds_server.go	xDS gRPC server.
pkg/envoy/listener.go	Build per-policy listeners.
pkg/proxy/proxy.go	Per-endpoint redirect coordinator.
pkg/ciliumenvoyconfig/manager.go	CEC reconciler.
images/cilium-envoy/	Cilium-flavoured Envoy image.

See features/network-policy.md and features/service-mesh.md.