istio/istio

Observability

Active contributors: hzxuzhonghu, ramaraochavali, howardjohn, keithmattix

What users do

Get RED metrics (Rate / Errors / Duration) for every service-to-service call without instrumenting code.
Get distributed traces with B3 / W3C / Datadog headers and span propagation.
Get access logs at HTTP-level granularity in JSON or text.
Apply per-namespace / per-workload sampling and customization via the Telemetry CRD.

CRDs and configuration

Mechanism	Scope	Used for
`MeshConfig.defaultProviders.{metrics,accessLogging,tracing}`	mesh-wide	Default backends
`MeshConfig.extensionProviders`	mesh-wide	Define backends (OTel, Zipkin, Datadog, Stackdriver, Prom)
`Telemetry` CR	namespace / workload	Override sampling, format, providers per-scope

The Telemetry schema lives at telemetry/v1/telemetry.proto. The behavior model: provider definitions live in MeshConfig; Telemetry resources select which providers apply to which workloads with what overrides.

Metrics

Two paths, depending on data plane:

Sidecar mode — Envoy emits Prometheus metrics on 15090/stats/prometheus. The istio_* family is built up by per-listener stats sinks configured by istiod. Common metrics: istio_requests_total, istio_request_duration_milliseconds, istio_request_bytes, istio_response_bytes, istio_tcp_connections_opened_total.
Ambient mode — Ztunnel emits L4 metrics; waypoint Envoys emit L7 metrics with the same istio_* family.

Stats are configured by:

manifests/charts/istio-control/istio-discovery/files/gen-istio.yaml (default stat sinks).
The Telemetry compiler in pilot/pkg/networking/core/telemetry/ (overrides).
prometheus.io/scrape: true annotation on injected pods (annotation-driven scrape config).

The "stats merging" feature on the istio-proxy container exposes both Envoy and app metrics on a single port (15020) — useful for users who want their app metrics in the same scrape.

Tracing

Istio doesn't generate traces — it propagates context and emits per-hop spans. Configuration:

The user defines a tracing provider in MeshConfig.extensionProviders:

extensionProviders:
  - name: otel
    opentelemetry:
      service: opentelemetry-collector.observability.svc.cluster.local
      port: 4317

A Telemetry resource references the provider and sets sampling rate:

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: otel
      randomSamplingPercentage: 1.0

The compiler is in pilot/pkg/networking/core/tracing.go. Supported providers: Zipkin, Datadog, OpenTelemetry, Stackdriver, Lightstep (deprecated, mapped to OTel).

For traces to be useful end-to-end, the application must propagate the trace headers (x-b3-*, traceparent, x-datadog-trace-id, etc.). Without that, Istio captures only per-hop "spans" with no parent linkage.

Access logging

Configured via:

MeshConfig.accessLogFile (legacy, mesh-wide on/off).
MeshConfig.extensionProviders.envoyFileAccessLog and envoyOtelAls (current).
Telemetry.accessLogging[] (per-scope override).

The compiler is pilot/pkg/networking/core/accesslog.go. Output formats: text (Envoy default), JSON, OTel-encoded.

A useful pattern: turn off access logs mesh-wide via Telemetry with disabled: true, then re-enable per-namespace for the namespaces under investigation. Cuts log volume substantially in large meshes.

Status overhead and proxy_status

Per-pod metrics — every sidecar adds a 15090/stats/prometheus endpoint scraped by Prom. At scale, this is the dominant Prom workload in many clusters. Filtering with metric.tagsToRemove in Telemetry cuts cardinality.
istiod metrics — pilot_xds_pushes, pilot_xds_push_time, pilot_proxy_convergence_time, pilot_total_xds_internal_errors. Surfaced at /metrics on port 15014.
proxy_status — istioctl proxy-status is a cluster-wide health check showing per-proxy ACK state for each xDS type. Backed by the StatusGen generator and the ConfigDump debug endpoint.

Bug reports

istioctl bug-report collects logs, configs, and proxy state into a tarball for a support case. Implementation: tools/bug-report/. Collects:

All istiod pod logs.
Per-injected-pod istio-proxy logs.
Pod ConfigDumps from each Envoy.
Cluster + namespace + service inventory.
IstioOperator and ConfigMap snapshot.

Output is a directory ready to attach to a GitHub issue.

ControlZ and debug

ControlZ — istiod's introspection HTTP server (port 9876 by default). Routes: log levels, metrics, env vars, signals. The library is pkg/ctrlz/.
istiod /debug — read-only HTTP endpoints for snapshots: /debug/syncz, /debug/configz, /debug/endpointz, /debug/edsz, /debug/cdsz, /debug/clusterz. Implementation: pilot/pkg/xds/debug.go.
istioctl proxy-config — pulls a sidecar's ConfigDump and pretty-prints it (istioctl proxy-config listener|route|cluster|endpoint|secret <pod>). Implementation: istioctl/pkg/writer/.

Key source files

File	Purpose
`pilot/pkg/networking/core/tracing.go`	Tracing provider compilation
`pilot/pkg/networking/core/accesslog.go`	Access log compilation
`pilot/pkg/networking/core/telemetry/telemetry.go`	Telemetry CR resolution
`pilot/pkg/model/telemetry.go`	Telemetry index
`pilot/pkg/xds/debug.go`	/debug HTTP endpoints
`pilot/pkg/xds/statusgen.go`	proxy-status generator
`tools/bug-report/`	bug-report CLI
`pkg/ctrlz/`	ControlZ