Service and load balancing

A service is the destination of a router. It owns the upstream URLs, the health-checking logic, and the load-balancing strategy. The implementation is split between pkg/server/service (assembly) and pkg/server/service/loadbalancer/* (per-strategy code).

Concepts

In dynamic configuration (pkg/config/dynamic/http_config.go):

type Service struct {
    LoadBalancer *ServersLoadBalancer
    Weighted     *WeightedRoundRobin
    Mirroring    *Mirroring
    Failover     *Failover
}

A service is a tagged union — exactly one of those four shapes is set. The TCP and UDP variants live in tcp_config.go and udp_config.go and are slightly simpler (no mirroring on UDP, for example).

Service kind	Backed by	Notes
LoadBalancer	One or more servers (URL + weight)	The most common case.
Weighted	A weighted graph of other services	Lets you compose services for traffic shaping.
Mirroring	A primary service plus mirror services with percentages	Forks traffic for testing or shadow deployments.
Failover	A primary service plus a fallback service	The fallback is used when the primary is unhealthy.

Service factory

pkg/server/service/service.go (~18k bytes) is the entry point. The Manager type:

type Manager struct {
    services         map[string]http.Handler
    rootCtx          context.Context
    api              http.Handler
    dashboard        http.Handler
    ...
    healthCheckers   map[string]*healthcheck.ServiceHealthChecker
    routinesPool     *safe.Pool
    transportManager TransportManager
}

func (m *Manager) BuildHTTP(ctx, serviceName) (http.Handler, error)

For each service it produces an http.Handler and, if the service has health checks, registers them with pkg/healthcheck. The handler chain it produces:

Service handler → load-balancer wrapper → smart round-tripper → upstream

pkg/server/service/managerfactory.go is the single place where the various manager types are constructed (HTTP, TCP, UDP). It wires the transport manager and the health-check registry.

Load balancers

Each load-balancing strategy is a self-contained subpackage under pkg/server/service/loadbalancer/:

Package	Strategy	Highlights
wrr/	Weighted round robin	Default. Selects servers in proportion to their weight.
p2c/	Power-of-two-choices	Picks two servers at random and uses the one with fewer in-flight requests. Recent addition.
hrw/	Highest random weight (rendezvous)	Hash-based; same hash always picks the same server until topology changes. Used for sticky-by-hash.
leasttime/	Least response time	Picks the server with the lowest measured response time.
failover/	Primary-then-fallback	Routes to a fallback service when the primary is marked unhealthy.
mirror/	Mirror	Sends a fraction of traffic to one or more mirror services without waiting for their responses.

Each strategy implements:

type Balancer interface {
    AddServer(name string, handler http.Handler, weight int)
    RemoveServer(name string)
    Servers() []http.Handler
    SetStatus(ctx context.Context, name string, up bool)
    ServeHTTP(w http.ResponseWriter, r *http.Request)
}

The exact interface varies slightly per strategy, but SetStatus is universal — that's how the health-checker drains a server.

Sticky sessions

pkg/server/service/loadbalancer/sticky.go implements cookie-based stickiness. A cookie keyed by service name pins a client to a backend; the helper in pkg/server/cookie handles encoding and the secure / httpOnly flags. Stickiness can be combined with WRR or HRW.

Health checking

pkg/healthcheck/healthcheck.go runs periodic HTTP probes against each upstream. Failed probes mark the server as down; a configurable number of consecutive successes mark it back up. Status changes call Balancer.SetStatus.

For TCP services, pkg/healthcheck includes a TCP variant that simply opens a connection (integration/tcp_healthcheck_test.go covers it end-to-end).

Smart round-tripper

The HTTP request hand-off to upstream goes through a RoundTripper built by pkg/server/service/smart_roundtripper.go. It wraps:

• The base transport from pkg/server/service/transport.go (TLS, dial timeouts, max idle, HTTP/2, etc.).
• A retry round-tripper if the router has the retry middleware.
• A circuit-breaker wrapper if configured.
• An OpenTelemetry tracing instrumentation.

Reverse proxy

pkg/proxy/ contains two implementations of the actual HTTP proxy:

Package	Use case
pkg/proxy/httputil/	Standard httputil.ReverseProxy. Used for HTTP/2 upstreams, WebSocket, and any case the fast path does not handle.
pkg/proxy/fast/	A custom HTTP/1.x proxy aimed at lower allocation overhead.

pkg/proxy/smart_builder.go decides which to use per service:

• HTTP/2 upstream → httputil.
• WebSocket upgrade or chunked encoding edge cases → httputil.
• Plain HTTP/1.1 with no upgrade → fast.

The decision is encoded in the Builder interface and exercised by pkg/proxy/smart_builder_test.go.

TCP and UDP services

TCP services live under pkg/server/service/tcp/. They behave like HTTP services but proxy raw bytes — no rewriting, no compression. The TCP load-balancer strategies are a subset of the HTTP ones (wrr, failover).

UDP services live under pkg/server/service/udp/. They forward datagrams to a backend with session affinity based on source address.

Servers transport

Static configuration at the top level (ServersTransport, TCPServersTransport) provides defaults for upstream connection behavior — TLS verification, root CAs, max-idle connections, etc. Dynamic configuration (dynamic.ServersTransport) can override these per service.

pkg/server/service/transport.go is the *http.Transport factory that materializes these settings.

Tests

• pkg/server/service/service_test.go exercises service assembly, weighting, and mirroring.
• pkg/server/service/transport_test.go covers TLS verification, custom root CAs, h2c, and HTTP/2 upstreams.
• pkg/server/service/loadbalancer/*/test_* cover each strategy.
• integration/healthcheck_test.go verifies end-to-end health-check + load-balancer behavior under failure.
• integration/websocket_test.go covers the proxy-builder choice for upgrades.

Entry points for modification

• A new load-balancing strategy: add a subpackage under pkg/server/service/loadbalancer/, expose a constructor, and register it in pkg/server/service/service.go where the LoadBalancer switch is built. Add the configuration shape to pkg/config/dynamic/http_config.go.
• A new health-check probe type: extend pkg/healthcheck/healthcheck.go.
• A new transport option: thread it through pkg/config/dynamic/http_config.go (ServersTransport) and pkg/server/service/transport.go.

For TLS aspects of upstreams (mTLS, root CAs, OCSP), see TLS.