V2 resource framework

internal/resource/, internal/controller/, and internal/storage/ together implement a generic, Kubernetes-influenced resource model. The next-generation Consul APIs (catalog v2, multicluster, mesh v2) are built on this framework rather than the original _endpoint.go pattern.

Purpose

The legacy server pattern wires every API as an XEndpoint struct with hand-written net/rpc handlers, MemDB tables, FSM commands, and HTTP endpoints. That's lots of boilerplate per concept and made cross-cutting features (versioning, watch, ACL hooks) inconsistent.

The v2 framework provides:

• A generic resource service (pbresource.ResourceService) over gRPC: Read, Write, WriteStatus, List, Delete, Watch.
• A resource type registry that maps protobuf message types (<Group>.<Version>.<Kind>) to validation, ACL, and lifecycle hooks.
• A controller framework that runs reconciliation loops similar to Kubernetes operators: watch for resource changes, compute desired state, write status back.
• A storage backend abstraction (Raft today, eventually possibly external) that persists resources versioned by a generation number and resource version.
• Tenancy model with partitions, peers, namespaces, and an ID/UID/Owner graph.

Directory layout

internal/
├── resource/                 # Type registry + ID/UID/tenancy + ACL plumbing
│   ├── registry.go
│   ├── acl.go
│   ├── tenancy.go
│   ├── ...
├── controller/               # Generic reconciler runtime
│   ├── controller.go
│   ├── runtime.go
│   ├── manager.go
│   ├── ...
├── storage/                  # Pluggable storage; Raft adapter included
│   ├── inmem/                # In-memory implementation for tests
│   ├── raft/                 # Production adapter on top of MemDB+Raft
│   ├── ...
├── catalog/                  # V2 catalog: workloads, services, endpoints, health-status (built on resource framework)
├── multicluster/             # V2 multicluster (peering successor)
├── go-sso/                   # OIDC SSO helpers (used by ACL auth-method)
├── protohcl/, resourcehcl/   # HCL ↔ protobuf decode for v2 resource types
├── radix/                    # Internal radix tree implementation (used by storage)
├── tools/                    # Per-package codegen helpers
├── testing/                  # Common test fixtures for v2 code
proto-public/pbresource/      # Public protobuf for the resource service + ID types

Key abstractions

Type	File	Purpose
Type	internal/resource/registry.go	A resource kind (group, version, name); registered globally
Registry	internal/resource/registry.go	Holds every Type and its hooks (validate, ACL, mutate)
ID / UID / Reference	proto-public/pbresource/resource.proto	Stable identifier including tenancy + name + UID
Resource	proto-public/pbresource/resource.proto	The persisted record: ID + Generation + Version + Owner + Data + Status
controller.Controller	internal/controller/controller.go	Defines a reconciler with watches and a queue
controller.Runtime	internal/controller/runtime.go	The dependency injection bag passed to reconcilers
storage.Backend	internal/storage/storage.go	The storage interface (Read/Write/List/WatchList)
RaftBackend	internal/storage/raft/	Production storage adapter
pbresource.ResourceService	proto-public/pbresource/	The gRPC service exposed to operators and SDKs

Resource lifecycle

sequenceDiagram
    participant Client as Client (CLI, SDK)
    participant Service as ResourceService (gRPC)
    participant Registry as resource.Registry
    participant Backend as storage.Backend (Raft)
    participant Ctrl as Controllers

    Client->>Service: Write(resource)
    Service->>Registry: validate type + ACL + mutate
    Service->>Backend: Write(resource)
    Backend->>Backend: bump version, persist via Raft
    Backend-->>Service: stored resource
    Service-->>Client: ack

    Backend->>Ctrl: WatchList event
    Ctrl->>Ctrl: enqueue reconcile(ID)
    Ctrl->>Backend: Read(deps)
    Ctrl->>Ctrl: compute desired children + status
    Ctrl->>Service: WriteStatus(...)
    Ctrl->>Service: Write(child resources, owner=ID)

The owner-relationship lets the framework cascade-delete: when a parent is deleted, every resource whose Owner points at it is removed automatically.

Controllers

Controllers live next to the resources they reconcile, e.g.:

• internal/catalog/controllers/<x> — catalog v2 reconciliation (workload health, endpoints derivation).
• internal/multicluster/controllers/<x> — peering / exporter reconciliation.

Each controller defines:

• Watches — which resource types and which IDs trigger the reconciler.
• Reconciler — the function that does the work.
• A queue with rate limiting / backoff (internal/controller/queue.go).

controller.Manager runs all registered controllers and a single Watch goroutine per resource type.

Storage adapter

The Raft adapter (internal/storage/raft/) plugs into the same Raft instance as the legacy state store. It uses MemDB tables to back the resource store and emits WatchList events to controllers. The inmem/ adapter is for unit tests.

ACL integration

Per-type ACL hooks are registered in internal/resource/. They are called by the resource service before reads and writes, so authorization is centralized rather than spread across many endpoint files.

Public surface

The consul resource ... CLI subcommands (command/resource/) talk to the gRPC service via proto-public/pbresource/. There are HTTP and gRPC variants of every CLI subcommand:

• consul resource list catalog.v2.Service
• consul resource read catalog.v2.Service my-service
• consul resource apply -f resource.hcl
• consul resource delete <type> <name>

internal/protohcl/ and internal/resourcehcl/ decode HCL into the proto messages.

Migration story

V1 endpoints continue to work; the v2 framework runs alongside them. There are bridges where a v1 catalog and v2 catalog need to stay in sync (e.g., agent/consul/leader_registrator_v1.go), but new functionality is increasingly built only in v2.

Integration points

• gRPC server: registered in agent/consul/server_grpc.go.
• Streaming: the resource service has a WatchList streaming RPC that reuses the underlying storage watch.
• CLI: command/resource/ plus the *-grpc variants under each subcommand directory.
• HCL: internal/protohcl/, internal/resourcehcl/.

Entry points for modification

• Add a new resource type: define proto in proto-public/pb<group>/, register with resource.Registry, add validation + ACL hooks, and (if needed) add a controller in internal/<group>/controllers/.
• Add a controller: scaffold under the relevant internal/<group>/controllers/<x>/ and register it with controller.Manager in the boot path.
• Custom storage backend: implement storage.Backend (see inmem/ as the simplest example).
• HCL support for a new field: see internal/protohcl/.