RPC API

Active contributors: Martin Atkins, Liam Cervante, terraform-core team.

Purpose

internal/rpcapi/ is a gRPC surface that lets external tools drive Terraform Core without invoking the CLI. It exists for two main consumers:

1. HCP Terraform uses it to execute stacks plans and applies inside its own infrastructure: a Terraform Core process exposes the rpcapi server and HCP Terraform talks to it over gRPC.
2. IDE plugins, language servers, and other tooling can use it to ask Terraform Core questions (resolve references, fetch schemas, evaluate expressions) without having to invoke terraform and parse its output.

The CLI exposes this server via the hidden terraform rpcapi subcommand, which is intentionally bypassed from the normal command package — see commands.go where rpcapi is built directly from rpcapi.CLICommandFactory and listed in HiddenCommands.

Directory layout

internal/rpcapi/
├── README.md
├── cli.go                  # the `terraform rpcapi` CLI entry point
├── server.go               # gRPC server bootstrap
├── setup.go                # session setup (handshake, plugin sources)
├── stopper.go              # graceful shutdown
├── handles.go              # opaque IDs for client-side resource references
├── packages.go             # source-package handling for stacks
├── plugin.go               # provider/provisioner plugin loading on the server side
├── convert.go              # type conversions cty ↔ DynamicValue
├── grpc_helpers.go         # shared marshaling helpers
├── grpc_testing.go         # test scaffolding
├── internal_client.go      # in-process client for testing
├── stacks.go               # 57 KB — the stacks RPC implementation
├── stacks_grpc_client.go   # client-side helper for stacks
├── stacks_inspector.go     # debug inspector for stacks runs
├── resource_identity.go    # identity feature integration
├── dependencies.go         # 28 KB — module/provider dependency analysis
├── dependencies_provider_schema.go
├── telemetry.go            # OTel instrumentation
├── dynrpcserver/           # dynamic gRPC service registration
├── terraform1/             # protobuf service definitions:
│   ├── dependencies/
│   ├── packages/
│   ├── setup/
│   └── stacks/             # the stacks gRPC service definitions
└── testdata/

Key services

The protobufs in internal/rpcapi/terraform1/ define several gRPC services. Each has its own subdirectory and its own server implementation in the parent.

Service	Purpose	Implementation
Setup	Session establishment, capability negotiation, configuration of the server (plugin sources, working directory).	setup.go
Dependencies	Resolve module/provider dependencies, compute schemas, run static analysis.	dependencies.go
Packages	Manage source bundles used by stacks (configuration archives shipped between client and server).	packages.go
Stacks	Plan, apply, refresh, and inspect stacks remotely.	stacks.go

Each service exposes typed methods; the implementations call into the relevant Core subsystem (internal/configs, internal/getproviders, internal/stacks/stackruntime, etc.) and stream results back over the gRPC channel.

How it works

sequenceDiagram
    autonumber
    participant Client as HCP Terraform / IDE
    participant Server as terraform rpcapi
    participant Setup as Setup service
    participant Stacks as Stacks service
    participant Runtime as stackruntime.Stack

    Client->>Server: connect via gRPC
    Server->>Setup: handshake
    Setup-->>Client: session handle
    Client->>Stacks: OpenStackConfiguration(packageHandle)
    Stacks->>Runtime: parse + validate stack config
    Stacks-->>Client: stackHandle + diagnostics
    Client->>Stacks: PlanStackChanges(stackHandle, plan params)
    Stacks->>Runtime: stackruntime.Plan
    loop streaming
        Runtime-->>Stacks: events (component started, plan ready, …)
        Stacks-->>Client: gRPC stream messages
    end
    Stacks-->>Client: final plan handle
    Client->>Stacks: ApplyStackChanges(stackHandle, planHandle)
    Stacks->>Runtime: apply
    Runtime-->>Stacks: events
    Stacks-->>Client: final state handle

Handles

handles.go defines the opaque ID system. Clients receive opaque handles for things they create (sessions, configurations, plans, stacks). The server maintains a handle table; clients pass handles back when they want to refer to those objects. This is how the API stays stateless from the client's perspective while still being efficient — large objects (like a parsed configuration tree) live entirely on the server.

Streaming

Long operations (plan, apply) emit a stream of events: started, in-progress, log lines, completed-with-changes. The client consumes the stream, allowing real-time progress display in HCP Terraform's UI.

Plugin loading on the server side

When the rpcapi server runs a stack, it needs the same provider plugins the engine would. plugin.go and setup.go accept "package handles" describing where to find the plugins; the server resolves and starts them. This is how the same Terraform Core binary serves both as a local CLI (downloading providers on demand) and as a stacks executor (using pre-staged plugin archives).

Conversions

convert.go and grpc_helpers.go handle the protocol-level translation between cty.Value / tfdiags.Diagnostics / addrs.X and their gRPC equivalents. The wire format for cty.Value is the same DynamicValue used by the provider plugin protocol; see plugin-protocol.

Bypassing the CLI layer

A subtle but important point: rpcapi is wired up in commands.go to bypass command.Meta and the rest of the internal/command package. The CLI factory is rpcapi.CLICommandFactory, not a cli.CommandFactory for a command.RpcApiCommand. The reason: internal/command exists to give humans a friendly interface; rpcapi exists to give automation a direct line to the engine, free of CLI ergonomics like progress bars, prompt handling, and view rendering.

This separation also keeps the RPC surface stable. The command package can change shape without breaking RPC clients, and vice versa.

internal_client

internal_client.go provides an in-process gRPC client used by tests and by some stack-runtime code that wants to drive the API without spinning up a separate server. The same protobuf-defined client/server contract is used; the test just connects via an in-memory listener.

Integration points

• CLI: terraform rpcapi invokes rpcapi.CLICommandFactory(...), which starts a gRPC server on stdin/stdout and waits for a client. Used by HCP Terraform agents.
• Stacks: stacks.go calls into internal/stacks/stackruntime/. This is the major consumer of the API surface.
• Configuration: dependencies.go and packages.go use internal/configs/ for parsing and internal/getproviders/ for resolution.
• Plugins: the server can load provider plugins from package handles or from a regular CLI-style provider source.

Entry points for modification

• Adding a new RPC method: edit the matching .proto file in terraform1/<service>/, run make protobuf, and implement the server method in the parent file (stacks.go, dependencies.go, etc.).
• Adding a new service: create a new subdirectory under terraform1/, define its proto, regenerate, and wire up registration in server.go.
• Changing the in-CLI dispatch: cli.go here, plus the rpcapi entry in commands.go at the repo root.

The RPC API is a public contract for HCP Terraform and is treated with the same backward-compatibility rigor as the JSON output formats. Don't break a method signature without a deprecation cycle.

For the consumer side of this API in stacks, see stacks. For the wire format used to ship cty.Values, see plugin-protocol.