Plugin protocol

Active contributors: Martin Atkins, James Bardin.

Purpose

Terraform talks to providers, provisioners, and (recently) state-storage plugins over gRPC. This subsystem is the host side of those conversations: handshake, message marshaling, schema fetching, and version negotiation. Two protocol versions are supported simultaneously; this page covers both.

Source of truth

The wire format is defined in:

• docs/plugin-protocol/tfplugin5.proto — protocol v5.
• docs/plugin-protocol/tfplugin6.proto — protocol v6.

Both proto files are accompanied by docs/plugin-protocol/object-wire-format.md, which explains how Terraform's cty.Value is encoded into the protobuf DynamicValue field.

The generated Go stubs are committed:

• internal/tfplugin5/tfplugin5.pb.go, tfplugin5_grpc.pb.go
• internal/tfplugin6/tfplugin6.pb.go, tfplugin6_grpc.pb.go

Regenerated via make protobuf, which invokes tools/protobuf-compile/.

Directory layout

internal/plugin/                    # protocol v5 host
├── discovery/                      # legacy: filesystem-based plugin discovery
├── grpc_provider.go                # GRPCProvider — provider proxy
├── grpc_provisioner.go             # GRPCProvisioner — provisioner proxy
├── plugin.go                       # ServeOpts and serve helpers
├── ui_input.go, ui_output.go       # UI bridge (rare; provisioners use these)
├── *_test.go

internal/plugin6/                   # protocol v6 host
├── grpc_provider.go                # protocol-v6 provider proxy
├── plugin.go
└── *_test.go

internal/providers/                 # provider-agnostic interfaces
├── interface.go                    # providers.Interface
├── mock.go                         # providers.MockProvider used in tests
├── doc.go
└── ...

internal/provisioners/              # provisioner-agnostic interfaces
├── interface.go
└── ...

internal/grpcwrap/                  # adapt providers.Interface ↔ gRPC server stubs

docs/plugin-protocol/
├── tfplugin5.proto, tfplugin6.proto
├── object-wire-format.md
└── releasing-new-version.md

Key types

Type	File	Description
providers.Interface	internal/providers/interface.go	Protocol-version-agnostic provider interface. Methods include GetProviderSchema, ConfigureProvider, ValidateResourceConfig, PlanResourceChange, ApplyResourceChange, ReadResource, ImportResourceState, UpgradeResourceState, ReadDataSource, OpenEphemeralResource, RenewEphemeralResource, CloseEphemeralResource, CallFunction, Stop, Close.
providers.GetProviderSchemaResponse	internal/providers/interface.go	The provider's declared schemas: provider config, per-resource, per-data-source, per-action, per-ephemeral.
providers.Factory	internal/providers/interface.go	func() (providers.Interface, error). Built from a downloaded plugin binary.
plugin.GRPCProvider (v5)	internal/plugin/grpc_provider.go	Implements providers.Interface by translating into v5 gRPC calls.
plugin6.GRPCProvider (v6)	internal/plugin6/grpc_provider.go	Same, for v6.
providers.MockProvider	internal/providers/mock.go	Replaces a real provider in tests, exposing per-method hook fields.
provisioners.Interface	internal/provisioners/interface.go	Same shape, much smaller surface, for provisioners.

How it works

sequenceDiagram
    autonumber
    participant Engine as terraform.Context vertex
    participant ProvIf as providers.Interface
    participant Wrap as plugin/plugin6.GRPCProvider
    participant Plugin as provider plugin process
    participant Cty as cty value codec

    Engine->>ProvIf: PlanResourceChange(req)
    ProvIf->>Wrap: forward
    Wrap->>Cty: encode prior_state, config, proposed_new
    Cty-->>Wrap: tfplugin5.DynamicValue / tfplugin6.DynamicValue
    Wrap->>Plugin: gRPC PlanResourceChange
    Plugin-->>Wrap: planned_state DynamicValue + diags
    Wrap->>Cty: decode planned_state
    Wrap-->>ProvIf: PlanResourceChangeResponse
    ProvIf-->>Engine: response

Two protocols, one interface

The engine never sees the gRPC stubs. It interacts exclusively with providers.Interface (and provisioners.Interface). The two GRPCProvider types in internal/plugin/ and internal/plugin6/ adapt those interfaces to their respective wire protocols.

This means:

• Provider authors choose which protocol version to implement (most use v5; newer features require v6).
• The engine code is protocol-version agnostic. A given provider just is a providers.Interface.
• Switching a provider's protocol version is invisible to the engine; only the loader changes which GRPCProvider to instantiate.

The decision happens at handshake time. hashicorp/go-plugin's Plugins map advertises both v5 and v6 servers; the provider binary picks whichever it implements.

What v6 adds

Compared to v5, protocol v6 adds:

• Server capabilities — providers declare optional features (e.g. plan_destroy support) so the host can avoid calling unsupported methods.
• Deferred actions — the provider can tell the host "I can't fully plan this yet," letting the host record a deferred change.
• Identity — a stable resource identifier separate from the address.
• Ephemeral resources — resource types whose values exist only for the duration of one operation.
• Functions — provider-defined functions callable from the language.
• Actions — first-class non-resource operations a provider exposes.
• List resources — a new resource mode supporting collection-style data sources for query blocks.

The engine's providers.Interface is the union of both protocols. Methods that v5 doesn't support return a "not implemented" error from the v5 wrapper. Code that uses these features generally checks GetProviderSchemaResponse.ServerCapabilities first.

Wire encoding for cty

cty.Value is encoded into the DynamicValue field as messagepack bytes. The format is documented in docs/plugin-protocol/object-wire-format.md. Marks (sensitive, ephemeral) are not sent over the wire; the host strips them before encoding and re-applies based on schema metadata returned by the provider.

The engine has helpers in internal/configs/hcl2shim/ and internal/plans/dynamic_value.go for the encoding side.

internal/grpcwrap/

This is the host-side helper that wraps a providers.Interface as a gRPC server. It's primarily used in tests and in the test-framework's mock-provider system, where Terraform spins up an in-process gRPC server that adapts a Go providers.Interface to whatever the configuration's wire side expects.

Plugin discovery

Two discovery paths exist:

• Modern: terraform init downloads providers via internal/getproviders. The resulting binaries live in .terraform/providers/.... The engine spawns them via hashicorp/go-plugin from there.
• Legacy: internal/plugin/discovery/ walks well-known directories (e.g. .terraform.d/plugins/) for SDKv1 providers. Used by older provider development workflows.

hashicorp/go-plugin handles the actual subprocess management — process supervision, gRPC handshake, stdout/stderr forwarding for log lines, and unix-socket-based connection setup.

Reattach

For provider development, the host can be told to connect to an already-running provider rather than spawning one. The TF_REATTACH_PROVIDERS env var carries a JSON describing each provider's PID and socket address; internal/getproviders/reattach/ parses it. See debugging.

Stop semantics

The Stop() method on providers.Interface is called when the user hits Ctrl-C. Implementations of GRPCProvider translate this to a gRPC Stop call, which the provider can use to cancel its in-progress work. Whether a provider actually honors this is up to the provider.

Integration points

• Engine: every vertex that does provider-affecting work calls into providers.Interface. Vertices retrieve the provider via EvalContext.Provider(addrs.AbsProviderConfig), which goes through a per-walk cache so that providers are configured once.
• Mock providers: tests use providers.MockProvider directly without the gRPC layer.
• CLI: terraform providers schema -json (internal/command/providers_schema.go) walks every provider in the configuration and dumps its schema; this is the easiest way to inspect the wire format from the user side.
• Provisioners: internal/provisioners/ and internal/builtin/provisioners/ form a smaller parallel of the provider host. The built-in provisioners (file, local-exec, remote-exec) talk to the engine through the same gRPC plumbing as if they were external plugins.

Entry points for modification

• Adding a method to the interface: extend providers.Interface, regenerate the protocol stubs (after editing the .proto), and update both GRPCProvider types. Choose carefully: a v5 vs v6 only feature requires the v5 wrapper to error or no-op.
• Adding a new wire field: edit the .proto, run make protobuf, update the host-side encoder.
• Adding mock-provider support for a new method: extend providers.MockProvider with a hook field and the matching default behavior.

The plugin protocol is one of Terraform's most stable surfaces; backward-compatibility for providers is non-negotiable. Don't change v5 semantics; for new features, extend v6 (and consider whether the provider-side SDK (terraform-plugin-go) needs corresponding updates first).