HCP Terraform / Cloud integration

Active contributors: terraform-core team, tf-core-cloud.

Purpose

Two coordinated subsystems implement Terraform's relationship with HCP Terraform (formerly Terraform Cloud):

• internal/cloud/ — the integration that backs the cloud {} configuration block.
• internal/backend/remote/ — the older backend that backs terraform { backend "remote" {} }.

Both speak to HCP Terraform via the hashicorp/go-tfe client and present the same OperationsBackend interface to the rest of Terraform Core. They share an enormous amount of code; you'll see commits that touch both directories together.

Why two

The remote backend is the original (2018-era) integration with Terraform Enterprise / Terraform Cloud. The cloud {} block is the newer (2021+) opt-in form, simpler to configure and with first-class support for the terraform login flow. The team gradually migrated functionality and docs to cloud {} while keeping remote for backward compatibility. As of 1.16-dev, both are still supported; the remote backend is increasingly thin and the cloud integration carries most of the active development.

Directory layout

internal/cloud/
├── README.md
├── cli.go                  # CLI integration helpers
├── backend.go              # 54 KB — the OperationsBackend implementation
├── backend_apply.go        # apply-specific handling (run polling, log streaming)
├── backend_plan.go         # plan-specific handling
├── backend_query.go        # query (1.16) handling
├── backend_show.go         # show against remote run
├── backend_context.go      # build a backend context for a run
├── backend_common.go       # shared helpers
├── backend_taskStages.go   # pre/post task stages
├── backend_taskStage_policyEvaluation.go     # policy enforcement (Sentinel/OPA)
├── backend_taskStage_taskResults.go          # third-party run tasks
├── backend_run_warning.go  # warnings to display when run is queued
├── cloud_integration.go    # coordinating with the cloud block
├── cloud_run.go            # run-creation logic
├── cloud_variables.go      # variable handling
├── cloudplan/              # cloud-stored plan handling
├── e2e/                    # acceptance tests
├── errors.go, retry.go     # error mapping and retry
├── migration.go            # state migration into HCP Terraform
├── state.go                # statemgr.Full implementation backed by HCP
├── test.go                 # test integration
├── testing.go, tfe_client_mock.go  # test scaffolding (~70 KB mock)
└── testdata/               # 46 sub-directories of fixtures

internal/backend/remote/
└── (similar structure, older)

Key types

Type	File	Description
cloud.Cloud	internal/cloud/backend.go	Implements Backend and OperationsBackend for the cloud {} block.
cloud.Cloud.Operation	cloud_run.go	Creates a run on HCP Terraform, polls for completion, streams logs back to the local CLI view.
cloud.RemoteState	internal/cloud/state.go	statemgr.Full implementation that reads/writes state via HCP API.
remote.Remote	internal/backend/remote/backend.go	The older remote backend; mirrors cloud.Cloud's shape.

How it works

sequenceDiagram
    autonumber
    actor User
    participant CLI as command.PlanCommand
    participant Cloud as cloud.Cloud
    participant TFE as HCP Terraform API
    participant Run as Remote Run

    User->>CLI: terraform plan
    CLI->>Cloud: Operation(ctx, op)
    Cloud->>TFE: CreateRun(workspace, configVersion, planOnly=true)
    TFE-->>Cloud: Run handle
    Cloud->>TFE: Subscribe to run events
    loop poll until terminal
        Cloud->>TFE: GetRun(id)
        TFE-->>Cloud: status, plan ID
        TFE-->>Cloud: log stream chunks
        Cloud->>CLI: forward via view
    end
    alt task stages configured
        TFE->>Cloud: pre-plan task stage results
        Cloud->>CLI: render
        TFE->>Cloud: post-plan policy results
        Cloud->>CLI: render
    end
    Cloud-->>CLI: Operation completed (or errored)
    CLI-->>User: exit

Operation lifecycle

The cloud.Cloud.Operation method (in cloud_run.go) is the centerpiece:

1. Configuration version. A tarball of the working directory is uploaded as a "configuration version" so HCP Terraform has the source.
2. Run creation. A new run is created against the target workspace, with parameters reflecting the local CLI options (auto-apply, plan-only, refresh-only, target lists, replace lists, variables).
3. Polling. The run is polled for status changes. Logs are streamed and printed to the local terminal so the user sees the same output as if it ran locally.
4. Task stages. HCP Terraform supports pre-plan, post-plan, pre-apply, and post-apply task stages — extension points where third-party services (Sentinel, OPA, run-task webhooks) can gate the run. backend_taskStages.go and the _policyEvaluation / _taskResults files render their results.
5. Apply. If applicable, the user is prompted to confirm and the apply runs remotely.
6. State. State is fetched from HCP Terraform after the run completes (or, for plan-only runs, the plan itself is retrieved and saved locally).

Variables

cloud_variables.go reconciles three sources of variables: the user's local .tfvars files, -var flags, and workspace-level variables stored in HCP Terraform. The reconciliation rules are subtle — workspace-level vs CLI-provided values have different precedences depending on whether the workspace is in "remote" or "local" execution mode.

Tests with HCP Terraform

The test runner (test.go in internal/cloud/) integrates with HCP Terraform's "test runs" feature. When cloud {} is configured and the user runs terraform test, the runs can be executed remotely with the workspace's saved variables and credentials.

Locks and lineage

cloud.RemoteState implements optimistic locking via HCP Terraform's API. The state's serial and lineage are used the same way as in the local file format — every write increments the serial; lineage UUIDs detect "different state on the other end."

Migration

migration.go handles the user's first transition into HCP Terraform: copying their local state up, mapping local workspaces onto HCP workspaces, and (optionally) deleting the original local file.

Configuration block

A typical cloud {} block:

terraform {
  cloud {
    organization = "my-org"
    workspaces {
      name = "production-network"
    }
  }
}

The matching parser is in internal/configs/cloud.go. Multiple-workspaces variants use workspaces { tags = ["env:prod"] } to select multiple workspaces by tag (used with terraform workspace select).

Login flow

terraform login (internal/command/login.go) opens a browser, has the user authenticate, and writes a token to ~/.terraform.d/credentials.tfrc.json. The token is consumed by svchost/auth (HashiCorp's service-host authentication library) when the cloud integration makes API calls. terraform logout revokes the token.

The browser-launching is wrapped in internal/command/webbrowser/.

Integration points

• CLI: the cloud subcommand (internal/command/cloud.go) is enabled when meta.AllowExperimentalFeatures is true; it's the developer-facing surface for the integration. Day-to-day, users interact via the cloud {} block plus standard subcommands.
• Backends: registered via internal/backend/init/init.go. The remote entry points at internal/backend/remote.New and the cloud integration is constructed via the Cloud field on command.Meta.
• State: cloud.RemoteState is a statemgr.Full like any other.
• Engine: when run remotely, the engine doesn't run locally at all — HCP Terraform spins up its own Terraform Core. The local CLI only orchestrates and renders.

Entry points for modification

• Adding support for a new HCP Terraform feature: usually a new method on cloud.Cloud plus updates to cloud_run.go to surface the result through the CLI.
• Adding a CLI flag that should be forwarded to HCP runs: cloud_run.go builds the RunCreateOptions; add the field there and in the matching argument struct.
• Changing the migration flow: internal/cloud/migration.go and internal/command/meta_backend_migrate.go.

For HCP Terraform-only behaviors (Sentinel, OPA), see HCP Terraform's own documentation; this codebase only consumes their results.