Language evaluation

Active contributors: Martin Atkins, James Bardin, Alisdair McDiarmid.

Purpose

internal/lang/ is everything that happens between "parsed configuration" and "concrete cty.Value". It evaluates HCL expressions in the context of an in-progress plan or apply, exposes the function library (max, for, templatefile, jsonencode, …), tracks references between objects, and threads value marks (sensitive, ephemeral) through every operation.

Directory layout

internal/lang/
├── eval.go              # Scope.EvalBlock / EvalExpr; the main entry points
├── scope.go             # lang.Scope — per-evaluation state
├── functions.go         # the master function table
├── function_results.go  # caching of function results
├── data.go              # the EvalData interface for resolving references
├── checks.go            # lang.Scope helpers for check / postcondition / precondition expressions
├── blocktoattr/         # transforms NestingSingle blocks into attributes for legacy compat
├── ephemeral/           # ephemeral-value support
├── format/              # cty value pretty-printing
├── funcs/               # one file per built-in function
├── globalref/           # cross-module reference analysis
├── langrefs/            # parsing references from HCL traversals
├── marks/               # value marks (Sensitive, Ephemeral, Deprecated, …)
└── types/               # shared cty type helpers

Key types

Type	File	Description
lang.Scope	scope.go	Holds an EvalData, the current cty.Path, the available functions, the static traversal validator, and a few flags. One per evaluation context (per-module, per-vertex).
lang.EvalData	data.go	Interface that knows how to resolve a addrs.Reference into a cty.Value. Implemented by the engine's EvalContext so that lang itself doesn't depend on engine internals.
lang.References	langrefs/references.go	Parses traversals out of HCL bodies and expressions to produce []*addrs.Reference. Used by graph builders to know what depends on what.
lang.ReferencesInBlock, ReferencesInExpr	wrappers in langrefs/	Convenience layers over References.
funcs.MakeFunctionsTable	functions.go	Builds the master map[string]function.Function exposed to expressions.
marks.Sensitive, marks.Ephemeral, marks.Deprecated	marks/marks.go	The cty marks used throughout Terraform.

How it works

graph LR
    Cfg[hcl.Expression / hcl.Body from configs] --> Scope[lang.Scope]
    Engine[engine EvalContext] --> Data[implements lang.EvalData]
    Data --> Scope
    Scope --> Refs["References(expr) → []*addrs.Reference"]
    Scope --> Eval["EvalExpr / EvalBlock"]
    Eval --> Vars["assemble traversal lookup table from EvalData"]
    Eval --> Funcs["bind function table from funcs/"]
    Eval --> HCL[hcl.Eval]
    HCL --> Result[cty.Value]
    Eval --> Marks[propagate marks through Result]

The engine's BuiltinEvalContext (internal/terraform/eval_context_builtin.go) implements lang.EvalData, so calls into lang automatically see the in-progress state, plan, variables, locals, outputs, and refs to other modules.

References

Anything that depends on another object goes through lang.References or its wrappers. The graph builders use this to add edges:

refs, _ := lang.ReferencesInExpr(addrs.ParseRef, expr)
for _, ref := range refs {
    g.Connect(dag.BasicEdge(currentVertex, vertexFor(ref.Subject)))
}

addrs.ParseRef produces a typed addrs.Referenceable from a traversal — this is how a string like aws_instance.example[count.index].id becomes a Go value.

Functions

internal/lang/funcs/ has one file per built-in function (or per related group). The master functions.go assembles them into a single map and adds Terraform-only specials like path.module, path.root, and the terraform.workspace reference.

functions.MakeFunctionsTable accepts a base directory because some functions (templatefile, file, fileexists) need to resolve paths relative to the calling module.

Function results can be cached via function_results.go to avoid recomputing pure functions multiple times during the same evaluation pass.

Marks

Marks are cty's mechanism for tagging values without changing their type. Terraform uses them for:

• marks.Sensitive — values from a sensitive variable or sensitive output, or from a provider attribute marked sensitive in the schema. Sensitivity propagates through every operation; a function that touches a sensitive value produces a sensitive result.
• marks.Ephemeral — values from ephemeral resources or ephemeral variables. Ephemeral values are never persisted and cannot end up in state, plans, or non-ephemeral outputs.
• marks.Deprecated — values from sources marked deprecated; emits warnings.
• A small handful of internal marks for tracking unknown causes.

marks exposes helpers like marks.Has, marks.Contains, and cty.UnmarkDeep. Engine code is liberal with cty.UnmarkDeep when it needs to feed values into APIs that don't preserve marks, but is responsible for re-marking the result.

Static and dynamic checks

evaluate_valid.go (in internal/terraform) uses lang to verify that all references in a configuration would be valid before any evaluation happens. This catches typos in resource names, refs to non-existent variables, and so on without needing to actually plan.

The function lang.Scope.EvalCheckBlock is used for precondition and postcondition blocks (internal/terraform/eval_conditions.go), with separate result rendering when an assertion fails.

globalref/

globalref/ is a small analyzer that builds a cross-module reference graph from a *configs.Config. It's used by the terraform graph command and by some of the refactoring helpers (internal/refactoring/) to understand reach.

blocktoattr/

blocktoattr/ is a compatibility shim. Some older provider schemas declared a NestingSingle block where a newer one would declare an attribute of object type. The shim lets users write either form. It runs at the boundary between the configuration and the schema validator.

Integration points

• Configuration: consumes *configs.* types — module variables, locals, resource attributes — but does not depend on internal/configs for parsing.
• Engine: internal/terraform.BuiltinEvalContext implements lang.EvalData. The engine constructs a fresh lang.Scope per evaluation context and passes the scope into vertex Execute methods.
• Validators: internal/terraform/evaluate_valid.go and internal/configs/provider_validation.go use lang.References to verify reach without running the engine.
• Stacks: internal/stacks/stackruntime/ reuses parts of lang (notably the function table) but layers its own evaluation on top.

Performance notes

Expression evaluation is on the hot path of any plan or apply. The codebase keeps it cheap by:

• Caching function results in function_results.go.
• Avoiding re-parsing of references — once lang.References has been called for a given expression, the result is reused.
• Using cty's structural sharing — marking and unmarking are O(structure) but copying is O(1) for unchanged values.

Entry points for modification

• Adding a new built-in function: create a new file under internal/lang/funcs/ exposing a function.Function, and register it in functions.go's MakeFunctionsTable. Add tests next to the new file.
• Adding a new mark: extend internal/lang/marks/marks.go with a new value, and propagate it through any code that does cty.UnmarkDeep and re-marks. Audit the points where marks are checked against the persistent boundary (state, plan, output).
• Changing reference parsing: edit internal/lang/langrefs/. Be aware that this is the only entry point used by the engine; behavior changes ripple to graph builders.
• Adding a new evaluation entry point: extend lang.Scope with a method, and update BuiltinEvalContext if the engine needs to expose new data to it.

For where evaluation is invoked, see terraform-core. For the typed values that flow in and out, see the cty documentation.