Key/value store

Consul's hierarchical key/value store is a small, strongly consistent, replicated database that operators use for application configuration, leader election, and dynamic feature flags. It's been in the codebase since 2014 and the wire format hasn't changed.

Pieces

Piece	Where
HTTP API	agent/kvs_endpoint.go
RPC endpoint	agent/consul/kvs_endpoint.go
State store	agent/consul/state/kvs.go, agent/consul/state/kvs_ce.go
Sessions (for locking)	agent/session_endpoint.go, agent/consul/session_endpoint.go, agent/consul/state/session.go
Tombstones (for delete blocking-queries)	agent/consul/state/graveyard.go, agent/consul/state/tombstone_gc.go
Transactional API	agent/txn_endpoint.go, agent/consul/txn_endpoint.go
CLI	command/kv/, command/lock/
Public Go client	api/kv.go, api/lock.go, api/semaphore.go

Data model

Keys are slash-delimited paths (e.g., myapp/prod/db/host). Values are arbitrary bytes (limited by kv_max_value_size in agent config; default 512 KB). Each key has:

• Value — bytes
• Flags — application-defined uint64
• CreateIndex, ModifyIndex, LockIndex — Raft indices
• Session — held session ID (when locked)

The semantics are last-write-wins with optional CAS (?cas=<index>).

API surface

Operation	HTTP	RPC
Get	GET /v1/kv/<key>	KVS.Get
Get with prefix	GET /v1/kv/<prefix>?recurse	KVS.List
Get keys only	GET /v1/kv/<prefix>?keys&separator=/	KVS.ListKeys
Put	PUT /v1/kv/<key>	KVS.Apply (KVSet)
Put with CAS	PUT /v1/kv/<key>?cas=<index>	KVS.Apply (KVCAS)
Delete	DELETE /v1/kv/<key>	KVS.Apply (KVDelete)
Delete prefix	DELETE /v1/kv/<prefix>?recurse	KVS.Apply (KVDeleteTree)
Acquire lock	PUT /v1/kv/<key>?acquire=<session>	KVS.Apply (KVLock)
Release lock	PUT /v1/kv/<key>?release=<session>	KVS.Apply (KVUnlock)

The HTTP layer accepts either single-key calls or the transactional /v1/txn endpoint that applies a list of operations atomically.

Locking via sessions

Sessions (agent/consul/state/session.go) are leases tied to either a node, a node + checks, or a TTL. To acquire a lock:

1. Create a session: PUT /v1/session/create.
2. Acquire the lock: PUT /v1/kv/<key>?acquire=<session-id> with a value containing the holder's identity.
3. Watchers see the LockIndex increment and can poll with blocking queries until released.
4. On node failure, the session invalidates and the lock auto-releases.

The standard leader-election pattern uses this primitive. The public Go client wraps it in api/lock.go (mutex) and api/semaphore.go (counted semaphore). The CLI exposes it as consul lock <prefix> <command>, which holds the lock while running an arbitrary command.

Transactions

POST /v1/txn supports atomic multi-key operations and mixed catalog + KV operations. The implementation is in agent/consul/txn_endpoint.go. Transactions support read+write composition: if the read fails (e.g., KVCheckIndex mismatch), the whole transaction aborts.

Watches and blocking queries

Every KV read accepts blocking-query parameters (?index= + ?wait=) and returns when the key changes. This is how dynamic configuration in client apps typically works:

curl "http://127.0.0.1:8500/v1/kv/myapp/config?index=42&wait=5m"

Tombstones (agent/consul/state/graveyard.go) preserve "this key was deleted at index N" so blocking queries that ask for "anything past index 41" can return a delete event rather than nothing. The tombstone_gc.go GC loop ages them out.

The streaming subscribe path (see Streaming) doesn't currently include KV; KV remains polling-based for now.

ACLs

KV is gated by the key_prefix and key rule families:

key_prefix "myapp/prod/" {
  policy = "write"
}
key "myapp/secret-key" {
  policy = "deny"
}

The agent's HTTP layer pulls the token, the server's KVS endpoint resolves it via acl/, and the state.Authorizer (agent/consul/state/kvs.go) filters out keys the caller can't read.

Operations

• consul kv put|get|delete|export|import — command/kv/.
• consul kv export dumps a tree as JSON; consul kv import restores it.
• Backups are normally taken with consul snapshot save, which captures KV alongside the rest of the FSM.
• The size of the KV tree is reported in consul.fsm.kvs metrics and is bounded by kv_max_value_size per value.

Entry points for modification

• Add a new KV-related RPC verb: extend agent/consul/kvs_endpoint.go and the FSM dispatch in agent/consul/fsm/commands_ce.go.
• Tune watch behavior: see agent/consul/state/kvs.go's use of memdb.WatchSet.
• Add HTTP behavior: agent/kvs_endpoint.go is small and easy to follow.