curl/curl

Transfer engine

Active contributors: Daniel Stenberg, Stefan Eissing, Viktor Szakats

Purpose

The transfer engine is the state machine that walks every libcurl transfer from creation to completion. It is implemented almost entirely in lib/multi.c (~128 KB, 4125 lines) with helpers in lib/transfer.c, lib/easy.c, and lib/multi_ev.c. Both curl_easy_perform() and the multi/socket APIs feed into the same engine.

Key abstractions

Symbol	File	Description
`struct Curl_multi`	`lib/multihandle.h`	The container: hashes for sockets, connections, and DNS; pending queues; event admin handle
`struct Curl_easy`	`lib/urldata.h`	The per-transfer state — the public `CURL *` is a pointer to one
`enum Curl_mstate` (`MSTATE_*`)	`lib/multihandle.h`	The 14 states a transfer walks through
`Curl_multi_runsingle()`	`lib/multi.c`	The state-step function. One call advances one easy handle as far as it can go without blocking.
`Curl_multi_pollset()`	`lib/multi.c`	Reports the FDs the current state wants polled
`Curl_multi_socket_action()`	`lib/multi.c`	The application callback used by event-driven applications
`multi_getresult()`	`lib/multi.c`	Drives `CURLMSG_DONE` messages out to the application
`Curl_multi_perform()`	`lib/multi.c`	The simpler poll-based loop (`curl_multi_perform`)
`Curl_xfer_setup()`	`lib/transfer.c`	Hooks send/recv buffers to the connection filter chain

Transfer states

MSTATE_INIT
  └─► MSTATE_PENDING        ◄── waiting for capacity (max-host / max-total connections)
       │
       ▼
  MSTATE_SETUP              ◄── parse URL, find or create connection
       │
       ▼
  MSTATE_CONNECT            ◄── kick off DNS, may fan out to happy-eyeballs
       │
       ▼
  MSTATE_RESOLVING          ◄── waiting on hostip/asyn-/doh
       │
       ▼
  MSTATE_CONNECTING         ◄── socket connect, TLS handshake, HTTP/2 SETTINGS
       │
       ▼
  MSTATE_PROTOCONNECT       ◄── post-connect protocol setup (SMTP EHLO, FTP banner, ...)
  MSTATE_PROTOCONNECTING        (only used by protocols that take additional async steps)
       │
       ▼
  MSTATE_DO                 ◄── send the request (HTTP request line/headers, FTP RETR, etc.)
  MSTATE_DOING              ◄── continued where DO did not finish in one step
  MSTATE_DOING_MORE         ◄── second-channel work (FTP data connection, MQTT subscribe, …)
  MSTATE_DID                ◄── DO finished, ready for body
       │
       ▼
  MSTATE_PERFORMING         ◄── exchanging body bytes
  MSTATE_RATELIMITING       ◄── --limit-rate sleep
       │
       ▼
  MSTATE_DONE               ◄── teardown and decide whether to retry
       │
       ▼
  MSTATE_COMPLETED          ◄── result available to application
       │
       ▼
  MSTATE_MSGSENT            ◄── CURLMSG_DONE delivered, awaits remove_handle

These names appear directly in lib/multi.c and as mstate in struct Curl_easy. Each has a state_* handler (state_setup, state_connecting, state_doing, ...) that decides what to do this tick and what state to jump to next.

How it works

stateDiagram-v2
    [*] --> INIT
    INIT --> PENDING: capacity full
    INIT --> SETUP: ok
    PENDING --> SETUP: slot freed
    SETUP --> CONNECT
    CONNECT --> RESOLVING: DNS needed
    CONNECT --> CONNECTING: addresses cached
    RESOLVING --> CONNECTING
    CONNECTING --> PROTOCONNECT: TLS+ALPN done
    PROTOCONNECT --> PROTOCONNECTING
    PROTOCONNECT --> DO
    PROTOCONNECTING --> DO
    DO --> DOING
    DOING --> DOING_MORE
    DOING --> DID
    DOING_MORE --> DID
    DID --> PERFORMING
    PERFORMING --> RATELIMITING: --limit-rate
    RATELIMITING --> PERFORMING
    PERFORMING --> DONE
    DONE --> COMPLETED
    COMPLETED --> MSGSENT
    MSGSENT --> [*]

For each state, Curl_multi_runsingle():

Calls the state_* handler.
Asks the connection filter chain whether the underlying transport is ready (via Curl_conn_ev_data_pause, Curl_conn_data_pending, etc.).
Updates the timeout list and the socket pollset.
Records CURLMSG_DONE if the transfer reached MSTATE_COMPLETED.

There is no preemption. The engine relies on protocol handlers and connection filters to either return CURLE_AGAIN (yield) or make progress. The pollset returned by Curl_multi_pollset tells the application which sockets to wait on.

Two API entry points, one engine

graph LR
    A[curl_easy_perform] --> B[curl_multi_init - private]
    B --> C[curl_multi_add_handle]
    C --> D[curl_multi_perform loop]
    D --> E[Curl_multi_runsingle]
    F[curl_multi_perform - app] --> E
    G[curl_multi_socket_action - app] --> H[event-driven dispatch]
    H --> E

curl_easy_perform (in lib/easy.c) wraps the multi interface in a blocking loop. When the application uses curl_multi_socket_action the application drives event readiness, but every easy handle still walks the same state machine.

Timeout management

Timeouts are stored per-easy-handle in data->state.timeoutlist (lib/llist.c linked list) ordered by absolute timestamp. The earliest timeout across all handles bounds the next curl_multi_wait/curl_multi_poll call. add_next_timeout() and multi_timeout() in lib/multi.c manage them.

Pending queue

When a handle reaches MSTATE_SETUP but cannot allocate a connection (because CURLOPT_MAXCONNECTS, CURLMOPT_MAX_HOST_CONNECTIONS, or CURLMOPT_MAX_TOTAL_CONNECTIONS is hit), it goes to MSTATE_PENDING. As soon as another handle finishes and frees a slot, process_pending_handles() promotes one of the pending handles back to MSTATE_SETUP. Pending priority is FIFO.

Integration points

Connection cache (lib/conncache.c): the SETUP state asks the cache for an existing usable connection or creates a new one. DONE returns the connection to the cache (or marks it for closure). See Connection management.
Protocol handlers (Curl_protocol): every state delegates protocol-specific work to the do_it, done, connecting, doing, connect_it, disconnect, proto_getsock, and proto_pollset callbacks of the active handler.
Connection filters (lib/cfilters.c): the bytes flowing in and out of the engine pass through the cfilter chain attached to the connection. See Connection filters.
Resolver (lib/hostip.c, lib/asyn-*.c, lib/doh.c): RESOLVING polls Curl_resolv_pollset until addresses are ready.

Entry points for modification

A new state — practically never. The state graph has been stable for years.
A new state-handler behaviour for an existing state → look at the state_* static functions in lib/multi.c.
Tweaking pending/promotion order → process_pending_handles(), move_pending_to_connect() in lib/multi.c.
The wakeup mechanism for curl_multi_socket_action → lib/multi_ev.c and lib/multi_ntfy.c.

Built by Factory AutoWiki from public repository content. It is a generated preview for codebase exploration, not source-maintained documentation.