Flight

Active contributors: David Li, Antoine Pitrou, Sutou Kouhei

Arrow Flight is a high-throughput RPC framework for moving Arrow record batches between processes. It runs on top of gRPC and uses Arrow IPC as its payload format. The protocol is defined in format/Flight.proto and format/FlightSql.proto. The C++ implementation lives in cpp/src/arrow/flight/.

Why Flight

A naive implementation of "transfer Arrow data over RPC" would serialize each record batch as a protobuf bytes field. That works, but it incurs:

• Extra memory copy from Arrow buffers into protobuf bytes.
• Length-prefixed framing inside an already-framed protobuf.
• Awkward parallelism: one logical query produces one stream, but a query engine often wants to fan out to many endpoints.

Flight solves these by:

• Keeping Arrow IPC's encapsulated message framing and putting it directly on gRPC's HTTP/2 byte stream — Arrow buffers are never copied into a protobuf message.
• Exposing endpoints (a list of (ticket, locations) pairs) so a query can be served from multiple nodes simultaneously and read in parallel.
• Defining a small set of well-known verbs (GetFlightInfo, DoGet, DoPut, DoExchange, DoAction, ListFlights, ListActions).

The verbs

Verb	Purpose
Handshake	Optional initial authentication exchange.
ListFlights	Lists available datasets matching a criteria filter.
GetFlightInfo	Returns a FlightInfo for a FlightDescriptor: schema, total records, and one or more endpoints.
GetSchema	Returns just the schema.
DoGet	Streams record batches for a Ticket.
DoPut	Streams record batches from client to server.
DoExchange	Bidirectional streaming.
DoAction	Server-defined actions (custom RPCs).
ListActions	Enumerates available actions.

The proto definitions live in format/Flight.proto. The C++ types in cpp/src/arrow/flight/types.h mirror the proto: FlightDescriptor, FlightInfo, FlightEndpoint, Ticket, Location, FlightPayload.

Server and client

Class	File	Purpose
FlightServerBase	cpp/src/arrow/flight/server.h	The server skeleton. Subclass it and override ListFlights, GetFlightInfo, DoGet, etc.
FlightClient	cpp/src/arrow/flight/client.h	The client. Returns FlightStreamReader / FlightStreamWriter for streaming verbs.
FlightCallOptions	cpp/src/arrow/flight/types.h	Per-call options (timeout, headers).
FlightServerOptions	cpp/src/arrow/flight/server.h	Server-wide options (TLS, auth handlers, middleware).

A typical server overrides DoGet to look up a record batch reader by ticket and return it; the framework handles serialization. cpp/src/arrow/flight/test_flight_server.cc is a worked example.

Middleware

Flight has both client and server middleware. Examples shipped in-tree:

• cpp/src/arrow/flight/server_tracing_middleware.cc and client_tracing_middleware.cc — OpenTelemetry trace propagation.
• cpp/src/arrow/flight/client_cookie_middleware.cc — propagates HTTP cookies across calls.
• cpp/src/arrow/flight/server_auth.h and client_auth.h — authentication callbacks.

Custom middleware is a subclass of ServerMiddleware / ClientMiddleware registered via ServerMiddlewareFactory / ClientMiddlewareFactory. They get callbacks for every call's start, headers, and end.

Transport

Flight has a pluggable transport layer (cpp/src/arrow/flight/transport.h). The default is gRPC, implemented in cpp/src/arrow/flight/transport/grpc/. There is also a UCX-based transport in cpp/src/arrow/flight/transport/ucx/ for HPC environments where gRPC's TCP-based transport is a bottleneck.

The serialization_internal.cc file (29 KB) is the gnarly heart of the implementation — it converts between Arrow IPC byte streams and gRPC's framed message stream without copying buffer data.

Flight SQL

Flight SQL is a higher-level layer on top of Flight that adds a SQL semantic. The proto is format/FlightSql.proto; the C++ implementation is in cpp/src/arrow/flight/sql/.

Adds verbs for:

• Statement execution (CommandStatementQuery, CommandStatementUpdate).
• Prepared statements (ActionCreatePreparedStatement, CommandPreparedStatementQuery).
• Catalog discovery (CommandGetCatalogs, CommandGetSchemas, CommandGetTables, CommandGetTableTypes).
• SQL info introspection (CommandGetSqlInfo).
• Type info (CommandGetXdbcTypeInfo).

There is a Flight SQL JDBC driver and ODBC driver implemented in separate Arrow repos that consume any Flight SQL server.

Performance benchmarks

cpp/src/arrow/flight/flight_benchmark.cc ships a benchmark harness that fans out a configurable workload across many threads/clients. Result throughput is typically tens of GB/s on localhost between two Arrow processes — comparable to a memcpy.

Testing

flight_internals_test.cc (35 KB) and flight_test.cc (66 KB) cover client/server interactions, middleware, auth, error mapping, and TLS. Cross-language Flight integration tests live in cpp/src/arrow/flight/integration_tests/ and are run by the integration matrix in .github/workflows/integration.yml.

Language wrappers

• PyArrow: python/pyarrow/_flight.pyx (~115 KB — among the largest Cython files). Public API in pyarrow.flight.
• R: r/R/flight.R exposes a thin wrapper.
• C-GLib + Ruby: c_glib/arrow-flight-glib/ and ruby/red-arrow-flight/.
• Flight SQL: python/pyarrow.flight.FlightSqlClient (Python), c_glib/arrow-flight-sql-glib/, ruby/red-arrow-flight-sql/.