io_uring

Purpose

io_uring/ implements the modern asynchronous I/O interface that replaced aio(7) for almost every new use case. A program submits operations via a shared submission queue (SQ) and reaps results from a completion queue (CQ); the kernel processes them without per-op syscalls. io_uring covers nearly the entire syscall surface — file I/O, network I/O, polling, timers, futexes, splice, message rings, network zero-copy receive — through a uniform op-code mechanism.

The subsystem was originally part of fs/io_uring.c and was promoted to a top-level directory.

Directory layout

io_uring/
├── io_uring.c, io_uring.h              # Core: ring setup, submission, completion, retries
├── opdef.c, opdef.h                    # Op-code dispatch table
├── register.c, register.h              # io_uring_register: feature toggles, fixed buffers, files
├── sqpoll.c, sqpoll.h                  # SQPOLL kernel thread
├── tctx.c, tctx.h                      # Per-task io_uring context
├── refs.h, slist.h, wait.{c,h}, tw.{c,h}  # Refcount, list, waitqueue, task-work helpers
├── fdinfo.c, query.c, register.c       # /proc/<pid>/fdinfo, query API
├── filetable.c, filetable.h            # Fixed-file table
├── kbuf.c, kbuf.h                      # Provided/registered buffers
├── notif.c, notif.h, zcrx.c, zcrx.h    # Zero-copy receive notifications
├── alloc_cache.{c,h}                   # Per-ring small-object cache
├── eventfd.{c,h}, epoll.{c,h}          # Bridge to other event mechanisms
├── futex.{c,h}                         # IORING_OP_FUTEX_*
├── poll.{c,h}, timeout.{c,h}, sync.{c,h}, splice.{c,h}, statx.{c,h}, xattr.{c,h}, openclose.{c,h}, fs.{c,h}, advise.{c,h}, truncate.{c,h}
├── rw.{c,h}, net.{c,h}, msg_ring.{c,h}, napi.{c,h}, uring_cmd.{c,h}, waitid.{c,h}
├── nop.{c,h}                           # No-op (for benchmarking the ring itself)
├── cancel.{c,h}                        # IORING_OP_ASYNC_CANCEL
├── io-wq.c, io-wq.h                    # Worker-queue offload
├── bpf-ops.{c,h}, bpf_filter.{c,h}, cmd_net.c, mock_file.c, loop.{c,h}
└── memmap.{c,h}, rsrc.{c,h}            # Memory mapping, registered resources

Key abstractions

Symbol	File	Purpose
struct io_ring_ctx	io_uring/io_uring.h	The kernel side of a ring.
struct io_kiocb (request)	io_uring/io_uring.h	One submission.
struct io_uring_sqe	include/uapi/linux/io_uring.h	The user-space SQE format.
struct io_uring_cqe	include/uapi/linux/io_uring.h	The user-space CQE format.
Op handlers: io_read, io_write, io_send, io_recv, …	per-op file (rw.c, net.c, …)	Implement each op.
io_op_defs[]	io_uring/opdef.c	Dispatch table.
io-wq	io_uring/io-wq.c	The bounded worker pool used when an op blocks.

How it works

graph LR
    APP[User-space app] -->|"writes SQE"| SQ["SQ ring (mmap'd)"]
    APP -->|io_uring_enter syscall<br/>or SQPOLL| KERN[Kernel]
    KERN -->|fast path| INLINE[Try inline submit]
    KERN -->|blocking| WQ[io-wq worker thread]
    INLINE --> CQ
    WQ --> CQ
    CQ["CQ ring (mmap'd)"] --> APP
    KERN -->|task_work on return| TW[task work queue]

A typical submit/complete:

1. App writes an SQE describing the op (opcode, fd, addr, len, user_data, …) into the SQ ring.
2. App calls io_uring_enter() (or, with SQPOLL, the kernel thread polls on its own).
3. The kernel fetches the SQE, looks up the op handler in io_op_defs, and tries it inline (non-blocking).
4. If the op blocks, it is offloaded to an io-wq worker thread.
5. When complete, the result is written as a CQE; eventfd-backed wakeups can be configured.

SQPOLL

Setting IORING_SETUP_SQPOLL spawns a kernel thread that polls the SQ ring. The app no longer needs to call io_uring_enter() for submission. Useful for absolute-lowest-latency submission paths.

Registered resources

io_uring_register() (io_uring/register.c) lets the app pre-register buffers, files, eventfds, ring fds, and personality credentials, eliminating per-op overhead.

Buffer management

io_uring/kbuf.c implements provided buffers: the app posts a pool of buffers, the kernel picks one when a recv arrives. Reduces the need for the app to choose where to put each completion.

Zero-copy

• Send: IORING_OP_SEND_ZC / IORING_OP_SENDMSG_ZC use page-pinning and skb fragments to avoid copy.
• Receive: zero-copy receive (zcrx.c) is the newer counterpart, integrating with the network page-pool.

Op-code surface

The dispatch table in io_uring/opdef.c lists every op. Categories:

• File I/O — READ, WRITE, READV, WRITEV, READ_FIXED, WRITE_FIXED, FSYNC, SYNC_FILE_RANGE, FALLOCATE, OPENAT, OPENAT2, CLOSE, STATX, LINKAT, UNLINKAT, RENAMEAT, MKDIRAT, SYMLINKAT.
• Networking — SEND, RECV, SENDMSG, RECVMSG, ACCEPT, CONNECT, SHUTDOWN, SOCKET, SEND_ZC.
• Polling/timers — POLL_ADD, POLL_REMOVE, TIMEOUT, LINK_TIMEOUT, TIMEOUT_REMOVE.
• Synchronization — FUTEX_WAIT, FUTEX_WAKE, FUTEX_WAITV, WAITID, EVENTFD.
• Other — NOP, ASYNC_CANCEL, SPLICE, TEE, MSG_RING, URING_CMD, XATTR_SET/GET, EPOLL_CTL, BIND, LISTEN.

Multi-shot

POLL_ADD, RECV, RECVMSG, ACCEPT, and a few others support multi-shot: post once, get many CQEs until cancelled or until a single failure.

URING_CMD (raw command interface)

IORING_OP_URING_CMD lets a driver expose op-code semantics to user space — used by NVMe pass-through, ublk, and block-device passthrough. The driver implements file_operations->uring_cmd. Source in io_uring/uring_cmd.c.

Integration points

• fs/: read/write paths come back through vfs_iter_* and kiocb.
• net/: sockets call io_uring's helpers via proto_ops. Specific net helpers in cmd_net.c.
• mm/: registered buffers pin user memory; page pool integration for zero-copy.
• drivers/nvme/, drivers/block/ublk_drv.c: implement uring_cmd.
• kernel/sched/: io-wq worker threads are kernel threads.
• kernel/futex/: integrated into IORING_OP_FUTEX_*.

Key source files

File	Purpose
io_uring/io_uring.c	Ring setup, submit, complete, retries.
io_uring/opdef.c	Op-code → handler dispatch.
io_uring/io-wq.c	Worker pool.
io_uring/sqpoll.c	SQPOLL thread.
io_uring/register.c	io_uring_register features.
io_uring/rw.c	Read/write ops.
io_uring/net.c	Networking ops.
io_uring/poll.c	Poll & multi-shot poll.
io_uring/uring_cmd.c	Raw op interface for drivers.
io_uring/kbuf.c	Provided/registered buffers.
io_uring/zcrx.c	Zero-copy receive.
include/uapi/linux/io_uring.h	The user-space ABI.

Entry points for modification

• Add a new op: extend enum io_uring_op in the UAPI header, add a per-op file, register in opdef.c. Read recent op additions in git log for examples.
• Add a per-driver pass-through command: implement file_operations->uring_cmd. See NVMe (drivers/nvme/host/ioctl.c) and ublk (drivers/block/ublk_drv.c).
• Tune scheduling on the worker pool: io_uring/io-wq.c.
• Add a register opcode (e.g. for a new resource type): register.c and rsrc.c.

Related pages

• Filesystems — VFS path under the read/write ops.
• Networking — sockets, send/recv ops, zero-copy.
• Block layer — bios spawned by READ_FIXED/WRITE_FIXED.
• Drivers — NVMe pass-through, ublk.