Embedded terminal

Purpose

:terminal opens a buffer that hosts a child process running on a pty. Output from the child is rendered into the buffer; keystrokes (in Terminal mode) are forwarded to the child. The implementation is split between an in-process terminal emulator (the vendored libvterm at src/nvim/vterm/), the editor-side glue (src/nvim/terminal.c), and the platform pty layer (src/nvim/os/pty_proc_*.c). It is what powers :terminal, vim.system({pty=true}), and the testing infrastructure that needs a real terminal grid.

Directory layout

src/nvim/
├── terminal.c (~77k bytes)        Buffer-as-terminal: spawn, feed, read
├── terminal.h
├── vterm/                          In-tree fork of libvterm (~16 source files)
└── os/
    ├── pty_proc_unix.c             Unix pty spawning
    ├── pty_proc_win.c              Windows ConPTY spawning
    └── pty_conpty_win.c            Windows-specific helpers

src/nvim/vterm/ is one of the rare third-party libraries Neovim has fully forked; upstream is no longer tracked.

Key abstractions

Type / function	File	Description
Terminal	terminal.c	The per-buffer terminal state. Owns the libvterm VTerm, the child process, the cell grid.
terminal_open(...)	terminal.c	Create a terminal buffer and spawn the child.
VTerm	vterm/vterm.h	The emulator's state machine. Receives bytes, emits screen-state changes.
pty_proc_T	os/pty_proc_*.h	Platform-specific pty-attached process.
terminal_send	terminal.c	Write bytes from the editor to the child.
terminal_receive	terminal.c	Bytes from the child arrived; feed libvterm.

How it works

sequenceDiagram
    participant U as User
    participant Buf as Terminal buffer
    participant Term as Terminal (terminal.c)
    participant VT as libvterm
    participant Pty as pty
    participant Child as Child process
    U->>Buf: :terminal bash
    Buf->>Term: terminal_open
    Term->>Pty: spawn(bash) on pty
    Pty->>Child: bash starts
    loop on bytes from child
        Child-->>Pty: stdout bytes
        Pty-->>Term: rstream callback
        Term->>VT: vterm_input_write(bytes)
        VT->>Term: damage callbacks (cells changed)
        Term->>Buf: ml_replace updated lines
    end
    U->>Buf: i, then types
    Buf->>Term: keystrokes (Terminal mode)
    Term->>Pty: pty_proc_send(bytes)
    Pty->>Child: stdin bytes

The Terminal struct owns:

• A libvterm VTerm instance — the in-process emulator.
• A cell grid in the buffer — populated as libvterm reports damage.
• A pty_proc_T — the child process's pty handle.
• A scrollback buffer — bytes that have scrolled off the top of the visible region but should still be in the buffer.

When bytes come in on the child's stdout, the editor pushes them through libvterm. libvterm interprets escape sequences and reports back via callbacks: "cells (0,0)-(80,1) changed", "cursor moved to (5,3)", "scrolled by 1 line", "title changed", "bell", and so on. The editor side translates these into buffer mutations: ml_replace for changed lines, extmarks for highlights, terminal-specific UI calls for cursor and scroll.

Spawning a child

terminal_open(...) creates the buffer, allocates a Terminal, and spawns through pty_proc_unix.c (Unix) or pty_conpty_win.c (Windows 10+). The child sees a real terminal: stdin/stdout/stderr are the pty, TERM is set to xterm-256color, the controlling tty is set so signals work.

Terminal mode

Terminal mode is the editor mode in which keystrokes are forwarded to the child instead of being interpreted as editor commands. It is entered by pressing i (or any insert-mode entry key) in a terminal buffer; left by <C-\><C-n>. The mode handler is terminal_execute in terminal.c — see Modes and the state machine.

Highlighting

The terminal carries SGR attributes (colors, bold, underline) per cell. The editor maps each unique attribute set to a per-buffer highlight group on the fly and stores the mapping in Terminal->highlight_attr_table. Extmarks paint the cells on render.

Resize

When the user resizes the editor window showing the terminal, the editor:

1. Calls vterm_set_size(vt, rows, cols) so libvterm reflows.
2. Calls pty_proc_resize(p, rows, cols) so the kernel sends SIGWINCH to the child.
3. Marks the buffer for full redraw.

libvterm internals

src/nvim/vterm/ is C and split into:

• vterm.c — the public state machine.
• parser.c — the escape-sequence parser (CSI, OSC, DCS, sixel-passthrough).
• screen.c — the cell grid with scrollback.
• state.c — DEC modes, charsets, cursor state.
• pen.c — SGR attribute tracking.
• unicode.c — wide-character handling.
• mouse.c — mouse-event encoding.
• keyboard.c — input encoding (translate VTerm keys to escape sequences for the child).
• encoding.c — UTF-8 decoding.

Most of this is upstream libvterm. The Neovim fork has accumulated kitty-keyboard-protocol support, sixel passthrough, and various bug fixes.

vim.system pty

vim.system({...}, opts) (in runtime/lua/vim/_core/system.lua) is the modern Lua surface for spawning a child. With pty = true, it goes through the same pty layer used by :terminal but the output is captured rather than rendered into a buffer. Used by tests and by plugins that need to drive child processes that require a real tty.

Integration points

• Terminal mode — src/nvim/state.c plus terminal_execute in terminal.c.
• Buffer infrastructure — terminal buffers are real buf_T instances with b_terminal non-NULL. Most editor commands work; some (:write) are filtered.
• jobstart — the older Vimscript surface. With pty = 1 it spawns through the same code as :terminal.
• TUI — passthrough sequences (sixel, kitty graphics) pass through libvterm and are forwarded to the outer terminal via tui.c.
• Autocmds — TermOpen, TermClose, TermEnter, TermLeave, TermRequest (Nvim-only) all fire in this path.

Entry points for modification

• A new escape sequence the child sends. Most belong in libvterm — touch vterm/parser.c and vterm/state.c accordingly. If it's a Neovim-specific extension (e.g., OSC 633 "VS Code shell integration"), parse it in terminal.c#parse_osc after libvterm has handed it off via the OSC callback.
• A new pty option. pty_proc_unix.c is the fork; add the option to pty_proc_T and propagate. Windows lives in pty_conpty_win.c.
• Terminal mode keybinding. :tnoremap plus terminal_execute is enough for almost anything.

Key source files

File	Purpose
src/nvim/terminal.c	Buffer-as-terminal glue, mode dispatch, libvterm callbacks
src/nvim/vterm/vterm.c	libvterm public surface
src/nvim/vterm/parser.c	Escape-sequence parsing
src/nvim/vterm/screen.c	Cell grid + scrollback
src/nvim/os/pty_proc_unix.c	Unix pty spawning
src/nvim/os/pty_conpty_win.c	Windows ConPTY
runtime/lua/vim/_core/system.lua	vim.system