Modes and the state machine

Purpose

Neovim is a modal editor. At any point in time it is in one mode (Normal, Insert, Visual, Command-line, Operator-pending, Terminal, ...) and a key press is interpreted in the context of that mode. The state machine that drives this is one of the few subsystems that has structurally survived from the Vim import: each mode is a VimState struct with enter, check, and execute callbacks; state_enter() is the generic loop.

Directory layout

The state-machine code is spread across the C tree because each mode owns its own file:

src/nvim/
├── state.c, state.h, state_defs.h    The generic state-enter loop
├── normal.c, normal.h, normal_defs.h Normal mode (~6,685 lines)
├── edit.c                            Insert mode (~130k bytes)
├── ex_getln.c                        Command-line mode (~150k bytes)
├── ex_docmd.c                        Ex command dispatch (~253k bytes)
├── terminal.c                        Terminal mode (job-attached buffers)
├── ops.c                             Operator-pending dispatch (~128k bytes)
├── getchar.c                         vgetc/safe_vgetc (~103k bytes)
└── globals.h                         The global State variable

Key abstractions

Type / function	File	Description
VimState	state_defs.h	{ check_callback, execute_callback } plus mode-specific data.
state_enter(VimState *)	state.c	The generic loop. Calls check, reads a key, calls execute.
NormalState, InsertState, CommandLineState, TerminalState	per-mode file	Concrete state structs.
vgetc() / safe_vgetc()	getchar.c	Read the next user key. Pumps the event loop while waiting.
Global State	globals.h	Bitmask: MODE_NORMAL, MODE_INSERT, MODE_CMDLINE, ... Read by :help mode().
curwin, curbuf	globals.h	Pointers to the current window and buffer. Mutated on every mode transition that changes focus.

How it works

stateDiagram-v2
    [*] --> Normal: main()
    Normal --> Insert: i, a, o, ...
    Normal --> Visual: v, V, <C-v>
    Normal --> Command: :
    Normal --> OpPending: d, y, c, ...
    Normal --> Terminal: :terminal + i
    Insert --> Normal: <Esc>
    Visual --> Normal: <Esc>
    Command --> Normal: <CR> / <Esc>
    OpPending --> Normal: motion (after operator)
    Terminal --> Normal: <C-\\><C-n>
    Visual --> OpPending: d / y / c

The actual implementation is a pushdown automaton, not a flat state machine. When you press : from Normal, the Normal mode's execute callback calls getcmdline(), which calls state_enter() again with a CommandLineState. The Normal state is suspended below it on the C call stack. When the command line returns (via <CR> or <Esc>), control unwinds back to the Normal mode's execute.

This is the only reason Vim's nested constructs (: from Visual, q: to edit a command, :! to run a shell, etc.) work cleanly: each is a recursive call into state_enter() with a different state.

state_enter itself

void state_enter(VimState *s) {
    for (;;) {
        ...
        int key = safe_vgetc();
        int execute_ret = s->execute(s, key);
        if (execute_ret == 0) break;       // explicit exit
        if (got_int || s->check(s) == 0) break;
    }
}

check is called before every vgetc() and is where each mode does its "before-keystroke" work: redraw, fire CursorHold, advance pending state. execute is called with the key and decides whether the loop should continue.

Per-mode entry points

Mode	Enter	Check	Execute
Normal	normal_enter (normal.c)	normal_check	normal_execute
Insert	insert_enter (edit.c)	insert_check	insert_execute
Command-line	command_line_enter (ex_getln.c)	command_line_check	command_line_execute
Terminal	terminal_enter (terminal.c)	(inline)	terminal_execute

Normal mode

src/nvim/normal.c is the largest of the four (6,685 lines). Most of it is the nv_cmds table — a switch-table-as-array of (key, function, flags) for every Normal-mode command. normal_execute() reads the next key, looks it up in nv_cmds, and dispatches to a nv_* function (e.g., nv_h, nv_dollar, nv_operator).

Operators (d, y, c, g~, ...) push the state into operator-pending without leaving Normal: they set oap (oparg_T *) and the next key is interpreted as a motion. The actual operation runs after the motion via do_pending_operator() (src/nvim/ops.c).

Insert mode

src/nvim/edit.c (~4,000 lines after counting). Insert mode owns autocomplete (src/nvim/insexpand.c, ~211k bytes — the largest single file in the api/edit area), abbreviation expansion, autoindent, and the digraph machinery.

Both i_* (insert-mode keys) and the various completion engines live here. <C-x><C-o> (omnicomplete), <C-x><C-n> (keyword), <C-x><C-f> (filename), and the Lua-backed completion path all dispatch out of ins_complete() in insexpand.c.

Command-line mode

src/nvim/ex_getln.c is the inverse of Insert mode: it builds a command line one character at a time, with its own completion (wildmenu, history, mappings) before handing the whole string off to do_cmdline().

do_cmdline() lives in src/nvim/ex_docmd.c (~253k bytes — the second-largest C file in the tree). It parses ranges, splits at |, expands <cword> style placeholders, and calls do_one_cmd() per command. do_one_cmd() then looks the command up in the table generated from src/nvim/ex_cmds.lua and invokes the ex_* handler.

Terminal mode

src/nvim/terminal.c is the buffer type that hosts a child process. When you :terminal, the editor opens a new buffer of type terminal, spawns a child via os/pty_proc_*.c, and feeds the child's output through libvterm to populate the buffer. Pressing i from Normal in a terminal buffer enters Terminal mode, where keystrokes are sent to the child instead of being interpreted as editor commands.

The two-way feed lives in terminal.c plus src/nvim/vterm/. See Embedded terminal.

Mode flags

State is a bitmask. Common values:

#define MODE_NORMAL      0x01
#define MODE_VISUAL      0x02
#define MODE_OP_PENDING  0x04
#define MODE_CMDLINE     0x08
#define MODE_INSERT      0x10
#define MODE_LANGMAP     0x20
#define MODE_REPLACE     0x40
#define MODE_VREPLACE    0x80
#define MODE_TERMINAL    0x100

The combinations matter: INSERT|REPLACE is replace-insert mode, OP_PENDING|VISUAL is operator-after-visual, etc. mode() (the user-facing function) returns a string derived from this bitmask.

Integration points

• Input — getchar.c#vgetc is what every mode calls. It applies mappings and pumps the event loop.
• Autocmds — mode transitions fire ModeChanged, InsertEnter, InsertLeave, etc., via src/nvim/autocmd.c.
• Redraw — every mode's check callback can request a redraw; the actual drawing happens in update_screen() in src/nvim/drawscreen.c.
• Lua callbacks — vim.on_key installs a callback that fires before any key is dispatched, regardless of mode.
• Command-line completion — src/nvim/cmdexpand.c (~128k bytes) handles <Tab> completion in Command-line mode.

Entry points for modification

• Add a Normal-mode command. Add an entry to nv_cmds in normal.c and write the handler.
• Change Insert-mode behavior. Most of it routes through ins_* functions in edit.c. Completion-specific changes go in insexpand.c.
• Add a new mode. Define a VimState subclass with enter/check/execute, allocate a MODE_* bit in state_defs.h, and provide a way to enter (Ex command, mapping, etc.).
• Tweak the input loop. getchar.c is fragile; touch it with care. The functions vgetc, safe_vgetc, vpeekc, and inchar are the high-traffic ones.

Key source files

File	Purpose
src/nvim/state.c, state.h	Generic state_enter loop
src/nvim/normal.c	Normal mode + operator dispatch
src/nvim/edit.c	Insert mode
src/nvim/insexpand.c	Insert-mode completion
src/nvim/ex_getln.c	Command-line mode
src/nvim/ex_docmd.c	Ex command parser/dispatcher
src/nvim/ops.c	Operators (d, y, c, ...)
src/nvim/getchar.c	The input loop and mapping
src/nvim/terminal.c	Terminal mode + libvterm integration
src/nvim/cmdexpand.c	Command-line completion
src/nvim/cmdhist.c	Command-line history
src/nvim/globals.h	The global State variable, curwin, curbuf