CLI shell

Active contributors: Mytherin, Mark Raasveldt, Mark

Purpose

tools/shell/ is the duckdb command-line shell — the binary you get when you run make. It provides an interactive SQL prompt, dot-commands (.help, .tables, .read), output rendering modes (box, csv, markdown, ascii, json, html, …), syntax highlighting, line editing, autocomplete, and progress bars. It is descended from the SQLite shell but has been substantially expanded and rewritten to take advantage of DuckDB-native features.

Directory layout

tools/shell/
├── shell.cpp                          The shell main loop and dot-command dispatch (~94 KB)
├── shell_command_line_option.cpp      Parses command-line options
├── shell_extension.cpp                Hooks autocomplete / shell-only extension surface
├── shell_helpers.cpp                  Output formatting helpers
├── shell_highlight.cpp                Syntax highlighter (~42 KB)
├── shell_metadata_command.cpp         Implementation of .tables / .schema / .indexes etc.
├── shell_progress_bar.cpp             Animated progress bar
├── shell_prompt.cpp                   Multi-line prompt + history
├── shell_render_table_metadata.cpp    Table-info rendering (used by .schema)
├── shell_renderer.cpp                 Output renderers (box, csv, json, latex, ...)
├── shell_windows.cpp                  Windows-specific shims
├── linenoise/                         Vendored line-editing library
├── include/                           Public headers
├── rc/                                Windows resource (icon, version)
└── tests/                             Shell-only test fixtures

Key abstractions

Type	File	Role
ShellState	tools/shell/include/shell_state.hpp	Per-process state of the shell: open database, connection, output settings, history file.
MetaCommand	tools/shell/shell_metadata_command.cpp	Implementation of dot-commands.
ShellRenderer	tools/shell/shell_renderer.cpp	Output mode dispatcher (box, csv, json, markdown, latex, html, line, list, table, …).
ShellHighlight	tools/shell/shell_highlight.cpp	SQL syntax highlighter for the prompt.
Linenoise*	tools/shell/linenoise/	Line editing, history, autocomplete callbacks.
ShellProgressBar	tools/shell/shell_progress_bar.cpp	Progress bar driven by PRAGMA enable_progress_bar.

How it works

graph LR
    Argv[CLI args] -->|shell_command_line_option| Cfg[Initial config]
    Cfg -->|duckdb_open + duckdb_connect| Conn[Connection]
    Conn --> Loop[REPL loop]
    Loop -->|line edit| Linenoise[linenoise + autocomplete]
    Loop -->|dot-command| Meta[MetaCommand]
    Loop -->|SQL| Conn
    Conn -->|chunks| Render[ShellRenderer]
    Render -->|stdout / file| Output[Formatted output]

The main loop:

1. shell.cpp parses command-line flags via shell_command_line_option.cpp, opens a database, and creates a connection through the C API (duckdb_open / duckdb_connect from systems/main).
2. Reads a line via linenoise with autocomplete callbacks that call into the autocomplete extension.
3. If the line starts with ., dispatches to shell_metadata_command.cpp (analogous to SQLite's .tables, .read, .mode, .import, etc.).
4. Otherwise, sends the SQL through duckdb_query. Streams chunks via ShellRenderer in the active output mode.
5. On Ctrl-C, calls duckdb_interrupt which routes to ClientContext::Interrupt in the engine.

Output modes

The shell supports a wide set of .mode settings:

Mode	Description
box (default)	Pretty-printed table using the engine's BoxRenderer (src/common/box_renderer.cpp).
column, list, line	Less ornate text formats.
csv, tabs	Delimited output.
json, jsonlines	JSON output (full result or NDJSON).
markdown, latex, html	Documentation-friendly formats.
quote, insert	SQL-friendly: emit INSERT INTO ... statements.
trash	Discard output.
duckbox	The wider DuckDB-native pretty box mode (default for interactive).

Implementation is in shell_renderer.cpp, with the inner table renderer reused from the engine.

Dot-commands

Selected commands implemented in shell_metadata_command.cpp:

| Command | Effect | | -------------------- | ------------------------------- | ------------------------ | | .help | Print all commands. | | .open <file> | Open a different database. | | .tables [pat] | List tables matching pattern. | | .schema [pat] | Print CREATE statements. | | .indexes [pat] | List indexes. | | .read <file> | Run SQL from a file. | | .shell <cmd> | Run a shell command. | | .import file table | Import a CSV/Parquet/JSON file. | | .export <dir> | Export the database. | | .mode <fmt> | Switch output mode. | | .timer on | off | Print query timings. | | .dump [tab] | Dump SQL. | | .bail on | off | Stop on the first error. | | .echo on | off | Echo input. | | .headers on | off | Show column headers. |

Autocomplete

The shell links against the autocomplete extension and registers a callback with linenoise so that pressing Tab triggers completion. The completion engine consults the catalog (visible tables, columns, types) and the SQL keyword/function list. See extensions/autocomplete-and-others.

Syntax highlighting

shell_highlight.cpp is a hand-written SQL tokenizer plus a colorizer. It runs on each redrawn input line and feeds linenoise the colored buffer. Highlighting respects the connection's keyword set (so dialect changes from extensions can be picked up).

Progress bars

When PRAGMA enable_progress_bar = true, the engine emits progress events that shell_progress_bar.cpp consumes and renders. The bar is opt-in because some terminals do not handle ANSI rewrites well.

Integration points

• C API: Everything user-facing goes through src/main/capi/.
• Autocomplete: Tab completion uses the autocomplete extension.
• BoxRenderer: src/common/box_renderer.cpp provides the default rendering.
• Tests: tools/shell/tests/ for shell-specific behavior, plus the engine test suite for SQL semantics.

Entry points for modification

• Adding a dot-command: extend the dispatch table in shell_metadata_command.cpp.
• Adding an output mode: implement a renderer subclass in shell_renderer.cpp.
• Tuning highlighting: adjust the keyword tables in shell_highlight.cpp.
• Cross-platform fixes: see shell_windows.cpp for Windows-specific paths.

Key source files

File	Purpose
tools/shell/shell.cpp	Main loop + dot-command dispatch.
tools/shell/shell_renderer.cpp	Output modes.
tools/shell/shell_metadata_command.cpp	Dot-commands.
tools/shell/shell_highlight.cpp	Syntax highlighting.
tools/shell/shell_prompt.cpp	Prompt + history.
tools/shell/linenoise/	Vendored line editor.

See tools/language-bindings for the in-tree client packaging and systems/main for the C API the shell calls.