Parser (parse.y)

parse.y is the historical, default Ruby parser — a Bison/Lrama LALR(1) grammar that has been edited continuously since the 1990s. At ~520 KB and over 13,000 lines, it is the single most intricate file in the tree. The parser produces a tree of NODE structs (node.h, rubyparser.h) that the compiler later walks to emit bytecode.

Purpose

Parse a stream of bytes into a Ruby AST while:

1. Implementing every Ruby language feature, including the ones that need lookahead-heavy disambiguation (heredocs, ambiguous do blocks, regular-expression vs division).
2. Producing useful error messages with line/column information.
3. Coexisting with Prism (prism/), which targets the same language but in a separate parser. Either parser can be selected at runtime via --parser=.
4. Being reusable: the same grammar powers Ripper (ext/ripper/) and the embeddable "universal parser" (universal_parser.c).

Files

File	Purpose
parse.y	The Bison/Lrama grammar. Hand-edited.
parse.c	Generated from parse.y by Lrama at build time. Do not edit.
node.h, rubyparser.h	NODE definitions and the rb_parser_* API.
node.c, node_dump.c, parser_node.h	NODE allocation, debug dumping, helpers.
ruby_parser.c	Glue layer: invokes the parser and converts errors into Ruby exceptions.
parser_st.c, parser_st.h	A self-contained hash table the parser uses (independent of st.c).
parser_value.h, parser_bits.h	Parser-only value type and bit helpers.
universal_parser.c	A stripped-down embedding mode used by external tools that don't link the full VM.
tool/lrama/	The Ruby-implemented Bison-compatible parser generator.
tool/ripper/, ext/ripper/	The Ripper library, which exposes the grammar as a tokenizer/AST library.

Architecture

graph LR
    src[".rb source"] -->|raw bytes| lex["Lexer in parse.y\n(yylex)"]
    lex -->|tokens + lvals| bison["LALR table\nfrom Lrama"]
    bison -->|grammar actions| nodes[NODE tree]
    nodes -->|to compile.c| compile[Bytecode compiler]
    nodes -->|to ext/ripper| ripper[Ripper]
    nodes -->|to universal_parser.c| tools[External tools]

The lexer is handwritten inside parse.y (the function yylex near the top of the file's C section). It tracks a state machine of "what we expect next" — Ruby's syntax requires significant context to tokenize correctly, especially around //regex, do/block, and heredoc bodies.

The grammar rules are LALR(1) and use Bison-style %token, %type, and %left/%right/%nonassoc annotations. Many rules drop into C-action blocks that allocate NODEs.

NODE tree

NODE (rubyparser.h) is a tagged union:

struct RNode {
    rb_node_t flags;          /* NODE_TYPE in low bits */
    int u_id;                 /* unique id */
    /* + payload fields per node type */
};

There are ~150 node types — every Ruby construct has one or more. Examples:

NODE type	Purpose
NODE_LIT	literal (Integer, Float, Symbol)
NODE_STR	String literal
NODE_DSTR	Dynamic string (with interpolation)
NODE_ARRAY	Array literal
NODE_HASH	Hash literal
NODE_CALL	Method call (obj.m(a,b))
NODE_FCALL	Function-style call (m(a,b))
NODE_VCALL	Variable-or-call (m)
NODE_IF / NODE_UNLESS	Branch
NODE_DEFN / NODE_DEFS	Method definition (instance / singleton)
NODE_ITER	Block (with iterator)
NODE_BLOCK_PASS	&blk argument
NODE_RESCUE / NODE_ENSURE	Exception handling

node_dump.c pretty-prints a NODE tree for debugging (./ruby --dump=parsetree -e '1 + 2').

Lrama integration

Lrama is Ruby's home-grown replacement for GNU Bison. It lives at tool/lrama/ and is itself a parser generator:

• Reads parse.y's %-prefixed declarations and rules.
• Builds LALR(1) state tables.
• Emits parse.c (and a header).
• Supports Bison-compatible options used by Ruby (precedence, types, error recovery).

The build invokes Lrama via make srcs. Before Ruby 3.3, GNU Bison was required to build CRuby; Lrama now removes that dependency.

The bison.lock-style mechanism that keeps generated parse.c consistent with parse.y is just a Make rule: parse.c: parse.y plus a stamp file. CI verifies that the regenerated parse.c matches the committed one.

Ripper

ext/ripper/ reuses parse.y to expose tokens, parse events, and an alternative AST to Ruby code:

require 'ripper'
pp Ripper.lex("1 + 2")
# => [[[1,0], :on_int,  "1", BEG],
#     [[1,1], :on_sp,   " ",  ...],
#     ...]

Internally, ext/ripper/ builds a second parse.c-equivalent from the same parse.y, but with action blocks that emit Ripper events instead of NODEs. The build rules are in ext/ripper/depend.

Universal parser

universal_parser.c provides a stripped-down embedding API: no VM, no VALUEs, just C structs. Used by tools like Ruby LSP fallbacks and code-completion daemons that need to call the parser without bringing all of CRuby along. The interface is in rubyparser.h (the rb_parser_* family).

Selecting the parser

./ruby --parser=parse.y -e 'p RUBY_DESCRIPTION'   # default
./ruby --parser=prism   -e 'p RUBY_DESCRIPTION'
RUBYOPT=--parser=prism ./ruby some_script.rb

Both parsers are always built into the binary. The choice affects:

• AST shape (NODE tree vs Prism AST).
• Error message wording.
• Edge-case behaviour around the very newest syntax features (Prism is sometimes ahead, sometimes behind).

prism_compile.c is the bridge from Prism's AST to the same iseq output that compile.c produces from a NODE tree, so the runtime is identical regardless of parser.

Entry points for modification

• Grammar tweak (e.g., a new piece of syntax sugar): edit parse.y, run make srcs, run make btest. Add tests under test/ruby/test_syntax.rb and spec/ruby/language/.
• Lexer state bug: search for the offending token in parse.y's lexer functions; the state machine is dense, so reproduce in a small file under bootstraptest/test_syntax.rb first.
• NODE walking: node.c and compile.c walk the NODE tree. node_dump.c is the canonical printer.
• Ripper events: ext/ripper/eventids2.c is the table of token kinds; tool/generic_erb.rb regenerates parts of it.

See prism.md for the alternative parser and compiler.md for what happens to the AST next.