Prism

Prism is the modern, hand-written, portable Ruby parser that lives alongside parse.y. Originally announced as YARP (Yet Another Ruby Parser), it was renamed to Prism and merged into CRuby in 2023. As of Ruby 3.3 it is a peer of parse.y; users select between them with --parser=.

The same prism/ directory is also published as a standalone library (the prism default gem) and is consumed by Ruby LSP, RuboCop, Sorbet, and other tools that need to parse Ruby outside CRuby.

Purpose

• Provide a portable Ruby parser that doesn't require a Yacc-style generator.
• Be error-tolerant: produce useful AST + error nodes from incomplete or invalid source, suitable for IDEs.
• Provide a typed AST with a stable schema, so external tools don't need to follow every CRuby change.
• Be fast — comparable to or faster than parse.y for large files.

Files

The prism/ directory layout:

Path	Purpose
prism/config.yml	The single source of truth: declarative spec of every node, token, flag, and option. ~128 KB.
prism/templates/	ERB templates that consume config.yml to render C, Ruby, JavaScript, etc.
prism/templates/template.rb	The ERB driver. Run during make srcs.
prism/prism.c	The hand-written core parser, augmented with templated boilerplate. ~980 KB.
prism/prism.h	Public API header.
prism/parser.c, parser.h	Parser state struct, allocation, lifecycle.
prism/encoding.c	Encoding-aware character classification (mirrors parse.y's lexer logic).
prism/regexp.c	Mini-parser for the inside of regex literals.
prism/options.c, options.h	Parse-time options (frozen-string-literal, encoding, scope, etc.).
prism/static_literals.c	Literal interning and constant folding.
prism/source.c, source.h	Source buffer and line-offset table.
prism/diagnostic.h	Error and warning kinds.
prism/internal/	Private headers.
prism/compiler/	Helpers for compiling Prism's output into CRuby iseqs.
prism_compile.c	The bridge from Prism's AST to CRuby bytecode.
prism_init.c	Boots Prism inside the running interpreter.
lib/prism.rb, lib/prism/, ext/prism/	The Ruby gem facade.

Architecture

graph LR
    src[".rb source bytes"] -->|tokens| lex["Hand-written lexer\nprism.c"]
    lex -->|recursive descent| parse["Parser\nprism.c"]
    parse -->|allocates| nodes["pm_node_t tree\nfrom config.yml"]
    parse -->|errors| diags["pm_diagnostic_list_t"]
    nodes -->|via prism_compile.c| iseq["CRuby iseq"]
    nodes -->|via lib/prism.rb| ruby["Prism::*\nRuby AST classes"]

Prism is a recursive-descent parser, not LALR. Each language construct corresponds to a parse_<thing> function in prism.c. The lexer is also hand-written and is tightly coupled to the parser (Ruby's syntax is too context-sensitive for a clean separation).

The token kinds, AST node types, and parser flags all live in config.yml:

- name: ArrayNode
  fields:
    - name: elements
      type: node[]
    - name: opening_loc
      type: location?
    - name: closing_loc
      type: location?
  comment: |
    Represents an array literal: [1, 2, 3].

prism/templates/template.rb walks the YAML and renders boilerplate per language:

• C: node struct definitions, allocation helpers, visitor scaffolding.
• Ruby: Prism::ArrayNode classes with accessors and accept(visitor) methods.
• JavaScript, Java, etc.: bindings for non-Ruby consumers (in the standalone gem repo).

This means adding a new node is mostly a matter of:

1. Edit prism/config.yml to declare it.
2. Edit prism.c to actually parse it.
3. Run the templater.

prism_compile.c

The compiler bridge prism_compile.c walks a Prism AST and emits exactly the same iseqs that compile.c emits from a NODE tree. The two compilers share the same instruction set (insns.def) and the same VM, so once an iseq exists, the rest of the runtime can't tell which parser produced it.

prism_compile.c is large (~470 KB) because it has to mirror the surface area of compile.c. Each Prism node type has a pm_compile_<kind> function that emits the equivalent bytecode.

Selecting Prism at runtime

./ruby --parser=prism -e 'p Prism.parse("1+2").value'

Prism.parse always uses Prism regardless of the global --parser flag (it's an explicit API), so tools like Ruby LSP can run on a CRuby that's been booted with parse.y as default and still get Prism behaviour.

RUBYOPT=--parser=prism flips the default for require, load, and the main script.

Error tolerance

Prism produces a pm_node_t tree even for source that has errors:

result = Prism.parse("def foo\n  bar = ")
result.errors  # => [#<Prism::ParseError ...>, ...]
result.value   # => still a usable AST with a missing-rhs marker

This is what makes Prism IDE-friendly. parse.y typically aborts at the first error.

Test corpus

Prism's tests live under test/prism/ and exercise every node kind:

• test/prism/parse_test.rb — parses thousands of fixture snippets and snapshots the resulting AST.
• test/prism/fixtures/ — the snippets themselves (covering the entire Ruby language).
• test/prism/snapshots/ — expected Prism::ParseResult.dump outputs.

spec/ruby/language/prism_spec.rb and friends cross-check Prism semantics against the language spec.

Sync with the upstream gem

Prism is also maintained in the standalone ruby/prism repository. tool/sync_default_gems.rb prism pulls upstream changes into this tree. PRs that affect the parser are usually opened upstream and synced down.

Entry points for modification

• New language feature: declare the node in prism/config.yml, parse it in prism.c, compile it in prism_compile.c. Mirror the same change in parse.y if you want both parsers to support it.
• Better error: add a PM_ERR_* constant in prism/diagnostic.h and emit it from the relevant parse_* function.
• Performance: profiling output reveals hot paths in prism.c; many functions are designed to be inlined.

See parser.md for the long-running parse.y parser and compiler.md for what happens once an AST exists.