Parser

Active contributors: DougGregor, hamishknight, ahoppen

Purpose

The parser turns Swift source text into an abstract syntax tree. Two implementations coexist: the long-standing C++ recursive-descent parser in lib/Parse/, and a newer Swift-implemented parser (swift-syntax) bridged via lib/ASTGen/. The Swift-side parser is gradually taking over.

Directory layout

lib/Parse/                      # C++ parser
├── Lexer.cpp
├── Parser.cpp
├── ParseDecl.cpp                # 10,976 lines — declarations
├── ParseExpr.cpp
├── ParseStmt.cpp
├── ParsePattern.cpp
├── ParseType.cpp
├── ParseRegex.cpp               # regex literal parsing
└── ParseGeneric.cpp

lib/ASTGen/                     # Swift-side parser → C++ AST bridge
└── Sources/ASTGen/             # Swift code that walks swift-syntax trees
                                # and constructs C++ AST nodes through bridging

include/swift/Parse/            # parser headers
└── Parser.h                    # the parser's public API

Key abstractions

Type	File	Description
swift::Lexer	lib/Parse/Lexer.cpp	Token stream over a SourceFile.
swift::Token	include/swift/Parse/Token.h	A single token (tok::identifier, tok::kw_func, ...).
swift::Parser	lib/Parse/Parser.cpp	Top-level recursive-descent parser.
swift::SourceFile	include/swift/AST/SourceFile.h	One .swift file as known to the AST.
swift::ParseDeclOptions	include/swift/Parse/Parser.h	Bitfield steering decl parsing (top-level vs nested).
swift::SourceManager	include/swift/Basic/SourceManager.h	Owns source buffers and assigns SourceLocs.

How it works

graph LR
    SrcText[".swift text"] --> Lexer[Lexer]
    Lexer -->|tokens| ParserCpp[lib/Parse Parser]
    Lexer -->|tokens| ParserSwift[swift-syntax Parser]
    ParserSwift -->|syntax tree| ASTGen[lib/ASTGen]
    ASTGen -->|"BridgedDecl, BridgedExpr"| ASTNodes
    ParserCpp -->|"Decl, Expr, Stmt"| ASTNodes
    ASTNodes --> Sema[Sema]

The C++ parser produces AST nodes directly. The Swift-side path is:

1. swift-syntax (in a sibling repo) consumes the source and produces a typed Syntax tree.
2. lib/ASTGen/ walks the syntax tree and constructs BridgedDecl, BridgedExpr, etc., which the C++ side wraps as ordinary Decl * / Expr *.

The two parsers can be enabled independently. The pipeline switches based on -enable-experimental-feature ParserASTGen / equivalent flags. New language constructs (macros, regex literals, typed throws) typically prefer the Swift parser because expressing them in swift-syntax is easier than in the C++ recursive-descent parser.

Lexer specifics

• The lexer is hand-written (no lex/flex). Source: lib/Parse/Lexer.cpp.
• Trivia (whitespace, comments) is preserved alongside tokens for IDE clients.
• The lexer also handles raw string literals, multi-line strings, regex literals (lib/Parse/ParseRegex.cpp), and the #sourceLocation directive.

Macro expansion

When the parser encounters a freestanding macro (#myMacro(...)) or an attached macro attribute (@MyMacro), it records a placeholder in the AST. Sema later expands it via the Swift macro infrastructure (lib/Macros/), which sends the syntax tree to a separate swift-plugin-server process.

Integration points

• AST construction -- both parsers populate lib/AST/ types directly. AST creation runs on the parser's thread.
• Diagnostics -- the parser emits via ctx.Diags.diagnose(...), with diagnostics defined in include/swift/AST/DiagnosticsParse.def.
• Source ranges -- every parsed node records SourceLoc ranges referencing back into SourceManager. These survive through Sema and into IDE clients.
• SyntaxKind -- the swift-syntax library exports SyntaxKind enums; ASTGen pattern-matches against them to dispatch.
• Incremental parsing -- SourceKit (tools/SourceKit/) reuses Parser for completions, but with a customized callbacks factory that records ranges of interest.

Entry points for modification

• Adding a new keyword: define it in include/swift/AST/TokenKinds.def and recognize it in Lexer::lexImpl.
• Adding new declaration syntax: extend ParseDecl.cpp (or the swift-syntax grammar + ASTGen).
• Adding a parse-time diagnostic: edit include/swift/AST/DiagnosticsParse.def, emit at the call site.
• Investigating a parser-only crash: build with -Xfrontend -dump-parse to print the AST as parsed (no Sema).

Related pages

• AST -- the data structures the parser fills in.
• Sema -- the next stage.
• SwiftCompilerSources -- bridging mechanics shared with ASTGen.