Functions

Lexing

lex(src: str, file_name: str) → list[SpannedToken]

Separate a source file into syntactic tokens

Parameters:

• src (str) – The content of the source file to lex

• file_name (str) – The name of the source file, used to annotate Spans

Returns:

The separated semantic tokens

Return type:

list[SpannedToken]

Preprocessing

preprocess(src: str, file_name: str, include_paths: Sequence[str | PathLike[str]], defines: Sequence[Define]) → PreprocessorResult

Preprocess a token stream, elaborating compiler directives

Parameters:

• src (str) – The content of the source file to preprocess

• file_name (str) – The name of the source file, used to annotate Spans

• include_paths (Sequence[str | PathLike[str]]) – The include paths to search for include directives

• defines (Sequence[Define]) – Any initial preprocessor definitions to operate with

Returns:

A result indicating either a successful preprocess or the resulting error

Return type:

PreprocessorResult

preprocess_from_lex(tokens: Sequence[SpannedToken], include_paths: Sequence[str | PathLike[str]], defines: Sequence[Define]) → PreprocessorResult

Same as preprocess(), but operates on the output of lex()

Comparitively, this incurs overhead from copying data between Rust and Python’s ownership models. Only use if you need to modify the output of lex() before preprocessing

Parameters:

• tokens (Sequence[SpannedToken]) – The tokens obtained (and possibly modified) from lex()

• include_paths (Sequence[str | PathLike[str]]) – The include paths to search for include directives

• defines (Sequence[Define]) – Any initial preprocessor definitions to operate with

Returns:

A result indicating either a successful preprocess or the resulting error

Return type:

PreprocessorResult

Parsing

parse(src: str, file_name: str, include_paths: Sequence[str | PathLike[str]], defines: Sequence[Define]) → ParserResult

Parse the token stream into a concrete syntax tree

Parameters:

• src (str) – The content of the source file to parse

• file_name (str) – The name of the source file, used to annotate Spans

• defines (Sequence[Define]) – Any initial preprocessor definitions to operate with

Returns:

A result indicating either a successful parse or the resulting error

Return type:

ParserResult

parse_from_preprocess(tokens: Sequence[SpannedToken]) → ParserResult

Same as parse(), but operates on the output of preprocess()

Comparitively, this incurs overhead from copying data between Rust and Python’s ownership models. Only use if you need to modify the output of preprocess() before parsing

Parameters:

tokens (Sequence[SpannedToken]) – The tokens obtained (and possibly modified) from preprocess() (or possibly lex(), if no preprocessing is required)

Returns:

A result indicating either a successful parse or the resulting error

Return type:

ParserResult

Utility

define_empty(name: str) → Define

Create a Define for a name with no replacement text

Parameters:

name (str) – The name being defined

Returns:

The preprocessor definition

Return type:

Define

define_text(name: str, text: str) → Define

Create a Define for a name with some replacement text

Parameters:

• name (str) – The name being defined

• text (str) – The text to substitute for name when expanded

Returns:

The preprocessor definition

Return type:

Define