7.1 KiB
Native LSL lexer and parser tools
libremetaverse-lsl-tools is the native Rust replacement for the lexer and
parser-generator runtime in the pinned LibreMetaverse.LslTools assembly. The
issue 81 boundary supplies source reading, comments, Unicode character sets,
deterministic DFA execution, reserved words, terminal symbols, EOF, source
locations, diagnostics, and Rust iterators. The issue 82 boundary supplies
grammar productions, parser tables, precedence, reductions, and recovery.
Deterministic checked-in generated grammar and token tables are added by issue
83.
Source and position contract
All public positions are UTF-16 code-unit offsets, matching the original
System.Char API. They are not UTF-8 byte offsets. SourceLineInfo reports a
one-based line and character position while retaining the filtered-line bounds
and the raw character position before removed comments. Non-BMP characters
therefore occupy two positions, exactly as they do in C#.
CsReader accepts strings, files, or explicitly encoded byte slices. It
normalizes CRLF and lone CR to LF, removes // and /* ... */ comments while
preserving their newline structure, tracks the removed UTF-16 lengths for raw
columns, and applies #line N "file" directives. Unterminated block comments,
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
bytes in ASCII mode return positioned errors instead of replacement text.
The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS. UTF-7 and platform code pages are deliberately not delegated to host APIs, so Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16 units and a token to 16 Mi units before unbounded allocation or matching can occur.
Lexer table contract
Dfa is a validated deterministic table of DfaState values and
CharacterMatcher transitions. Matchers support exact UTF-16 values, ranges,
sets, all .NET Unicode general categories, category unions, any character, and
EOF. The runtime applies maximum munch, preserves rule order for overlapping
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
table representation through YyLexer::emit_dfa.
Accepting states carry a TokenDefinition, action number, optional reserved
word table, and one of four actions: emit, skip, emit and change start
condition, or skip and change start condition. Reserved words are exact by
default and use Unicode uppercase only when the table requests the reference
U { ... } behavior. TOKEN preserves token name, number, lexeme, semantic
value, half-open UTF-16 span, and source location. Configured EOF is emitted
once at the end of the filtered buffer.
The old C# enumerators remain callable as compatibility adapters. New Rust code
should use Lexer::iter or Lexer::next_token; both stop deterministically
after a diagnostic and never yield a fabricated token. Token and symbol
Pass members perform real lookup against the mapped parser symbol/literal
tables and populate the supplied parser-entry priority. Missing or malformed
table data is an explicit positioned error.
Grammar and parser contract
Grammar builds validated canonical LR(0) item sets with deterministic LALR(1)
lookahead propagation. Symbols and productions retain stable numeric
identities. Table construction rejects missing start/EOF declarations,
undeclared right-hand-side symbols, invalid precedence declarations, oversized
state sets, incomplete goto tables, and reduction underflow. Empty productions
and recursive nonterminals are supported.
Shift/reduce conflicts use yacc-compatible rules: a declared higher precedence
wins, equal left precedence reduces, equal right precedence shifts, and equal
nonassociative precedence installs a rejecting table entry. Undeclared
shift/reduce conflicts shift; reduce/reduce conflicts select the lower
production number. YyParser::conflicts preserves every decision and emit
writes a byte-stable table description independent of hash iteration.
Parser consumes the native lexer and returns a typed ParseTree through the
mapped SYMBOL semantic value. Syntax recovery uses the conventional terminal
number zero: it pops to a state that can shift error, shifts an explicit error
node, and discards input to the next valid lookahead. Diagnostics distinguish
the original syntax error from successful recovery. Parsing is bounded to
65,536 states, 1,048,576 live stack entries, 16,777,216 operations, and 1,000
recovery attempts.
Compatibility SymbolSet clones share deterministic FIRST/FOLLOW membership,
and cloned CSymbol values share registered production metadata. This retains
the observable reference behavior of the C# grammar model without unsafe code.
Parser-source serialization and the old Builder-driven generated-output
members return InvalidOperation explicitly until issue 83 supplies the
checked-in generator; they never report a false success.
Diagnostics and migration
ErrorHandler collects structured Diagnostic values. Each diagnostic has a
stable numeric code, DiagnosticCategory, severity, message, input fragment,
and SourceLineInfo. Invalid characters and start conditions are recorded at
the offending UTF-16 position. Compatibility exception constructors retain
their original number, input, symbol/token location, handled state, and fatal
or stop behavior. A handler configured to throw increments its counter and
returns immediately without reporting, matching the reference order.
The principal API mapping is:
| C# concept | Native Rust API |
|---|---|
CsReader, LineManager, SourceLineInfo |
Same mapped names, UTF-16-safe source model |
Dfa, Dfa.Action, YyLexer |
Dfa, DfaAction, YyLexer, plus typed state builders |
SYMBOL, TOKEN, EOF, Null |
Same mapped names with owned Rust values |
Lexer._Enumerator |
LexerEnumerator; prefer LexerIterator |
Charset, CatTest |
Same mapped names plus DotNetUnicodeCategory |
CSToolsException, ErrorHandler |
Same mapped names plus structured Diagnostic |
CSymbol, SymbolSet, Production, Precedence |
Same mapped names plus typed Grammar builders |
YyParser, parser entries |
Deterministic native tables and conflict records |
Parser, ParseStackEntry, Error, recoveredError |
Bounded parsing, typed trees, and recovery |
No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code page, or runtime source generation is used by this boundary.
Reproducible verification
Run the issue-owned gates with one build job:
CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
python3 tools/check_milestone_10_issue_81.py
python3 tools/check_milestone_10_issue_82.py
python3 tools/generate_api_shims.py --check
The package contains 37 focused native and compatibility fixtures. The Gitea
workflow runs the audit and workspace compile on ubuntu-latest only.