Chili Palmer
2558e2f49f
Some checks failed
Native code generation / deterministic (push) Failing after 2m36s
Imaging and meshing gate / native (push) Successful in 5m29s
JPEG 2000 feature / linux (push) Successful in 2m46s
Native Rust workspace compile / compile (push) Failing after 1m58s
Skia feature / linux (push) Successful in 31m14s
162 lines
9.1 KiB
Markdown
162 lines
9.1 KiB
Markdown
# Native LSL lexer and parser tools
|
|
|
|
`libremetaverse-lsl-tools` is the native Rust replacement for the lexer and
|
|
parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
|
|
issue 81 boundary supplies source reading, comments, Unicode character sets,
|
|
deterministic DFA execution, reserved words, terminal symbols, EOF, source
|
|
locations, diagnostics, and Rust iterators. The issue 82 boundary supplies
|
|
grammar productions, parser tables, precedence, reductions, and recovery.
|
|
The issue 83 boundary adds deterministic checked-in grammar/token tables and
|
|
native replacements for the retired parser-generator object model.
|
|
|
|
## Source and position contract
|
|
|
|
All public positions are UTF-16 code-unit offsets, matching the original
|
|
`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
|
|
one-based line and character position while retaining the filtered-line bounds
|
|
and the raw character position before removed comments. Non-BMP characters
|
|
therefore occupy two positions, exactly as they do in C#.
|
|
|
|
`CsReader` accepts strings, files, or explicitly encoded byte slices. It
|
|
normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
|
|
preserving their newline structure, tracks the removed UTF-16 lengths for raw
|
|
columns, and applies `#line N "file"` directives. Unterminated block comments,
|
|
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
|
|
bytes in ASCII mode return positioned errors instead of replacement text.
|
|
|
|
The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
|
|
UTF-7 and platform code pages are deliberately not delegated to host APIs, so
|
|
Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
|
|
units and a token to 16 Mi units before unbounded allocation or matching can
|
|
occur.
|
|
|
|
## Lexer table contract
|
|
|
|
`Dfa` is a validated deterministic table of `DfaState` values and
|
|
`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
|
|
sets, all .NET Unicode general categories, category unions, any character, and
|
|
EOF. The runtime applies maximum munch, preserves rule order for overlapping
|
|
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
|
|
table representation through `YyLexer::emit_dfa`.
|
|
|
|
Accepting states carry a `TokenDefinition`, action number, optional reserved
|
|
word table, and one of four actions: emit, skip, emit and change start
|
|
condition, or skip and change start condition. Reserved words are exact by
|
|
default and use Unicode uppercase only when the table requests the reference
|
|
`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
|
|
value, half-open UTF-16 span, and source location. Configured EOF is emitted
|
|
once at the end of the filtered buffer.
|
|
|
|
The old C# enumerators remain callable as compatibility adapters. New Rust code
|
|
should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
|
|
after a diagnostic and never yield a fabricated token. Token and symbol
|
|
`Pass` members perform real lookup against the mapped parser symbol/literal
|
|
tables and populate the supplied parser-entry priority. Missing or malformed
|
|
table data is an explicit positioned error.
|
|
|
|
## Grammar and parser contract
|
|
|
|
`Grammar` builds validated canonical LR(0) item sets with deterministic LALR(1)
|
|
lookahead propagation. Symbols and productions retain stable numeric
|
|
identities. Table construction rejects missing start/EOF declarations,
|
|
undeclared right-hand-side symbols, invalid precedence declarations, oversized
|
|
state sets, incomplete goto tables, and reduction underflow. Empty productions
|
|
and recursive nonterminals are supported.
|
|
|
|
Shift/reduce conflicts use yacc-compatible rules: a declared higher precedence
|
|
wins, equal left precedence reduces, equal right precedence shifts, and equal
|
|
nonassociative precedence installs a rejecting table entry. Undeclared
|
|
shift/reduce conflicts shift; reduce/reduce conflicts select the lower
|
|
production number. `YyParser::conflicts` preserves every decision and `emit`
|
|
writes a byte-stable table description independent of hash iteration.
|
|
|
|
`Parser` consumes the native lexer and returns a typed `ParseTree` through the
|
|
mapped `SYMBOL` semantic value. Syntax recovery uses the conventional terminal
|
|
number zero: it pops to a state that can shift `error`, shifts an explicit error
|
|
node, and discards input to the next valid lookahead. Diagnostics distinguish
|
|
the original syntax error from successful recovery. Parsing is bounded to
|
|
65,536 states, 1,048,576 live stack entries, 16,777,216 operations, and 1,000
|
|
recovery attempts.
|
|
|
|
Compatibility `SymbolSet` clones share deterministic FIRST/FOLLOW membership,
|
|
and cloned `CSymbol` values share registered production metadata. This retains
|
|
the observable reference behavior of the C# grammar model without unsafe code.
|
|
Legacy parser-runtime serialization members that cannot faithfully round-trip
|
|
the richer native LALR machine remain explicit `InvalidOperation` results.
|
|
Callers should use the stable `YyParser::emit` and `YyLexer::emit_dfa` formats,
|
|
or regenerate the canonical checked-in table module described below.
|
|
|
|
## Checked-in generator contract
|
|
|
|
[`codegen/inputs/lsl_tools_grammar.json`](../../codegen/inputs/lsl_tools_grammar.json)
|
|
is the reviewed source of truth reconstructed from the pinned 4.5
|
|
`yycs0syntax.cs` and `yycs0tokens.cs` tables at upstream commit
|
|
`2aa70bb68513b39795da5d13c88f31b86e85a3ba`. It records every token,
|
|
nonterminal, production, and semantic-action class. The portable Python
|
|
generator uses the standard library plus the workspace Rust formatter and emits
|
|
`src/generated_tables.rs`; it does not invoke C#, a C# source generator, or a
|
|
.NET runtime.
|
|
|
|
Regenerate with `python3 tools/generate_lsl_tables.py`. Use
|
|
`python3 tools/generate_lsl_tables.py --check` in reviews and CI. The check
|
|
renders the complete output in memory and compares bytes, so repeated runs are
|
|
independent of hash iteration, locale, host operating system, and timestamps.
|
|
The checked-in module supplies the canonical 25-production grammar, a Unicode
|
|
lexer with reserved `base`, `this`, and `new` tokens, the 84 generated syntax
|
|
classes, the 13 generated token classes, and the four `cs0` runtime wrappers.
|
|
|
|
The old `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, NFA, delegate, factory,
|
|
and 4.5 `Serialiser` APIs are live Rust compatibility adapters. New code should
|
|
prefer `Grammar`, typed `DfaState` builders, `generated_parser`, and
|
|
`generated_lexer`; these avoid CLR-style method pointers and class-shaped
|
|
generated inheritance while retaining deterministic mapped behavior.
|
|
|
|
## Diagnostics and migration
|
|
|
|
`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
|
|
stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
|
|
and `SourceLineInfo`. Invalid characters and start conditions are recorded at
|
|
the offending UTF-16 position. Compatibility exception constructors retain
|
|
their original number, input, symbol/token location, handled state, and fatal
|
|
or stop behavior. A handler configured to throw increments its counter and
|
|
returns immediately without reporting, matching the reference order.
|
|
|
|
The principal API mapping is:
|
|
|
|
| C# concept | Native Rust API |
|
|
| --- | --- |
|
|
| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
|
|
| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
|
|
| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
|
|
| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
|
|
| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
|
|
| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
|
|
| `CSymbol`, `SymbolSet`, `Production`, `Precedence` | Same mapped names plus typed `Grammar` builders |
|
|
| `YyParser`, parser entries | Deterministic native tables and conflict records |
|
|
| `Parser`, `ParseStackEntry`, `Error`, `recoveredError` | Bounded parsing, typed trees, and recovery |
|
|
| `yycs0syntax`, `yycs0tokens` | `generated_parser`, `generated_lexer`, checked-in native types |
|
|
| `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, `Nfa` | Portable Rust generator adapters; prefer typed grammar/DFA builders |
|
|
| `Serialiser` | Deterministic 4.5-compatible integer stream for supported mapped values |
|
|
|
|
No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
|
|
page, or runtime source generation is used by this boundary.
|
|
|
|
## Reproducible verification
|
|
|
|
Run the issue-owned gates with one build job:
|
|
|
|
```sh
|
|
CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
|
|
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
|
|
CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
|
|
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
|
|
python3 tools/check_milestone_10_issue_81.py
|
|
python3 tools/check_milestone_10_issue_82.py
|
|
python3 tools/check_milestone_10_issue_83.py
|
|
python3 tools/generate_lsl_tables.py --check
|
|
python3 tools/generate_api_shims.py --check
|
|
```
|
|
|
|
The package contains 46 focused native and compatibility fixtures. The Gitea
|
|
workflow runs the audit and workspace compile on `ubuntu-latest` only.
|