Chili Palmer
73cef10054
Some checks failed
Native code generation / deterministic (push) Failing after 8m20s
Imaging and meshing gate / native (push) Successful in 5m20s
JPEG 2000 feature / linux (push) Successful in 2m46s
Native Rust workspace compile / compile (push) Failing after 7m14s
Skia feature / linux (push) Successful in 30m39s
95 lines
4.9 KiB
Markdown
95 lines
4.9 KiB
Markdown
# Native LSL lexer and parser tools
|
|
|
|
`libremetaverse-lsl-tools` is the native Rust replacement for the lexer and
|
|
parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
|
|
issue 81 boundary supplies source reading, comments, Unicode character sets,
|
|
deterministic DFA execution, reserved words, terminal symbols, EOF, source
|
|
locations, diagnostics, and Rust iterators. Grammar productions, reductions,
|
|
and recovery are added by issue 82; deterministic generated grammar and token
|
|
tables are added by issue 83.
|
|
|
|
## Source and position contract
|
|
|
|
All public positions are UTF-16 code-unit offsets, matching the original
|
|
`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
|
|
one-based line and character position while retaining the filtered-line bounds
|
|
and the raw character position before removed comments. Non-BMP characters
|
|
therefore occupy two positions, exactly as they do in C#.
|
|
|
|
`CsReader` accepts strings, files, or explicitly encoded byte slices. It
|
|
normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
|
|
preserving their newline structure, tracks the removed UTF-16 lengths for raw
|
|
columns, and applies `#line N "file"` directives. Unterminated block comments,
|
|
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
|
|
bytes in ASCII mode return positioned errors instead of replacement text.
|
|
|
|
The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
|
|
UTF-7 and platform code pages are deliberately not delegated to host APIs, so
|
|
Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
|
|
units and a token to 16 Mi units before unbounded allocation or matching can
|
|
occur.
|
|
|
|
## Lexer table contract
|
|
|
|
`Dfa` is a validated deterministic table of `DfaState` values and
|
|
`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
|
|
sets, all .NET Unicode general categories, category unions, any character, and
|
|
EOF. The runtime applies maximum munch, preserves rule order for overlapping
|
|
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
|
|
table representation through `YyLexer::emit_dfa`.
|
|
|
|
Accepting states carry a `TokenDefinition`, action number, optional reserved
|
|
word table, and one of four actions: emit, skip, emit and change start
|
|
condition, or skip and change start condition. Reserved words are exact by
|
|
default and use Unicode uppercase only when the table requests the reference
|
|
`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
|
|
value, half-open UTF-16 span, and source location. Configured EOF is emitted
|
|
once at the end of the filtered buffer.
|
|
|
|
The old C# enumerators remain callable as compatibility adapters. New Rust code
|
|
should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
|
|
after a diagnostic and never yield a fabricated token. Token and symbol
|
|
`Pass` members perform real lookup against the mapped parser symbol/literal
|
|
tables and populate the supplied parser-entry priority. Missing or malformed
|
|
table data is an explicit positioned error.
|
|
|
|
## Diagnostics and migration
|
|
|
|
`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
|
|
stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
|
|
and `SourceLineInfo`. Invalid characters and start conditions are recorded at
|
|
the offending UTF-16 position. Compatibility exception constructors retain
|
|
their original number, input, symbol/token location, handled state, and fatal
|
|
or stop behavior. A handler configured to throw increments its counter and
|
|
returns immediately without reporting, matching the reference order.
|
|
|
|
The principal API mapping is:
|
|
|
|
| C# concept | Native Rust API |
|
|
| --- | --- |
|
|
| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
|
|
| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
|
|
| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
|
|
| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
|
|
| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
|
|
| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
|
|
|
|
No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
|
|
page, or runtime source generation is used by this boundary.
|
|
|
|
## Reproducible verification
|
|
|
|
Run the issue-owned gates with one build job:
|
|
|
|
```sh
|
|
CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
|
|
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
|
|
CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
|
|
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
|
|
python3 tools/check_milestone_10_issue_81.py
|
|
python3 tools/generate_api_shims.py --check
|
|
```
|
|
|
|
The package contains 27 focused native and compatibility fixtures. The Gitea
|
|
workflow runs the audit and workspace compile on `ubuntu-latest` only.
|