MetaCrate/crates/libremetaverse-lsl-tools/README.md

Native LSL lexer and parser tools

libremetaverse-lsl-tools is the native Rust replacement for the lexer and parser-generator runtime in the pinned LibreMetaverse.LslTools assembly. The issue 81 boundary supplies source reading, comments, Unicode character sets, deterministic DFA execution, reserved words, terminal symbols, EOF, source locations, diagnostics, and Rust iterators. Grammar productions, reductions, and recovery are added by issue 82; deterministic generated grammar and token tables are added by issue 83.

Source and position contract

All public positions are UTF-16 code-unit offsets, matching the original System.Char API. They are not UTF-8 byte offsets. SourceLineInfo reports a one-based line and character position while retaining the filtered-line bounds and the raw character position before removed comments. Non-BMP characters therefore occupy two positions, exactly as they do in C#.

CsReader accepts strings, files, or explicitly encoded byte slices. It normalizes CRLF and lone CR to LF, removes // and /* ... */ comments while preserving their newline structure, tracks the removed UTF-16 lengths for raw columns, and applies #line N "file" directives. Unterminated block comments, invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII bytes in ASCII mode return positioned errors instead of replacement text.

The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS. UTF-7 and platform code pages are deliberately not delegated to host APIs, so Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16 units and a token to 16 Mi units before unbounded allocation or matching can occur.

Lexer table contract

Dfa is a validated deterministic table of DfaState values and CharacterMatcher transitions. Matchers support exact UTF-16 values, ranges, sets, all .NET Unicode general categories, category unions, any character, and EOF. The runtime applies maximum munch, preserves rule order for overlapping transitions, rejects empty non-EOF matches, and exposes a deterministic textual table representation through YyLexer::emit_dfa.

Accepting states carry a TokenDefinition, action number, optional reserved word table, and one of four actions: emit, skip, emit and change start condition, or skip and change start condition. Reserved words are exact by default and use Unicode uppercase only when the table requests the reference U { ... } behavior. TOKEN preserves token name, number, lexeme, semantic value, half-open UTF-16 span, and source location. Configured EOF is emitted once at the end of the filtered buffer.

The old C# enumerators remain callable as compatibility adapters. New Rust code should use Lexer::iter or Lexer::next_token; both stop deterministically after a diagnostic and never yield a fabricated token. Token and symbol Pass members perform real lookup against the mapped parser symbol/literal tables and populate the supplied parser-entry priority. Missing or malformed table data is an explicit positioned error.

Diagnostics and migration

ErrorHandler collects structured Diagnostic values. Each diagnostic has a stable numeric code, DiagnosticCategory, severity, message, input fragment, and SourceLineInfo. Invalid characters and start conditions are recorded at the offending UTF-16 position. Compatibility exception constructors retain their original number, input, symbol/token location, handled state, and fatal or stop behavior. A handler configured to throw increments its counter and returns immediately without reporting, matching the reference order.

The principal API mapping is:

C# concept	Native Rust API
CsReader, LineManager, SourceLineInfo	Same mapped names, UTF-16-safe source model
Dfa, Dfa.Action, YyLexer	Dfa, DfaAction, YyLexer, plus typed state builders
SYMBOL, TOKEN, EOF, Null	Same mapped names with owned Rust values
Lexer._Enumerator	LexerEnumerator; prefer LexerIterator
Charset, CatTest	Same mapped names plus DotNetUnicodeCategory
CSToolsException, ErrorHandler	Same mapped names plus structured Diagnostic

No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code page, or runtime source generation is used by this boundary.

Reproducible verification

Run the issue-owned gates with one build job:

CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
python3 tools/check_milestone_10_issue_81.py
python3 tools/generate_api_shims.py --check

The package contains 27 focused native and compatibility fixtures. The Gitea workflow runs the audit and workspace compile on ubuntu-latest only.