# Native LSL lexer and parser tools

`metacrate-lsl-tools` is the native Rust replacement for the lexer and
parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
issue 81 boundary supplies source reading, comments, Unicode character sets,
deterministic DFA execution, reserved words, terminal symbols, EOF, source
locations, diagnostics, and Rust iterators. The issue 82 boundary supplies
grammar productions, parser tables, precedence, reductions, and recovery.
The issue 83 boundary adds deterministic checked-in grammar/token tables and
native replacements for the retired parser-generator object model.

## Source and position contract

All public positions are UTF-16 code-unit offsets, matching the original
`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
one-based line and character position while retaining the filtered-line bounds
and the raw character position before removed comments. Non-BMP characters
therefore occupy two positions, exactly as they do in C#.

`CsReader` accepts strings, files, or explicitly encoded byte slices. It
normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
preserving their newline structure, tracks the removed UTF-16 lengths for raw
columns, and applies `#line N "file"` directives. Unterminated block comments,
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
bytes in ASCII mode return positioned errors instead of replacement text.

The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
UTF-7 and platform code pages are deliberately not delegated to host APIs, so
Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
units and a token to 16 Mi units before unbounded allocation or matching can
occur.

## Lexer table contract

`Dfa` is a validated deterministic table of `DfaState` values and
`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
sets, all .NET Unicode general categories, category unions, any character, and
EOF. The runtime applies maximum munch, preserves rule order for overlapping
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
table representation through `YyLexer::emit_dfa`.

Accepting states carry a `TokenDefinition`, action number, optional reserved
word table, and one of four actions: emit, skip, emit and change start
condition, or skip and change start condition. Reserved words are exact by
default and use Unicode uppercase only when the table requests the reference
`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
value, half-open UTF-16 span, and source location. Configured EOF is emitted
once at the end of the filtered buffer.

The old C# enumerators remain callable as compatibility adapters. New Rust code
should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
after a diagnostic and never yield a fabricated token. Token and symbol
`Pass` members perform real lookup against the mapped parser symbol/literal
tables and populate the supplied parser-entry priority. Missing or malformed
table data is an explicit positioned error.

## Grammar and parser contract

`Grammar` builds validated canonical LR(0) item sets with deterministic LALR(1)
lookahead propagation. Symbols and productions retain stable numeric
identities. Table construction rejects missing start/EOF declarations,
undeclared right-hand-side symbols, invalid precedence declarations, oversized
state sets, incomplete goto tables, and reduction underflow. Empty productions
and recursive nonterminals are supported.

Shift/reduce conflicts use yacc-compatible rules: a declared higher precedence
wins, equal left precedence reduces, equal right precedence shifts, and equal
nonassociative precedence installs a rejecting table entry. Undeclared
shift/reduce conflicts shift; reduce/reduce conflicts select the lower
production number. `YyParser::conflicts` preserves every decision and `emit`
writes a byte-stable table description independent of hash iteration.

`Parser` consumes the native lexer and returns a typed `ParseTree` through the
mapped `SYMBOL` semantic value. Syntax recovery uses the conventional terminal
number zero: it pops to a state that can shift `error`, shifts an explicit error
node, and discards input to the next valid lookahead. Diagnostics distinguish
the original syntax error from successful recovery. Parsing is bounded to
65,536 states, 1,048,576 live stack entries, 16,777,216 operations, and 1,000
recovery attempts.

Compatibility `SymbolSet` clones share deterministic FIRST/FOLLOW membership,
and cloned `CSymbol` values share registered production metadata. This retains
the observable reference behavior of the C# grammar model without unsafe code.
Legacy parser-runtime serialization members that cannot faithfully round-trip
the richer native LALR machine remain explicit `InvalidOperation` results.
Callers should use the stable `YyParser::emit` and `YyLexer::emit_dfa` formats,
or regenerate the canonical checked-in table module described below.

## Checked-in generator contract

[`codegen/inputs/lsl_tools_grammar.json`](../../codegen/inputs/lsl_tools_grammar.json)
is the reviewed source of truth reconstructed from the pinned 4.5
`yycs0syntax.cs` and `yycs0tokens.cs` tables at upstream commit
`2aa70bb68513b39795da5d13c88f31b86e85a3ba`. It records every token,
nonterminal, production, and semantic-action class. The portable Python
generator uses the standard library plus the workspace Rust formatter and emits
`src/generated_tables.rs`; it does not invoke C#, a C# source generator, or a
.NET runtime.

Regenerate with `python3 tools/generate_lsl_tables.py`. Use
`python3 tools/generate_lsl_tables.py --check` in reviews and CI. The check
renders the complete output in memory and compares bytes, so repeated runs are
independent of hash iteration, locale, host operating system, and timestamps.
The checked-in module supplies the canonical 25-production grammar, a Unicode
lexer with reserved `base`, `this`, and `new` tokens, the 84 generated syntax
classes, the 13 generated token classes, and the four `cs0` runtime wrappers.

The old `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, NFA, delegate, factory,
and 4.5 `Serialiser` APIs are live Rust compatibility adapters. New code should
prefer `Grammar`, typed `DfaState` builders, `generated_parser`, and
`generated_lexer`; these avoid CLR-style method pointers and class-shaped
generated inheritance while retaining deterministic mapped behavior.

## Diagnostics and migration

`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
and `SourceLineInfo`. Invalid characters and start conditions are recorded at
the offending UTF-16 position. Compatibility exception constructors retain
their original number, input, symbol/token location, handled state, and fatal
or stop behavior. A handler configured to throw increments its counter and
returns immediately without reporting, matching the reference order.

The principal API mapping is:

| C# concept | Native Rust API |
| --- | --- |
| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
| `CSymbol`, `SymbolSet`, `Production`, `Precedence` | Same mapped names plus typed `Grammar` builders |
| `YyParser`, parser entries | Deterministic native tables and conflict records |
| `Parser`, `ParseStackEntry`, `Error`, `recoveredError` | Bounded parsing, typed trees, and recovery |
| `yycs0syntax`, `yycs0tokens` | `generated_parser`, `generated_lexer`, checked-in native types |
| `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, `Nfa` | Portable Rust generator adapters; prefer typed grammar/DFA builders |
| `Serialiser` | Deterministic 4.5-compatible integer stream for supported mapped values |

No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
page, or runtime source generation is used by this boundary.

## Reproducible verification

Run the issue-owned gates with one build job:

```sh
CARGO_BUILD_JOBS=1 cargo test -p metacrate-lsl-tools --locked
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
CARGO_BUILD_JOBS=1 cargo clippy -p metacrate-lsl-tools --all-targets --locked -- -D warnings
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p metacrate-lsl-tools --no-deps --locked
python3 tools/check_milestone_10_issue_81.py
python3 tools/check_milestone_10_issue_82.py
python3 tools/check_milestone_10_issue_83.py
python3 tools/generate_lsl_tables.py --check
python3 tools/generate_api_shims.py --check
```

The package contains 46 focused native and compatibility fixtures. The Gitea
workflow runs the audit and workspace compile on `ubuntu-latest` only.