# Native LSL lexer and parser tools

`libremetaverse-lsl-tools` is the native Rust replacement for the lexer and
parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
issue 81 boundary supplies source reading, comments, Unicode character sets,
deterministic DFA execution, reserved words, terminal symbols, EOF, source
locations, diagnostics, and Rust iterators. Grammar productions, reductions,
and recovery are added by issue 82; deterministic generated grammar and token
tables are added by issue 83.

## Source and position contract

All public positions are UTF-16 code-unit offsets, matching the original
`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
one-based line and character position while retaining the filtered-line bounds
and the raw character position before removed comments. Non-BMP characters
therefore occupy two positions, exactly as they do in C#.

`CsReader` accepts strings, files, or explicitly encoded byte slices. It
normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
preserving their newline structure, tracks the removed UTF-16 lengths for raw
columns, and applies `#line N "file"` directives. Unterminated block comments,
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
bytes in ASCII mode return positioned errors instead of replacement text.

The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
UTF-7 and platform code pages are deliberately not delegated to host APIs, so
Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
units and a token to 16 Mi units before unbounded allocation or matching can
occur.

## Lexer table contract

`Dfa` is a validated deterministic table of `DfaState` values and
`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
sets, all .NET Unicode general categories, category unions, any character, and
EOF. The runtime applies maximum munch, preserves rule order for overlapping
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
table representation through `YyLexer::emit_dfa`.

Accepting states carry a `TokenDefinition`, action number, optional reserved
word table, and one of four actions: emit, skip, emit and change start
condition, or skip and change start condition. Reserved words are exact by
default and use Unicode uppercase only when the table requests the reference
`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
value, half-open UTF-16 span, and source location. Configured EOF is emitted
once at the end of the filtered buffer.

The old C# enumerators remain callable as compatibility adapters. New Rust code
should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
after a diagnostic and never yield a fabricated token. Token and symbol
`Pass` members perform real lookup against the mapped parser symbol/literal
tables and populate the supplied parser-entry priority. Missing or malformed
table data is an explicit positioned error.

## Diagnostics and migration

`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
and `SourceLineInfo`. Invalid characters and start conditions are recorded at
the offending UTF-16 position. Compatibility exception constructors retain
their original number, input, symbol/token location, handled state, and fatal
or stop behavior. A handler configured to throw increments its counter and
returns immediately without reporting, matching the reference order.

The principal API mapping is:

| C# concept | Native Rust API |
| --- | --- |
| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |

No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
page, or runtime source generation is used by this boundary.

## Reproducible verification

Run the issue-owned gates with one build job:

```sh
CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
python3 tools/check_milestone_10_issue_81.py
python3 tools/generate_api_shims.py --check
```

The package contains 27 focused native and compatibility fixtures. The Gitea
workflow runs the audit and workspace compile on `ubuntu-latest` only.