Implement native LSL lexer runtime (#81)

crates/libremetaverse-lsl-tools/README.md

@@ -0,0 +1,94 @@
+# Native LSL lexer and parser tools
+
+`libremetaverse-lsl-tools` is the native Rust replacement for the lexer and
+parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
+issue 81 boundary supplies source reading, comments, Unicode character sets,
+deterministic DFA execution, reserved words, terminal symbols, EOF, source
+locations, diagnostics, and Rust iterators. Grammar productions, reductions,
+and recovery are added by issue 82; deterministic generated grammar and token
+tables are added by issue 83.
+
+## Source and position contract
+
+All public positions are UTF-16 code-unit offsets, matching the original
+`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
+one-based line and character position while retaining the filtered-line bounds
+and the raw character position before removed comments. Non-BMP characters
+therefore occupy two positions, exactly as they do in C#.
+
+`CsReader` accepts strings, files, or explicitly encoded byte slices. It
+normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
+preserving their newline structure, tracks the removed UTF-16 lengths for raw
+columns, and applies `#line N "file"` directives. Unterminated block comments,
+invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
+bytes in ASCII mode return positioned errors instead of replacement text.
+
+The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
+UTF-7 and platform code pages are deliberately not delegated to host APIs, so
+Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
+units and a token to 16 Mi units before unbounded allocation or matching can
+occur.
+
+## Lexer table contract
+
+`Dfa` is a validated deterministic table of `DfaState` values and
+`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
+sets, all .NET Unicode general categories, category unions, any character, and
+EOF. The runtime applies maximum munch, preserves rule order for overlapping
+transitions, rejects empty non-EOF matches, and exposes a deterministic textual
+table representation through `YyLexer::emit_dfa`.
+
+Accepting states carry a `TokenDefinition`, action number, optional reserved
+word table, and one of four actions: emit, skip, emit and change start
+condition, or skip and change start condition. Reserved words are exact by
+default and use Unicode uppercase only when the table requests the reference
+`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
+value, half-open UTF-16 span, and source location. Configured EOF is emitted
+once at the end of the filtered buffer.
+
+The old C# enumerators remain callable as compatibility adapters. New Rust code
+should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
+after a diagnostic and never yield a fabricated token. Token and symbol
+`Pass` members perform real lookup against the mapped parser symbol/literal
+tables and populate the supplied parser-entry priority. Missing or malformed
+table data is an explicit positioned error.
+
+## Diagnostics and migration
+
+`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
+stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
+and `SourceLineInfo`. Invalid characters and start conditions are recorded at
+the offending UTF-16 position. Compatibility exception constructors retain
+their original number, input, symbol/token location, handled state, and fatal
+or stop behavior. A handler configured to throw increments its counter and
+returns immediately without reporting, matching the reference order.
+
+The principal API mapping is:
+
+| C# concept | Native Rust API |
+| --- | --- |
+| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
+| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
+| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
+| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
+| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
+| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
+
+No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
+page, or runtime source generation is used by this boundary.
+
+## Reproducible verification
+
+Run the issue-owned gates with one build job:
+
+```sh
+CARGO_BUILD_JOBS=1 cargo test -p libremetaverse-lsl-tools --locked
+CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
+CARGO_BUILD_JOBS=1 cargo clippy -p libremetaverse-lsl-tools --all-targets --locked -- -D warnings
+RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p libremetaverse-lsl-tools --no-deps --locked
+python3 tools/check_milestone_10_issue_81.py
+python3 tools/generate_api_shims.py --check
+```
+
+The package contains 27 focused native and compatibility fixtures. The Gitea
+workflow runs the audit and workspace compile on `ubuntu-latest` only.