Implement safe public LSL delivery (#129)

crates/metacrate-lsl-tools/README.md

@@ -0,0 +1,161 @@
+# Native LSL lexer and parser tools
+
+`metacrate-lsl-tools` is the native Rust replacement for the lexer and
+parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
+issue 81 boundary supplies source reading, comments, Unicode character sets,
+deterministic DFA execution, reserved words, terminal symbols, EOF, source
+locations, diagnostics, and Rust iterators. The issue 82 boundary supplies
+grammar productions, parser tables, precedence, reductions, and recovery.
+The issue 83 boundary adds deterministic checked-in grammar/token tables and
+native replacements for the retired parser-generator object model.
+
+## Source and position contract
+
+All public positions are UTF-16 code-unit offsets, matching the original
+`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
+one-based line and character position while retaining the filtered-line bounds
+and the raw character position before removed comments. Non-BMP characters
+therefore occupy two positions, exactly as they do in C#.
+
+`CsReader` accepts strings, files, or explicitly encoded byte slices. It
+normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
+preserving their newline structure, tracks the removed UTF-16 lengths for raw
+columns, and applies `#line N "file"` directives. Unterminated block comments,
+invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
+bytes in ASCII mode return positioned errors instead of replacement text.
+
+The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
+UTF-7 and platform code pages are deliberately not delegated to host APIs, so
+Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
+units and a token to 16 Mi units before unbounded allocation or matching can
+occur.
+
+## Lexer table contract
+
+`Dfa` is a validated deterministic table of `DfaState` values and
+`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
+sets, all .NET Unicode general categories, category unions, any character, and
+EOF. The runtime applies maximum munch, preserves rule order for overlapping
+transitions, rejects empty non-EOF matches, and exposes a deterministic textual
+table representation through `YyLexer::emit_dfa`.
+
+Accepting states carry a `TokenDefinition`, action number, optional reserved
+word table, and one of four actions: emit, skip, emit and change start
+condition, or skip and change start condition. Reserved words are exact by
+default and use Unicode uppercase only when the table requests the reference
+`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
+value, half-open UTF-16 span, and source location. Configured EOF is emitted
+once at the end of the filtered buffer.
+
+The old C# enumerators remain callable as compatibility adapters. New Rust code
+should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
+after a diagnostic and never yield a fabricated token. Token and symbol
+`Pass` members perform real lookup against the mapped parser symbol/literal
+tables and populate the supplied parser-entry priority. Missing or malformed
+table data is an explicit positioned error.
+
+## Grammar and parser contract
+
+`Grammar` builds validated canonical LR(0) item sets with deterministic LALR(1)
+lookahead propagation. Symbols and productions retain stable numeric
+identities. Table construction rejects missing start/EOF declarations,
+undeclared right-hand-side symbols, invalid precedence declarations, oversized
+state sets, incomplete goto tables, and reduction underflow. Empty productions
+and recursive nonterminals are supported.
+
+Shift/reduce conflicts use yacc-compatible rules: a declared higher precedence
+wins, equal left precedence reduces, equal right precedence shifts, and equal
+nonassociative precedence installs a rejecting table entry. Undeclared
+shift/reduce conflicts shift; reduce/reduce conflicts select the lower
+production number. `YyParser::conflicts` preserves every decision and `emit`
+writes a byte-stable table description independent of hash iteration.
+
+`Parser` consumes the native lexer and returns a typed `ParseTree` through the
+mapped `SYMBOL` semantic value. Syntax recovery uses the conventional terminal
+number zero: it pops to a state that can shift `error`, shifts an explicit error
+node, and discards input to the next valid lookahead. Diagnostics distinguish
+the original syntax error from successful recovery. Parsing is bounded to
+65,536 states, 1,048,576 live stack entries, 16,777,216 operations, and 1,000
+recovery attempts.
+
+Compatibility `SymbolSet` clones share deterministic FIRST/FOLLOW membership,
+and cloned `CSymbol` values share registered production metadata. This retains
+the observable reference behavior of the C# grammar model without unsafe code.
+Legacy parser-runtime serialization members that cannot faithfully round-trip
+the richer native LALR machine remain explicit `InvalidOperation` results.
+Callers should use the stable `YyParser::emit` and `YyLexer::emit_dfa` formats,
+or regenerate the canonical checked-in table module described below.
+
+## Checked-in generator contract
+
+[`codegen/inputs/lsl_tools_grammar.json`](../../codegen/inputs/lsl_tools_grammar.json)
+is the reviewed source of truth reconstructed from the pinned 4.5
+`yycs0syntax.cs` and `yycs0tokens.cs` tables at upstream commit
+`2aa70bb68513b39795da5d13c88f31b86e85a3ba`. It records every token,
+nonterminal, production, and semantic-action class. The portable Python
+generator uses the standard library plus the workspace Rust formatter and emits
+`src/generated_tables.rs`; it does not invoke C#, a C# source generator, or a
+.NET runtime.
+
+Regenerate with `python3 tools/generate_lsl_tables.py`. Use
+`python3 tools/generate_lsl_tables.py --check` in reviews and CI. The check
+renders the complete output in memory and compares bytes, so repeated runs are
+independent of hash iteration, locale, host operating system, and timestamps.
+The checked-in module supplies the canonical 25-production grammar, a Unicode
+lexer with reserved `base`, `this`, and `new` tokens, the 84 generated syntax
+classes, the 13 generated token classes, and the four `cs0` runtime wrappers.
+
+The old `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, NFA, delegate, factory,
+and 4.5 `Serialiser` APIs are live Rust compatibility adapters. New code should
+prefer `Grammar`, typed `DfaState` builders, `generated_parser`, and
+`generated_lexer`; these avoid CLR-style method pointers and class-shaped
+generated inheritance while retaining deterministic mapped behavior.
+
+## Diagnostics and migration
+
+`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
+stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
+and `SourceLineInfo`. Invalid characters and start conditions are recorded at
+the offending UTF-16 position. Compatibility exception constructors retain
+their original number, input, symbol/token location, handled state, and fatal
+or stop behavior. A handler configured to throw increments its counter and
+returns immediately without reporting, matching the reference order.
+
+The principal API mapping is:
+
+| C# concept | Native Rust API |
+| --- | --- |
+| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
+| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
+| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
+| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
+| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
+| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
+| `CSymbol`, `SymbolSet`, `Production`, `Precedence` | Same mapped names plus typed `Grammar` builders |
+| `YyParser`, parser entries | Deterministic native tables and conflict records |
+| `Parser`, `ParseStackEntry`, `Error`, `recoveredError` | Bounded parsing, typed trees, and recovery |
+| `yycs0syntax`, `yycs0tokens` | `generated_parser`, `generated_lexer`, checked-in native types |
+| `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, `Nfa` | Portable Rust generator adapters; prefer typed grammar/DFA builders |
+| `Serialiser` | Deterministic 4.5-compatible integer stream for supported mapped values |
+
+No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
+page, or runtime source generation is used by this boundary.
+
+## Reproducible verification
+
+Run the issue-owned gates with one build job:
+
+```sh
+CARGO_BUILD_JOBS=1 cargo test -p metacrate-lsl-tools --locked
+CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
+CARGO_BUILD_JOBS=1 cargo clippy -p metacrate-lsl-tools --all-targets --locked -- -D warnings
+RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p metacrate-lsl-tools --no-deps --locked
+python3 tools/check_milestone_10_issue_81.py
+python3 tools/check_milestone_10_issue_82.py
+python3 tools/check_milestone_10_issue_83.py
+python3 tools/generate_lsl_tables.py --check
+python3 tools/generate_api_shims.py --check
+```
+
+The package contains 46 focused native and compatibility fixtures. The Gitea
+workflow runs the audit and workspace compile on `ubuntu-latest` only.