No generated YY class behavior is silently omitted.
Prerequisite
All global API/test gates and the foundational/world implementations needed by these extension crates are complete.
Project constraints
Full native Rust implementation only; no wrapped C# parser, viewer, renderer, or service.
Preserve mapped APIs and exact translated tests; reuse lower layers and keep generated output deterministic.
Validate bounded input, callbacks/tasks, docs, coverage, and workspace gates.
## Objective
Implement public token/symbol types, character sets, comments, source line management, lexer behavior, and parser diagnostics in native Rust.
## Deliverables
- Preserve tokenization, Unicode/category behavior, comments, keywords, literals, positions, EOF, error text/categories, and recovery entry points.
- Replace C# enumerators with Rust iterators while retaining mapped public concepts.
- Use bounded inputs and deterministic output.
## Validation
- All lexer/token/diagnostic fixtures pass.
- Invalid input reports matching locations/categories.
- No generated YY class behavior is silently omitted.
## Prerequisite
All global API/test gates and the foundational/world implementations needed by these extension crates are complete.
## Project constraints
- Full native Rust implementation only; no wrapped C# parser, viewer, renderer, or service.
- Preserve mapped APIs and exact translated tests; reuse lower layers and keep generated output deterministic.
- Validate bounded input, callbacks/tasks, docs, coverage, and workspace gates.
hugo
added this to the 10 - Rendering, RLV, and LSL tools milestone 2026-08-08 08:23:18 +00:00
This replaces the LSL lexer boundary with native Rust: mapped token/symbol/EOF and compatibility enumerator APIs; UTF-16 source positions; comment filtering, CR/LF normalization, #line handling, and raw-column tracking; portable validated ASCII/UTF-8/UTF-16 decoding; complete .NET Unicode category matching; bounded deterministic maximum-munch DFA execution; reserved-word remapping; structured diagnostics and recovery entry points; and idiomatic Rust iteration. It also adds API migration/position contracts, generated-shim coverage, a dedicated audit, and ubuntu-latest CI integration. Generator-owned NFA construction and table serialization are explicitly rejected rather than silently fabricated and will be implemented by issue #83.
issue #81 audit and deterministic shim regeneration check passed
full workspace/all-target cargo check passed
git diff whitespace audit passed
Implemented and pushed in commit 73cef10.
This replaces the LSL lexer boundary with native Rust: mapped token/symbol/EOF and compatibility enumerator APIs; UTF-16 source positions; comment filtering, CR/LF normalization, #line handling, and raw-column tracking; portable validated ASCII/UTF-8/UTF-16 decoding; complete .NET Unicode category matching; bounded deterministic maximum-munch DFA execution; reserved-word remapping; structured diagnostics and recovery entry points; and idiomatic Rust iteration. It also adds API migration/position contracts, generated-shim coverage, a dedicated audit, and ubuntu-latest CI integration. Generator-owned NFA construction and table serialization are explicitly rejected rather than silently fabricated and will be implemented by issue #83.
Verification on the committed tree:
- 27 focused lexer/token/source/diagnostic fixtures passed
- strict clippy (-D warnings) passed
- rustdoc with -D warnings passed
- standalone mapped API consumer passed
- issue #81 audit and deterministic shim regeneration check passed
- full workspace/all-target cargo check passed
- git diff whitespace audit passed
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Objective
Implement public token/symbol types, character sets, comments, source line management, lexer behavior, and parser diagnostics in native Rust.
Deliverables
Validation
Prerequisite
All global API/test gates and the foundational/world implementations needed by these extension crates are complete.
Project constraints
Implemented and pushed in commit
73cef10.This replaces the LSL lexer boundary with native Rust: mapped token/symbol/EOF and compatibility enumerator APIs; UTF-16 source positions; comment filtering, CR/LF normalization, #line handling, and raw-column tracking; portable validated ASCII/UTF-8/UTF-16 decoding; complete .NET Unicode category matching; bounded deterministic maximum-munch DFA execution; reserved-word remapping; structured diagnostics and recovery entry points; and idiomatic Rust iteration. It also adds API migration/position contracts, generated-shim coverage, a dedicated audit, and ubuntu-latest CI integration. Generator-owned NFA construction and table serialization are explicitly rejected rather than silently fabricated and will be implemented by issue #83.
Verification on the committed tree: