LSL tools: implement token model, lexer, source locations, and diagnostics #81

New Issue

hugo · 2026-08-08T08:23:18Z

hugo commented

2026-08-08 08:23:18 +00:00

Objective

Implement public token/symbol types, character sets, comments, source line management, lexer behavior, and parser diagnostics in native Rust.

Deliverables

Preserve tokenization, Unicode/category behavior, comments, keywords, literals, positions, EOF, error text/categories, and recovery entry points.
Replace C# enumerators with Rust iterators while retaining mapped public concepts.
Use bounded inputs and deterministic output.

Validation

All lexer/token/diagnostic fixtures pass.
Invalid input reports matching locations/categories.
No generated YY class behavior is silently omitted.

Prerequisite

All global API/test gates and the foundational/world implementations needed by these extension crates are complete.

Project constraints

Full native Rust implementation only; no wrapped C# parser, viewer, renderer, or service.
Preserve mapped APIs and exact translated tests; reuse lower layers and keep generated output deterministic.
Validate bounded input, callbacks/tasks, docs, coverage, and workspace gates.

## Objective Implement public token/symbol types, character sets, comments, source line management, lexer behavior, and parser diagnostics in native Rust. ## Deliverables - Preserve tokenization, Unicode/category behavior, comments, keywords, literals, positions, EOF, error text/categories, and recovery entry points. - Replace C# enumerators with Rust iterators while retaining mapped public concepts. - Use bounded inputs and deterministic output. ## Validation - All lexer/token/diagnostic fixtures pass. - Invalid input reports matching locations/categories. - No generated YY class behavior is silently omitted. ## Prerequisite All global API/test gates and the foundational/world implementations needed by these extension crates are complete. ## Project constraints - Full native Rust implementation only; no wrapped C# parser, viewer, renderer, or service. - Preserve mapped APIs and exact translated tests; reuse lower layers and keep generated output deterministic. - Validate bounded input, callbacks/tasks, docs, coverage, and workspace gates.

hugo added this to the 10 - Rendering, RLV, and LSL tools milestone 2026-08-08 08:23:18 +00:00

hugo added the enhancement label 2026-08-08 08:23:18 +00:00

hugo referenced this issue from a commit

2026-08-11 01:55:23 +00:00

Implement native LSL lexer runtime (#81)

hugo commented

2026-08-11 01:55:51 +00:00

Implemented and pushed in commit 73cef10.

This replaces the LSL lexer boundary with native Rust: mapped token/symbol/EOF and compatibility enumerator APIs; UTF-16 source positions; comment filtering, CR/LF normalization, #line handling, and raw-column tracking; portable validated ASCII/UTF-8/UTF-16 decoding; complete .NET Unicode category matching; bounded deterministic maximum-munch DFA execution; reserved-word remapping; structured diagnostics and recovery entry points; and idiomatic Rust iteration. It also adds API migration/position contracts, generated-shim coverage, a dedicated audit, and ubuntu-latest CI integration. Generator-owned NFA construction and table serialization are explicitly rejected rather than silently fabricated and will be implemented by issue #83.

Verification on the committed tree:

27 focused lexer/token/source/diagnostic fixtures passed
strict clippy (-D warnings) passed
rustdoc with -D warnings passed
standalone mapped API consumer passed
issue #81 audit and deterministic shim regeneration check passed
full workspace/all-target cargo check passed
git diff whitespace audit passed

Implemented and pushed in commit 73cef10. This replaces the LSL lexer boundary with native Rust: mapped token/symbol/EOF and compatibility enumerator APIs; UTF-16 source positions; comment filtering, CR/LF normalization, #line handling, and raw-column tracking; portable validated ASCII/UTF-8/UTF-16 decoding; complete .NET Unicode category matching; bounded deterministic maximum-munch DFA execution; reserved-word remapping; structured diagnostics and recovery entry points; and idiomatic Rust iteration. It also adds API migration/position contracts, generated-shim coverage, a dedicated audit, and ubuntu-latest CI integration. Generator-owned NFA construction and table serialization are explicitly rejected rather than silently fabricated and will be implemented by issue #83. Verification on the committed tree: - 27 focused lexer/token/source/diagnostic fixtures passed - strict clippy (-D warnings) passed - rustdoc with -D warnings passed - standalone mapped API consumer passed - issue #81 audit and deterministic shim regeneration check passed - full workspace/all-target cargo check passed - git diff whitespace audit passed

hugo closed this issue

2026-08-11 01:55:58 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/MetaCrate#81