Implement safe public LSL delivery (#129)
This commit is contained in:
16
crates/metacrate-lsl-tools/Cargo.toml
Normal file
16
crates/metacrate-lsl-tools/Cargo.toml
Normal file
@@ -0,0 +1,16 @@
|
||||
[package]
|
||||
name = "metacrate-lsl-tools"
|
||||
version.workspace = true
|
||||
edition.workspace = true
|
||||
rust-version.workspace = true
|
||||
license.workspace = true
|
||||
repository.workspace = true
|
||||
description = "Native LSL lexer and parser tooling for the MetaCrate LibreMetaverse rewrite"
|
||||
|
||||
[dependencies]
|
||||
libremetaverse-types = { version = "0.0.1", path = "../libremetaverse-types" }
|
||||
unicode-general-category = "1.1"
|
||||
regex = "1"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
161
crates/metacrate-lsl-tools/README.md
Normal file
161
crates/metacrate-lsl-tools/README.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Native LSL lexer and parser tools
|
||||
|
||||
`metacrate-lsl-tools` is the native Rust replacement for the lexer and
|
||||
parser-generator runtime in the pinned `LibreMetaverse.LslTools` assembly. The
|
||||
issue 81 boundary supplies source reading, comments, Unicode character sets,
|
||||
deterministic DFA execution, reserved words, terminal symbols, EOF, source
|
||||
locations, diagnostics, and Rust iterators. The issue 82 boundary supplies
|
||||
grammar productions, parser tables, precedence, reductions, and recovery.
|
||||
The issue 83 boundary adds deterministic checked-in grammar/token tables and
|
||||
native replacements for the retired parser-generator object model.
|
||||
|
||||
## Source and position contract
|
||||
|
||||
All public positions are UTF-16 code-unit offsets, matching the original
|
||||
`System.Char` API. They are not UTF-8 byte offsets. `SourceLineInfo` reports a
|
||||
one-based line and character position while retaining the filtered-line bounds
|
||||
and the raw character position before removed comments. Non-BMP characters
|
||||
therefore occupy two positions, exactly as they do in C#.
|
||||
|
||||
`CsReader` accepts strings, files, or explicitly encoded byte slices. It
|
||||
normalizes CRLF and lone CR to LF, removes `//` and `/* ... */` comments while
|
||||
preserving their newline structure, tracks the removed UTF-16 lengths for raw
|
||||
columns, and applies `#line N "file"` directives. Unterminated block comments,
|
||||
invalid UTF-8/UTF-16, unpaired surrogates, trailing UTF-16 bytes, and non-ASCII
|
||||
bytes in ASCII mode return positioned errors instead of replacement text.
|
||||
|
||||
The portable encoding set is UTF-8, UTF-16LE, UTF-16BE, ASCII, and ASCIICAPS.
|
||||
UTF-7 and platform code pages are deliberately not delegated to host APIs, so
|
||||
Linux and Windows produce the same result. A source is bounded to 64 Mi UTF-16
|
||||
units and a token to 16 Mi units before unbounded allocation or matching can
|
||||
occur.
|
||||
|
||||
## Lexer table contract
|
||||
|
||||
`Dfa` is a validated deterministic table of `DfaState` values and
|
||||
`CharacterMatcher` transitions. Matchers support exact UTF-16 values, ranges,
|
||||
sets, all .NET Unicode general categories, category unions, any character, and
|
||||
EOF. The runtime applies maximum munch, preserves rule order for overlapping
|
||||
transitions, rejects empty non-EOF matches, and exposes a deterministic textual
|
||||
table representation through `YyLexer::emit_dfa`.
|
||||
|
||||
Accepting states carry a `TokenDefinition`, action number, optional reserved
|
||||
word table, and one of four actions: emit, skip, emit and change start
|
||||
condition, or skip and change start condition. Reserved words are exact by
|
||||
default and use Unicode uppercase only when the table requests the reference
|
||||
`U { ... }` behavior. `TOKEN` preserves token name, number, lexeme, semantic
|
||||
value, half-open UTF-16 span, and source location. Configured EOF is emitted
|
||||
once at the end of the filtered buffer.
|
||||
|
||||
The old C# enumerators remain callable as compatibility adapters. New Rust code
|
||||
should use `Lexer::iter` or `Lexer::next_token`; both stop deterministically
|
||||
after a diagnostic and never yield a fabricated token. Token and symbol
|
||||
`Pass` members perform real lookup against the mapped parser symbol/literal
|
||||
tables and populate the supplied parser-entry priority. Missing or malformed
|
||||
table data is an explicit positioned error.
|
||||
|
||||
## Grammar and parser contract
|
||||
|
||||
`Grammar` builds validated canonical LR(0) item sets with deterministic LALR(1)
|
||||
lookahead propagation. Symbols and productions retain stable numeric
|
||||
identities. Table construction rejects missing start/EOF declarations,
|
||||
undeclared right-hand-side symbols, invalid precedence declarations, oversized
|
||||
state sets, incomplete goto tables, and reduction underflow. Empty productions
|
||||
and recursive nonterminals are supported.
|
||||
|
||||
Shift/reduce conflicts use yacc-compatible rules: a declared higher precedence
|
||||
wins, equal left precedence reduces, equal right precedence shifts, and equal
|
||||
nonassociative precedence installs a rejecting table entry. Undeclared
|
||||
shift/reduce conflicts shift; reduce/reduce conflicts select the lower
|
||||
production number. `YyParser::conflicts` preserves every decision and `emit`
|
||||
writes a byte-stable table description independent of hash iteration.
|
||||
|
||||
`Parser` consumes the native lexer and returns a typed `ParseTree` through the
|
||||
mapped `SYMBOL` semantic value. Syntax recovery uses the conventional terminal
|
||||
number zero: it pops to a state that can shift `error`, shifts an explicit error
|
||||
node, and discards input to the next valid lookahead. Diagnostics distinguish
|
||||
the original syntax error from successful recovery. Parsing is bounded to
|
||||
65,536 states, 1,048,576 live stack entries, 16,777,216 operations, and 1,000
|
||||
recovery attempts.
|
||||
|
||||
Compatibility `SymbolSet` clones share deterministic FIRST/FOLLOW membership,
|
||||
and cloned `CSymbol` values share registered production metadata. This retains
|
||||
the observable reference behavior of the C# grammar model without unsafe code.
|
||||
Legacy parser-runtime serialization members that cannot faithfully round-trip
|
||||
the richer native LALR machine remain explicit `InvalidOperation` results.
|
||||
Callers should use the stable `YyParser::emit` and `YyLexer::emit_dfa` formats,
|
||||
or regenerate the canonical checked-in table module described below.
|
||||
|
||||
## Checked-in generator contract
|
||||
|
||||
[`codegen/inputs/lsl_tools_grammar.json`](../../codegen/inputs/lsl_tools_grammar.json)
|
||||
is the reviewed source of truth reconstructed from the pinned 4.5
|
||||
`yycs0syntax.cs` and `yycs0tokens.cs` tables at upstream commit
|
||||
`2aa70bb68513b39795da5d13c88f31b86e85a3ba`. It records every token,
|
||||
nonterminal, production, and semantic-action class. The portable Python
|
||||
generator uses the standard library plus the workspace Rust formatter and emits
|
||||
`src/generated_tables.rs`; it does not invoke C#, a C# source generator, or a
|
||||
.NET runtime.
|
||||
|
||||
Regenerate with `python3 tools/generate_lsl_tables.py`. Use
|
||||
`python3 tools/generate_lsl_tables.py --check` in reviews and CI. The check
|
||||
renders the complete output in memory and compares bytes, so repeated runs are
|
||||
independent of hash iteration, locale, host operating system, and timestamps.
|
||||
The checked-in module supplies the canonical 25-production grammar, a Unicode
|
||||
lexer with reserved `base`, `this`, and `new` tokens, the 84 generated syntax
|
||||
classes, the 13 generated token classes, and the four `cs0` runtime wrappers.
|
||||
|
||||
The old `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, NFA, delegate, factory,
|
||||
and 4.5 `Serialiser` APIs are live Rust compatibility adapters. New code should
|
||||
prefer `Grammar`, typed `DfaState` builders, `generated_parser`, and
|
||||
`generated_lexer`; these avoid CLR-style method pointers and class-shaped
|
||||
generated inheritance while retaining deterministic mapped behavior.
|
||||
|
||||
## Diagnostics and migration
|
||||
|
||||
`ErrorHandler` collects structured `Diagnostic` values. Each diagnostic has a
|
||||
stable numeric code, `DiagnosticCategory`, severity, message, input fragment,
|
||||
and `SourceLineInfo`. Invalid characters and start conditions are recorded at
|
||||
the offending UTF-16 position. Compatibility exception constructors retain
|
||||
their original number, input, symbol/token location, handled state, and fatal
|
||||
or stop behavior. A handler configured to throw increments its counter and
|
||||
returns immediately without reporting, matching the reference order.
|
||||
|
||||
The principal API mapping is:
|
||||
|
||||
| C# concept | Native Rust API |
|
||||
| --- | --- |
|
||||
| `CsReader`, `LineManager`, `SourceLineInfo` | Same mapped names, UTF-16-safe source model |
|
||||
| `Dfa`, `Dfa.Action`, `YyLexer` | `Dfa`, `DfaAction`, `YyLexer`, plus typed state builders |
|
||||
| `SYMBOL`, `TOKEN`, `EOF`, `Null` | Same mapped names with owned Rust values |
|
||||
| `Lexer._Enumerator` | `LexerEnumerator`; prefer `LexerIterator` |
|
||||
| `Charset`, `CatTest` | Same mapped names plus `DotNetUnicodeCategory` |
|
||||
| `CSToolsException`, `ErrorHandler` | Same mapped names plus structured `Diagnostic` |
|
||||
| `CSymbol`, `SymbolSet`, `Production`, `Precedence` | Same mapped names plus typed `Grammar` builders |
|
||||
| `YyParser`, parser entries | Deterministic native tables and conflict records |
|
||||
| `Parser`, `ParseStackEntry`, `Error`, `recoveredError` | Bounded parsing, typed trees, and recovery |
|
||||
| `yycs0syntax`, `yycs0tokens` | `generated_parser`, `generated_lexer`, checked-in native types |
|
||||
| `GenBase`, `SymbolsGen`, `TokensGen`, `Regex`, `Nfa` | Portable Rust generator adapters; prefer typed grammar/DFA builders |
|
||||
| `Serialiser` | Deterministic 4.5-compatible integer stream for supported mapped values |
|
||||
|
||||
No C#, .NET runtime, dynamically loaded class, macOS-only API, platform code
|
||||
page, or runtime source generation is used by this boundary.
|
||||
|
||||
## Reproducible verification
|
||||
|
||||
Run the issue-owned gates with one build job:
|
||||
|
||||
```sh
|
||||
CARGO_BUILD_JOBS=1 cargo test -p metacrate-lsl-tools --locked
|
||||
CARGO_BUILD_JOBS=1 cargo check --manifest-path tests/api-compile/Cargo.toml --locked
|
||||
CARGO_BUILD_JOBS=1 cargo clippy -p metacrate-lsl-tools --all-targets --locked -- -D warnings
|
||||
RUSTDOCFLAGS='-D warnings' CARGO_BUILD_JOBS=1 cargo doc -p metacrate-lsl-tools --no-deps --locked
|
||||
python3 tools/check_milestone_10_issue_81.py
|
||||
python3 tools/check_milestone_10_issue_82.py
|
||||
python3 tools/check_milestone_10_issue_83.py
|
||||
python3 tools/generate_lsl_tables.py --check
|
||||
python3 tools/generate_api_shims.py --check
|
||||
```
|
||||
|
||||
The package contains 46 focused native and compatibility fixtures. The Gitea
|
||||
workflow runs the audit and workspace compile on `ubuntu-latest` only.
|
||||
4830
crates/metacrate-lsl-tools/src/generated.rs
Normal file
4830
crates/metacrate-lsl-tools/src/generated.rs
Normal file
File diff suppressed because it is too large
Load Diff
857
crates/metacrate-lsl-tools/src/generated_tables.rs
Normal file
857
crates/metacrate-lsl-tools/src/generated_tables.rs
Normal file
@@ -0,0 +1,857 @@
|
||||
// @generated by tools/generate_lsl_tables.py; do not edit by hand.
|
||||
#![allow(non_camel_case_types)]
|
||||
#![allow(non_snake_case)]
|
||||
#![allow(clippy::missing_errors_doc)]
|
||||
#![allow(clippy::must_use_candidate)]
|
||||
|
||||
use std::ops::{Deref, DerefMut};
|
||||
|
||||
use libremetaverse_types::compat::{Object, UnicodeCategory};
|
||||
|
||||
use crate::{
|
||||
CharacterMatcher, Dfa, DfaAccept, DfaState, Error, ErrorHandler, Grammar, Lexer, LexerAction,
|
||||
Parser, ResWds, SYMBOL, TOKEN, TokenDefinition, UnicodeClass, YyLexer, YyParser,
|
||||
};
|
||||
|
||||
pub const LSL_GENERATOR_SOURCE_COMMIT: &str = "2aa70bb68513b39795da5d13c88f31b86e85a3ba";
|
||||
pub const LSL_GENERATOR_SOURCE_PARSER: &str = "LibreMetaverse.LslTools/YYClass/yycs0syntax.cs";
|
||||
pub const LSL_GENERATOR_SOURCE_LEXER: &str = "LibreMetaverse.LslTools/YYClass/yycs0tokens.cs";
|
||||
pub const LSL_GENERATOR_SOURCE_LICENSE: &str = "BSD-3-Clause";
|
||||
pub const LSL_GENERATOR_SERIALIZATION_VERSION: &str = "4.5";
|
||||
pub const GENERATED_PARSER_DATA: &[i32] = &[
|
||||
1, 19, 2, 22, 2, 1, 3, 3, 1, 4, 4, 1, 4, 5, 1, 3, 6, 1, 2, 7, 1, 3, 8, 1, 5, 9, 1, 9, 10, 1, 6,
|
||||
11, 1, 6, 12, 1, 6, 13, 1, 6, 14, 1, 6, 15, 1, 6, 16, 0, 6, 17, 0, 5, 18, 0, 4, 19, 0, 9, 20,
|
||||
0, 4, 21, 0, 4, 22, 0, 8, 23, 0, 4, 25, 1, 19, 3, 16, 20, 16, 2, 16, 0, 3, 16, 2, 16, 18, 4,
|
||||
16, 2, 16, 21, 5, 17, 0, 6, 17, 2, 17, 18, 7, 20, 5, 23, 12, 17, 13, 22, 8, 21, 4, 23, 12, 17,
|
||||
13, 9, 22, 0, 10, 22, 5, 8, 3, 12, 17, 13, 11, 22, 5, 8, 4, 12, 17, 13, 12, 23, 1, 6, 13, 23,
|
||||
4, 6, 14, 17, 15, 14, 18, 1, 6, 15, 18, 1, 7, 16, 18, 1, 9, 17, 18, 1, 3, 18, 18, 1, 4, 19, 18,
|
||||
4, 4, 14, 17, 15, 20, 18, 1, 8, 21, 18, 2, 5, 23, 22, 18, 2, 5, 6, 23, 18, 3, 12, 17, 13, 24,
|
||||
18, 3, 10, 16, 11, 25, 18, 3, 14, 17, 15,
|
||||
];
|
||||
|
||||
pub fn generated_symbol_name(number: i32) -> Option<&'static str> {
|
||||
match number {
|
||||
2 => Some("EOF"),
|
||||
3 => Some("BASE"),
|
||||
4 => Some("THIS"),
|
||||
5 => Some("NEW"),
|
||||
6 => Some("ID"),
|
||||
7 => Some("ANY"),
|
||||
8 => Some("COLON"),
|
||||
9 => Some("SEMICOLON"),
|
||||
10 => Some("LBRACE"),
|
||||
11 => Some("RBRACE"),
|
||||
12 => Some("LPAREN"),
|
||||
13 => Some("RPAREN"),
|
||||
14 => Some("LBRACK"),
|
||||
15 => Some("RBRACK"),
|
||||
16 => Some("GStuff"),
|
||||
17 => Some("Stuff"),
|
||||
18 => Some("Item"),
|
||||
19 => Some("ClassBody"),
|
||||
20 => Some("Cons"),
|
||||
21 => Some("Call"),
|
||||
22 => Some("BaseCall"),
|
||||
23 => Some("Name"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn generated_parser() -> Result<YyParser, Error> {
|
||||
let mut grammar = Grammar::new(19, 2)?;
|
||||
grammar.add_symbol("EOF", 2, true)?;
|
||||
grammar.add_symbol("BASE", 3, true)?;
|
||||
grammar.add_symbol("THIS", 4, true)?;
|
||||
grammar.add_symbol("NEW", 5, true)?;
|
||||
grammar.add_symbol("ID", 6, true)?;
|
||||
grammar.add_symbol("ANY", 7, true)?;
|
||||
grammar.add_symbol("COLON", 8, true)?;
|
||||
grammar.add_symbol("SEMICOLON", 9, true)?;
|
||||
grammar.add_symbol("LBRACE", 10, true)?;
|
||||
grammar.add_symbol("RBRACE", 11, true)?;
|
||||
grammar.add_symbol("LPAREN", 12, true)?;
|
||||
grammar.add_symbol("RPAREN", 13, true)?;
|
||||
grammar.add_symbol("LBRACK", 14, true)?;
|
||||
grammar.add_symbol("RBRACK", 15, true)?;
|
||||
grammar.add_symbol("GStuff", 16, false)?;
|
||||
grammar.add_symbol("Stuff", 17, false)?;
|
||||
grammar.add_symbol("Item", 18, false)?;
|
||||
grammar.add_symbol("ClassBody", 19, false)?;
|
||||
grammar.add_symbol("Cons", 20, false)?;
|
||||
grammar.add_symbol("Call", 21, false)?;
|
||||
grammar.add_symbol("BaseCall", 22, false)?;
|
||||
grammar.add_symbol("Name", 23, false)?;
|
||||
grammar.add_production(19, vec![16, 20, 16])?;
|
||||
grammar.add_production(16, vec![])?;
|
||||
grammar.add_production(16, vec![16, 18])?;
|
||||
grammar.add_production(16, vec![16, 21])?;
|
||||
grammar.add_production(17, vec![])?;
|
||||
grammar.add_production(17, vec![17, 18])?;
|
||||
grammar.add_production(20, vec![23, 12, 17, 13, 22])?;
|
||||
grammar.add_production(21, vec![23, 12, 17, 13])?;
|
||||
grammar.add_production(22, vec![])?;
|
||||
grammar.add_production(22, vec![8, 3, 12, 17, 13])?;
|
||||
grammar.add_production(22, vec![8, 4, 12, 17, 13])?;
|
||||
grammar.add_production(23, vec![6])?;
|
||||
grammar.add_production(23, vec![6, 14, 17, 15])?;
|
||||
grammar.add_production(18, vec![6])?;
|
||||
grammar.add_production(18, vec![7])?;
|
||||
grammar.add_production(18, vec![9])?;
|
||||
grammar.add_production(18, vec![3])?;
|
||||
grammar.add_production(18, vec![4])?;
|
||||
grammar.add_production(18, vec![4, 14, 17, 15])?;
|
||||
grammar.add_production(18, vec![8])?;
|
||||
grammar.add_production(18, vec![5, 23])?;
|
||||
grammar.add_production(18, vec![5, 6])?;
|
||||
grammar.add_production(18, vec![12, 17, 13])?;
|
||||
grammar.add_production(18, vec![10, 16, 11])?;
|
||||
grammar.add_production(18, vec![14, 17, 15])?;
|
||||
let mut parser = grammar.build()?;
|
||||
parser.arr = GENERATED_PARSER_DATA.to_vec();
|
||||
Ok(parser)
|
||||
}
|
||||
|
||||
fn accept(name: &str, number: i32, action: LexerAction, action_number: i32) -> DfaAccept {
|
||||
DfaAccept {
|
||||
token: TokenDefinition {
|
||||
name: name.to_owned(),
|
||||
number,
|
||||
},
|
||||
action,
|
||||
action_number,
|
||||
reserved_words: (name == "ID").then(|| "ID".to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn generated_lexer_with_handler(error_handler: ErrorHandler) -> Result<YyLexer, Error> {
|
||||
let punctuation = [
|
||||
(58, "COLON", 8),
|
||||
(59, "SEMICOLON", 9),
|
||||
(123, "LBRACE", 10),
|
||||
(125, "RBRACE", 11),
|
||||
(40, "LPAREN", 12),
|
||||
(41, "RPAREN", 13),
|
||||
(91, "LBRACK", 14),
|
||||
(93, "RBRACK", 15),
|
||||
];
|
||||
let punctuation_start = 3usize;
|
||||
let any_state = punctuation_start + punctuation.len();
|
||||
let mut states = vec![DfaState::default(); any_state + 1];
|
||||
states[0] = states[0]
|
||||
.clone()
|
||||
.transition(CharacterMatcher::UnicodeClass(UnicodeClass::WhiteSpace), 1)
|
||||
.transition(CharacterMatcher::UnicodeClass(UnicodeClass::Letter), 2)
|
||||
.transition(CharacterMatcher::Exact(u16::from(b'_')), 2);
|
||||
for (offset, (unit, _, _)) in punctuation.iter().enumerate() {
|
||||
states[0]
|
||||
.transitions
|
||||
.push((CharacterMatcher::Exact(*unit), punctuation_start + offset));
|
||||
}
|
||||
states[0]
|
||||
.transitions
|
||||
.push((CharacterMatcher::Any, any_state));
|
||||
states[1] = states[1]
|
||||
.clone()
|
||||
.transition(CharacterMatcher::UnicodeClass(UnicodeClass::WhiteSpace), 1)
|
||||
.accepting(accept("ANY", 7, LexerAction::Skip, -1));
|
||||
states[2] = states[2]
|
||||
.clone()
|
||||
.transition(CharacterMatcher::UnicodeClass(UnicodeClass::Letter), 2)
|
||||
.transition(CharacterMatcher::UnicodeClass(UnicodeClass::Number), 2)
|
||||
.transition(CharacterMatcher::Exact(u16::from(b'_')), 2)
|
||||
.accepting(accept("ID", 6, LexerAction::Emit, 0));
|
||||
for (offset, (_, name, number)) in punctuation.iter().enumerate() {
|
||||
states[punctuation_start + offset] =
|
||||
DfaState::default().accepting(accept(name, *number, LexerAction::Emit, *number));
|
||||
}
|
||||
states[any_state] = DfaState::default().accepting(accept("ANY", 7, LexerAction::Emit, 1));
|
||||
let mut lexer = YyLexer::new(error_handler)?;
|
||||
lexer.set_start_dfa("YYINITIAL", Dfa::from_states(states, 0)?)?;
|
||||
lexer.set_reserved_words(
|
||||
"ID",
|
||||
ResWds::from_pairs(
|
||||
[
|
||||
("base", TokenDefinition::new("BASE", 3)?),
|
||||
("this", TokenDefinition::new("THIS", 4)?),
|
||||
("new", TokenDefinition::new("NEW", 5)?),
|
||||
],
|
||||
false,
|
||||
)?,
|
||||
)?;
|
||||
lexer.using_eof = true;
|
||||
for (name, number) in [
|
||||
("EOF", 2),
|
||||
("BASE", 3),
|
||||
("THIS", 4),
|
||||
("NEW", 5),
|
||||
("ID", 6),
|
||||
("ANY", 7),
|
||||
("COLON", 8),
|
||||
("SEMICOLON", 9),
|
||||
("LBRACE", 10),
|
||||
("RBRACE", 11),
|
||||
("LPAREN", 12),
|
||||
("RPAREN", 13),
|
||||
("LBRACK", 14),
|
||||
("RBRACK", 15),
|
||||
] {
|
||||
lexer
|
||||
.tokens
|
||||
.0
|
||||
.insert(Object::String(name.to_owned()), Object::Integer(number));
|
||||
}
|
||||
let _ = lexer.using_cat(UnicodeCategory(0))?;
|
||||
Ok(lexer)
|
||||
}
|
||||
|
||||
pub fn generated_lexer() -> Result<YyLexer, Error> {
|
||||
generated_lexer_with_handler(ErrorHandler::new_with_boolean(false)?)
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct GeneratedLexerToken<const NUMBER: i32>(TOKEN);
|
||||
|
||||
impl<const NUMBER: i32> GeneratedLexerToken<NUMBER> {
|
||||
pub fn new(lexer: Lexer) -> Result<Self, Error> {
|
||||
let name = generated_symbol_name(NUMBER).ok_or(Error::Argument)?;
|
||||
Ok(Self(TOKEN::generated_with_lexer(lexer, name, NUMBER)?))
|
||||
}
|
||||
pub fn yyname(&self) -> String {
|
||||
self.0.yyname()
|
||||
}
|
||||
pub const fn yynum(&self) -> i32 {
|
||||
NUMBER
|
||||
}
|
||||
pub fn yytext(&self) -> String {
|
||||
self.0.yytext()
|
||||
}
|
||||
}
|
||||
|
||||
impl<const NUMBER: i32> Deref for GeneratedLexerToken<NUMBER> {
|
||||
type Target = TOKEN;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl<const NUMBER: i32> DerefMut for GeneratedLexerToken<NUMBER> {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.0
|
||||
}
|
||||
}
|
||||
pub type BASE = GeneratedLexerToken<3>;
|
||||
pub type THIS = GeneratedLexerToken<4>;
|
||||
pub type NEW = GeneratedLexerToken<5>;
|
||||
pub type ID = GeneratedLexerToken<6>;
|
||||
pub type ANY = GeneratedLexerToken<7>;
|
||||
pub type COLON = GeneratedLexerToken<8>;
|
||||
pub type SEMICOLON = GeneratedLexerToken<9>;
|
||||
pub type LBRACE = GeneratedLexerToken<10>;
|
||||
pub type RBRACE = GeneratedLexerToken<11>;
|
||||
pub type LPAREN = GeneratedLexerToken<12>;
|
||||
pub type RPAREN = GeneratedLexerToken<13>;
|
||||
pub type LBRACK = GeneratedLexerToken<14>;
|
||||
pub type RBRACK = GeneratedLexerToken<15>;
|
||||
|
||||
fn generated_parser_identity(generated: i32) -> Option<(&'static str, i32)> {
|
||||
match generated {
|
||||
27..=35 => Some(("GStuff", 16)),
|
||||
36..=41 => Some(("Stuff", 17)),
|
||||
63..=98 => Some(("Item", 18)),
|
||||
24..=26 => Some(("ClassBody", 19)),
|
||||
42..=44 => Some(("Cons", 20)),
|
||||
45..=47 => Some(("Call", 21)),
|
||||
48..=56 => Some(("BaseCall", 22)),
|
||||
57..=62 => Some(("Name", 23)),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn stack_text(parser: &Parser, index: i32) -> Result<String, Error> {
|
||||
Ok(parser.stack_at(index)?.m_value.raw_text())
|
||||
}
|
||||
|
||||
fn generated_semantic_text(generated: i32, parser: &Parser) -> Result<String, Error> {
|
||||
Ok(match generated {
|
||||
26 => stack_text(parser, 1)?,
|
||||
32 | 35 | 41 => format!("{}{}", stack_text(parser, 1)?, stack_text(parser, 0)?),
|
||||
44 => format!(
|
||||
"{}({}){}",
|
||||
stack_text(parser, 4)?.trim(),
|
||||
stack_text(parser, 2)?,
|
||||
stack_text(parser, 0)?
|
||||
),
|
||||
47 => format!(
|
||||
"{}({})",
|
||||
stack_text(parser, 3)?.trim(),
|
||||
stack_text(parser, 1)?
|
||||
),
|
||||
53 => format!("base{}", stack_text(parser, 1)?),
|
||||
56 => format!("this{}", stack_text(parser, 1)?),
|
||||
59 => format!(" {} ", stack_text(parser, 0)?),
|
||||
62 => format!("{}[{}]", stack_text(parser, 3)?, stack_text(parser, 1)?),
|
||||
65 | 68 => stack_text(parser, 0)?,
|
||||
71 => ";\n".to_owned(),
|
||||
74 => " base ".to_owned(),
|
||||
77 => " this ".to_owned(),
|
||||
80 => format!(" this[{}]", stack_text(parser, 1)?),
|
||||
83 => ":".to_owned(),
|
||||
86 | 89 => format!(" new {}", stack_text(parser, 0)?),
|
||||
92 => format!("({})", stack_text(parser, 1)?),
|
||||
95 => format!("{{{}}}\n", stack_text(parser, 1)?),
|
||||
98 => format!("[{}]", stack_text(parser, 1)?),
|
||||
_ => String::new(),
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct GeneratedParserSymbol<const GENERATED: i32>(TOKEN);
|
||||
|
||||
impl<const GENERATED: i32> GeneratedParserSymbol<GENERATED> {
|
||||
pub fn new(parser: Parser) -> Result<Self, Error> {
|
||||
let (name, number) = generated_parser_identity(GENERATED).ok_or(Error::Argument)?;
|
||||
let text = generated_semantic_text(GENERATED, &parser)?;
|
||||
Ok(Self(TOKEN::generated_with_parser(
|
||||
parser, name, number, text,
|
||||
)?))
|
||||
}
|
||||
pub fn yyname(&self) -> String {
|
||||
self.0.yyname()
|
||||
}
|
||||
pub fn yynum(&self) -> i32 {
|
||||
self.0.yynum()
|
||||
}
|
||||
pub fn yytext(&self) -> String {
|
||||
self.0.yytext()
|
||||
}
|
||||
}
|
||||
|
||||
impl<const GENERATED: i32> Deref for GeneratedParserSymbol<GENERATED> {
|
||||
type Target = TOKEN;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl<const GENERATED: i32> DerefMut for GeneratedParserSymbol<GENERATED> {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.0
|
||||
}
|
||||
}
|
||||
pub type ClassBody_1 = GeneratedParserSymbol<24>;
|
||||
pub type ClassBody_2 = GeneratedParserSymbol<25>;
|
||||
pub type ClassBody_2_1 = GeneratedParserSymbol<26>;
|
||||
pub type GStuff_1 = GeneratedParserSymbol<27>;
|
||||
pub type GStuff_2 = GeneratedParserSymbol<28>;
|
||||
pub type GStuff_2_1 = GeneratedParserSymbol<29>;
|
||||
pub type GStuff_3 = GeneratedParserSymbol<30>;
|
||||
pub type GStuff_4 = GeneratedParserSymbol<31>;
|
||||
pub type GStuff_4_1 = GeneratedParserSymbol<32>;
|
||||
pub type GStuff_5 = GeneratedParserSymbol<33>;
|
||||
pub type GStuff_6 = GeneratedParserSymbol<34>;
|
||||
pub type GStuff_6_1 = GeneratedParserSymbol<35>;
|
||||
pub type Stuff_1 = GeneratedParserSymbol<36>;
|
||||
pub type Stuff_2 = GeneratedParserSymbol<37>;
|
||||
pub type Stuff_2_1 = GeneratedParserSymbol<38>;
|
||||
pub type Stuff_3 = GeneratedParserSymbol<39>;
|
||||
pub type Stuff_4 = GeneratedParserSymbol<40>;
|
||||
pub type Stuff_4_1 = GeneratedParserSymbol<41>;
|
||||
pub type Cons_1 = GeneratedParserSymbol<42>;
|
||||
pub type Cons_2 = GeneratedParserSymbol<43>;
|
||||
pub type Cons_2_1 = GeneratedParserSymbol<44>;
|
||||
pub type Call_1 = GeneratedParserSymbol<45>;
|
||||
pub type Call_2 = GeneratedParserSymbol<46>;
|
||||
pub type Call_2_1 = GeneratedParserSymbol<47>;
|
||||
pub type BaseCall_1 = GeneratedParserSymbol<48>;
|
||||
pub type BaseCall_2 = GeneratedParserSymbol<49>;
|
||||
pub type BaseCall_2_1 = GeneratedParserSymbol<50>;
|
||||
pub type BaseCall_3 = GeneratedParserSymbol<51>;
|
||||
pub type BaseCall_4 = GeneratedParserSymbol<52>;
|
||||
pub type BaseCall_4_1 = GeneratedParserSymbol<53>;
|
||||
pub type BaseCall_5 = GeneratedParserSymbol<54>;
|
||||
pub type BaseCall_6 = GeneratedParserSymbol<55>;
|
||||
pub type BaseCall_6_1 = GeneratedParserSymbol<56>;
|
||||
pub type Name_1 = GeneratedParserSymbol<57>;
|
||||
pub type Name_2 = GeneratedParserSymbol<58>;
|
||||
pub type Name_2_1 = GeneratedParserSymbol<59>;
|
||||
pub type Name_3 = GeneratedParserSymbol<60>;
|
||||
pub type Name_4 = GeneratedParserSymbol<61>;
|
||||
pub type Name_4_1 = GeneratedParserSymbol<62>;
|
||||
pub type Item_1 = GeneratedParserSymbol<63>;
|
||||
pub type Item_2 = GeneratedParserSymbol<64>;
|
||||
pub type Item_2_1 = GeneratedParserSymbol<65>;
|
||||
pub type Item_3 = GeneratedParserSymbol<66>;
|
||||
pub type Item_4 = GeneratedParserSymbol<67>;
|
||||
pub type Item_4_1 = GeneratedParserSymbol<68>;
|
||||
pub type Item_5 = GeneratedParserSymbol<69>;
|
||||
pub type Item_6 = GeneratedParserSymbol<70>;
|
||||
pub type Item_6_1 = GeneratedParserSymbol<71>;
|
||||
pub type Item_7 = GeneratedParserSymbol<72>;
|
||||
pub type Item_8 = GeneratedParserSymbol<73>;
|
||||
pub type Item_8_1 = GeneratedParserSymbol<74>;
|
||||
pub type Item_9 = GeneratedParserSymbol<75>;
|
||||
pub type Item_10 = GeneratedParserSymbol<76>;
|
||||
pub type Item_10_1 = GeneratedParserSymbol<77>;
|
||||
pub type Item_11 = GeneratedParserSymbol<78>;
|
||||
pub type Item_12 = GeneratedParserSymbol<79>;
|
||||
pub type Item_12_1 = GeneratedParserSymbol<80>;
|
||||
pub type Item_13 = GeneratedParserSymbol<81>;
|
||||
pub type Item_14 = GeneratedParserSymbol<82>;
|
||||
pub type Item_14_1 = GeneratedParserSymbol<83>;
|
||||
pub type Item_15 = GeneratedParserSymbol<84>;
|
||||
pub type Item_16 = GeneratedParserSymbol<85>;
|
||||
pub type Item_16_1 = GeneratedParserSymbol<86>;
|
||||
pub type Item_17 = GeneratedParserSymbol<87>;
|
||||
pub type Item_18 = GeneratedParserSymbol<88>;
|
||||
pub type Item_18_1 = GeneratedParserSymbol<89>;
|
||||
pub type Item_19 = GeneratedParserSymbol<90>;
|
||||
pub type Item_20 = GeneratedParserSymbol<91>;
|
||||
pub type Item_20_1 = GeneratedParserSymbol<92>;
|
||||
pub type Item_21 = GeneratedParserSymbol<93>;
|
||||
pub type Item_22 = GeneratedParserSymbol<94>;
|
||||
pub type Item_22_1 = GeneratedParserSymbol<95>;
|
||||
pub type Item_23 = GeneratedParserSymbol<96>;
|
||||
pub type Item_24 = GeneratedParserSymbol<97>;
|
||||
pub type Item_24_1 = GeneratedParserSymbol<98>;
|
||||
pub type GStuff = GeneratedParserSymbol<27>;
|
||||
pub type Stuff = GeneratedParserSymbol<36>;
|
||||
pub type Item = GeneratedParserSymbol<63>;
|
||||
pub type ClassBody = GeneratedParserSymbol<24>;
|
||||
pub type Cons = GeneratedParserSymbol<42>;
|
||||
pub type Call = GeneratedParserSymbol<45>;
|
||||
pub type BaseCall = GeneratedParserSymbol<48>;
|
||||
pub type Name = GeneratedParserSymbol<57>;
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct yycs0syntax(pub YyParser);
|
||||
impl yycs0syntax {
|
||||
pub fn new() -> Result<Self, Error> {
|
||||
Ok(Self(generated_parser()?))
|
||||
}
|
||||
pub fn action(&self, parser: Parser, symbol: SYMBOL, action: i32) -> Result<Object, Error> {
|
||||
self.0.action(parser, symbol, action)
|
||||
}
|
||||
pub fn class_body_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ClassBody_1::new(parser)?))
|
||||
}
|
||||
pub fn class_body_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ClassBody_2::new(parser)?))
|
||||
}
|
||||
pub fn class_body_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ClassBody_2_1::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_1::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_2::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_2_1::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_3_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_3::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_4_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_4::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_4_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_4_1::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_5_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_5::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_6_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_6::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_6_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff_6_1::new(parser)?))
|
||||
}
|
||||
pub fn stuff_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_1::new(parser)?))
|
||||
}
|
||||
pub fn stuff_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_2::new(parser)?))
|
||||
}
|
||||
pub fn stuff_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_2_1::new(parser)?))
|
||||
}
|
||||
pub fn stuff_3_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_3::new(parser)?))
|
||||
}
|
||||
pub fn stuff_4_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_4::new(parser)?))
|
||||
}
|
||||
pub fn stuff_4_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff_4_1::new(parser)?))
|
||||
}
|
||||
pub fn cons_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Cons_1::new(parser)?))
|
||||
}
|
||||
pub fn cons_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Cons_2::new(parser)?))
|
||||
}
|
||||
pub fn cons_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Cons_2_1::new(parser)?))
|
||||
}
|
||||
pub fn call_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Call_1::new(parser)?))
|
||||
}
|
||||
pub fn call_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Call_2::new(parser)?))
|
||||
}
|
||||
pub fn call_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Call_2_1::new(parser)?))
|
||||
}
|
||||
pub fn base_call_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_1::new(parser)?))
|
||||
}
|
||||
pub fn base_call_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_2::new(parser)?))
|
||||
}
|
||||
pub fn base_call_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_2_1::new(parser)?))
|
||||
}
|
||||
pub fn base_call_3_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_3::new(parser)?))
|
||||
}
|
||||
pub fn base_call_4_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_4::new(parser)?))
|
||||
}
|
||||
pub fn base_call_4_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_4_1::new(parser)?))
|
||||
}
|
||||
pub fn base_call_5_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_5::new(parser)?))
|
||||
}
|
||||
pub fn base_call_6_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_6::new(parser)?))
|
||||
}
|
||||
pub fn base_call_6_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall_6_1::new(parser)?))
|
||||
}
|
||||
pub fn name_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_1::new(parser)?))
|
||||
}
|
||||
pub fn name_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_2::new(parser)?))
|
||||
}
|
||||
pub fn name_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_2_1::new(parser)?))
|
||||
}
|
||||
pub fn name_3_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_3::new(parser)?))
|
||||
}
|
||||
pub fn name_4_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_4::new(parser)?))
|
||||
}
|
||||
pub fn name_4_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name_4_1::new(parser)?))
|
||||
}
|
||||
pub fn item_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_1::new(parser)?))
|
||||
}
|
||||
pub fn item_2_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_2::new(parser)?))
|
||||
}
|
||||
pub fn item_2_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_2_1::new(parser)?))
|
||||
}
|
||||
pub fn item_3_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_3::new(parser)?))
|
||||
}
|
||||
pub fn item_4_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_4::new(parser)?))
|
||||
}
|
||||
pub fn item_4_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_4_1::new(parser)?))
|
||||
}
|
||||
pub fn item_5_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_5::new(parser)?))
|
||||
}
|
||||
pub fn item_6_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_6::new(parser)?))
|
||||
}
|
||||
pub fn item_6_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_6_1::new(parser)?))
|
||||
}
|
||||
pub fn item_7_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_7::new(parser)?))
|
||||
}
|
||||
pub fn item_8_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_8::new(parser)?))
|
||||
}
|
||||
pub fn item_8_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_8_1::new(parser)?))
|
||||
}
|
||||
pub fn item_9_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_9::new(parser)?))
|
||||
}
|
||||
pub fn item_10_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_10::new(parser)?))
|
||||
}
|
||||
pub fn item_10_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_10_1::new(parser)?))
|
||||
}
|
||||
pub fn item_11_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_11::new(parser)?))
|
||||
}
|
||||
pub fn item_12_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_12::new(parser)?))
|
||||
}
|
||||
pub fn item_12_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_12_1::new(parser)?))
|
||||
}
|
||||
pub fn item_13_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_13::new(parser)?))
|
||||
}
|
||||
pub fn item_14_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_14::new(parser)?))
|
||||
}
|
||||
pub fn item_14_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_14_1::new(parser)?))
|
||||
}
|
||||
pub fn item_15_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_15::new(parser)?))
|
||||
}
|
||||
pub fn item_16_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_16::new(parser)?))
|
||||
}
|
||||
pub fn item_16_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_16_1::new(parser)?))
|
||||
}
|
||||
pub fn item_17_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_17::new(parser)?))
|
||||
}
|
||||
pub fn item_18_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_18::new(parser)?))
|
||||
}
|
||||
pub fn item_18_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_18_1::new(parser)?))
|
||||
}
|
||||
pub fn item_19_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_19::new(parser)?))
|
||||
}
|
||||
pub fn item_20_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_20::new(parser)?))
|
||||
}
|
||||
pub fn item_20_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_20_1::new(parser)?))
|
||||
}
|
||||
pub fn item_21_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_21::new(parser)?))
|
||||
}
|
||||
pub fn item_22_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_22::new(parser)?))
|
||||
}
|
||||
pub fn item_22_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_22_1::new(parser)?))
|
||||
}
|
||||
pub fn item_23_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_23::new(parser)?))
|
||||
}
|
||||
pub fn item_24_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_24::new(parser)?))
|
||||
}
|
||||
pub fn item_24_1_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item_24_1::new(parser)?))
|
||||
}
|
||||
pub fn g_stuff_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(GStuff::new(parser)?))
|
||||
}
|
||||
pub fn stuff_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Stuff::new(parser)?))
|
||||
}
|
||||
pub fn item_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Item::new(parser)?))
|
||||
}
|
||||
pub fn class_body_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ClassBody::new(parser)?))
|
||||
}
|
||||
pub fn cons_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Cons::new(parser)?))
|
||||
}
|
||||
pub fn call_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Call::new(parser)?))
|
||||
}
|
||||
pub fn base_call_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BaseCall::new(parser)?))
|
||||
}
|
||||
pub fn name_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(Name::new(parser)?))
|
||||
}
|
||||
pub fn error_factory(parser: Parser) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(crate::LslError::new_with_parser(parser)?))
|
||||
}
|
||||
}
|
||||
impl Deref for yycs0syntax {
|
||||
type Target = YyParser;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl DerefMut for yycs0syntax {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.0
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct yycs0tokens(pub YyLexer);
|
||||
impl yycs0tokens {
|
||||
pub fn new(error_handler: ErrorHandler) -> Result<Self, Error> {
|
||||
Ok(Self(generated_lexer_with_handler(error_handler)?))
|
||||
}
|
||||
pub fn base_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(BASE::new(lexer)?))
|
||||
}
|
||||
pub fn this_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(THIS::new(lexer)?))
|
||||
}
|
||||
pub fn new_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(NEW::new(lexer)?))
|
||||
}
|
||||
pub fn id_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ID::new(lexer)?))
|
||||
}
|
||||
pub fn any_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(ANY::new(lexer)?))
|
||||
}
|
||||
pub fn colon_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(COLON::new(lexer)?))
|
||||
}
|
||||
pub fn semicolon_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(SEMICOLON::new(lexer)?))
|
||||
}
|
||||
pub fn lbrace_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(LBRACE::new(lexer)?))
|
||||
}
|
||||
pub fn rbrace_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(RBRACE::new(lexer)?))
|
||||
}
|
||||
pub fn lparen_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(LPAREN::new(lexer)?))
|
||||
}
|
||||
pub fn rparen_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(RPAREN::new(lexer)?))
|
||||
}
|
||||
pub fn lbrack_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(LBRACK::new(lexer)?))
|
||||
}
|
||||
pub fn rbrack_factory(lexer: Lexer) -> Result<Object, Error> {
|
||||
Ok(Object::opaque(RBRACK::new(lexer)?))
|
||||
}
|
||||
pub fn old_action(
|
||||
&self,
|
||||
lexer: Lexer,
|
||||
yytext: &mut String,
|
||||
action: i32,
|
||||
reject: &mut bool,
|
||||
) -> Result<TOKEN, Error> {
|
||||
if matches!(action, 8 | 55 | 74) {
|
||||
match action {
|
||||
8 => "yym.yy_begin",
|
||||
55 => "((cs0tokens)yym)",
|
||||
_ => "((cs0syntax)yyq)",
|
||||
}
|
||||
.clone_into(yytext);
|
||||
return TOKEN::generated_with_lexer(lexer, "ANY", 7);
|
||||
}
|
||||
*reject = true;
|
||||
Err(Error::InvalidOperation)
|
||||
}
|
||||
}
|
||||
impl Deref for yycs0tokens {
|
||||
type Target = YyLexer;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.0
|
||||
}
|
||||
}
|
||||
impl DerefMut for yycs0tokens {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.0
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct cs0tokens {
|
||||
pub out: String,
|
||||
pub lexer: Lexer,
|
||||
}
|
||||
impl cs0tokens {
|
||||
pub fn new() -> Result<Self, Error> {
|
||||
Self::new_with_yy_lexer(generated_lexer()?)
|
||||
}
|
||||
pub fn new_with_constructor() -> Result<Self, Error> {
|
||||
Self::new()
|
||||
}
|
||||
pub fn new_with_error_handler(handler: ErrorHandler) -> Result<Self, Error> {
|
||||
Self::new_with_yy_lexer(generated_lexer_with_handler(handler)?)
|
||||
}
|
||||
pub fn new_with_yy_lexer(tokens: YyLexer) -> Result<Self, Error> {
|
||||
Ok(Self {
|
||||
out: String::new(),
|
||||
lexer: Lexer::new(tokens)?,
|
||||
})
|
||||
}
|
||||
}
|
||||
impl Deref for cs0tokens {
|
||||
type Target = Lexer;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.lexer
|
||||
}
|
||||
}
|
||||
impl DerefMut for cs0tokens {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.lexer
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct cs0syntax {
|
||||
pub out: String,
|
||||
pub cls: String,
|
||||
pub par: String,
|
||||
pub ctx: String,
|
||||
pub defconseen: bool,
|
||||
pub parser: Parser,
|
||||
}
|
||||
impl cs0syntax {
|
||||
pub fn new() -> Result<Self, Error> {
|
||||
Self::new_with_yy_parser(generated_parser()?)
|
||||
}
|
||||
pub fn new_with_constructor() -> Result<Self, Error> {
|
||||
Self::new()
|
||||
}
|
||||
pub fn new_with_yy_parser(symbols: YyParser) -> Result<Self, Error> {
|
||||
let lexer = Lexer::new(generated_lexer()?)?;
|
||||
Self::from_parts(symbols, lexer)
|
||||
}
|
||||
pub fn new_with_yy_parser_error_handler(
|
||||
mut symbols: YyParser,
|
||||
handler: ErrorHandler,
|
||||
) -> Result<Self, Error> {
|
||||
symbols.erh = handler.clone();
|
||||
let lexer = Lexer::new(generated_lexer_with_handler(handler)?)?;
|
||||
Self::from_parts(symbols, lexer)
|
||||
}
|
||||
fn from_parts(symbols: YyParser, lexer: Lexer) -> Result<Self, Error> {
|
||||
Ok(Self {
|
||||
out: String::new(),
|
||||
cls: String::new(),
|
||||
par: String::new(),
|
||||
ctx: String::new(),
|
||||
defconseen: false,
|
||||
parser: Parser::new(symbols, lexer)?,
|
||||
})
|
||||
}
|
||||
}
|
||||
impl Deref for cs0syntax {
|
||||
type Target = Parser;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
&self.parser
|
||||
}
|
||||
}
|
||||
impl DerefMut for cs0syntax {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
&mut self.parser
|
||||
}
|
||||
}
|
||||
1337
crates/metacrate-lsl-tools/src/generator.rs
Normal file
1337
crates/metacrate-lsl-tools/src/generator.rs
Normal file
File diff suppressed because it is too large
Load Diff
3171
crates/metacrate-lsl-tools/src/lexer.rs
Normal file
3171
crates/metacrate-lsl-tools/src/lexer.rs
Normal file
File diff suppressed because it is too large
Load Diff
28
crates/metacrate-lsl-tools/src/lib.rs
Normal file
28
crates/metacrate-lsl-tools/src/lib.rs
Normal file
@@ -0,0 +1,28 @@
|
||||
//! Native `LSL` tooling corresponding to `LibreMetaverse.LslTools`.
|
||||
//!
|
||||
//! This crate provides the bounded lexer, parser, diagnostics, syntax model,
|
||||
//! and deterministic source generation used by editor and migration tooling.
|
||||
//! It is pure Rust and does not require a grid connection.
|
||||
|
||||
extern crate self as metacrate_lsl_tools;
|
||||
|
||||
mod generated;
|
||||
mod generated_tables;
|
||||
mod generator;
|
||||
mod lexer;
|
||||
mod lsl_validation;
|
||||
mod parser;
|
||||
|
||||
pub use generated::*;
|
||||
pub use generated_tables::*;
|
||||
pub use lexer::{
|
||||
CharacterMatcher, DfaAccept, DfaState, Diagnostic, DiagnosticCategory, DiagnosticSeverity,
|
||||
DotNetUnicodeCategory, InputEncoding, LexerAction, LexerIterator, MAX_SOURCE_UNITS,
|
||||
MAX_TOKEN_UNITS, TokenDefinition, UnicodeClass,
|
||||
};
|
||||
pub use libremetaverse_types::Error;
|
||||
pub use lsl_validation::{LslSyntaxError, validate_lsl_source};
|
||||
pub use parser::{
|
||||
Associativity, ERROR_TOKEN, Grammar, GrammarProduction, GrammarSymbol, MAX_PARSER_STACK,
|
||||
MAX_PARSER_STATES, MAX_PARSER_STEPS, MAX_RECOVERY_ERRORS, ParseTree, ParserConflict,
|
||||
};
|
||||
134
crates/metacrate-lsl-tools/src/lsl_validation.rs
Normal file
134
crates/metacrate-lsl-tools/src/lsl_validation.rs
Normal file
@@ -0,0 +1,134 @@
|
||||
//! Bounded structural LSL source validation for native `MetaCrate` consumers.
|
||||
|
||||
use crate::MAX_SOURCE_UNITS;
|
||||
use std::fmt;
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum LslSyntaxError {
|
||||
Empty,
|
||||
Oversized,
|
||||
MissingDefaultState,
|
||||
UnterminatedString,
|
||||
UnterminatedComment,
|
||||
MismatchedDelimiter,
|
||||
InvalidCharacter,
|
||||
}
|
||||
|
||||
impl fmt::Display for LslSyntaxError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
write!(f, "{self:?}")
|
||||
}
|
||||
}
|
||||
impl std::error::Error for LslSyntaxError {}
|
||||
|
||||
/// Performs the bounded lexical and delimiter parse required before generated
|
||||
/// LSL may cross an inventory boundary. This rejects malformed lexical states
|
||||
/// and structural parse failures; it does not claim simulator runtime safety.
|
||||
///
|
||||
/// # Errors
|
||||
///
|
||||
/// Returns [`LslSyntaxError`] when the source exceeds its hard bound or its
|
||||
/// lexical/structural parse does not form a complete LSL default state.
|
||||
pub fn validate_lsl_source(source: &str) -> Result<(), LslSyntaxError> {
|
||||
if source.trim().is_empty() {
|
||||
return Err(LslSyntaxError::Empty);
|
||||
}
|
||||
if source.len() > MAX_SOURCE_UNITS {
|
||||
return Err(LslSyntaxError::Oversized);
|
||||
}
|
||||
if !source
|
||||
.split(|c: char| !c.is_alphanumeric() && c != '_')
|
||||
.any(|token| token == "default")
|
||||
{
|
||||
return Err(LslSyntaxError::MissingDefaultState);
|
||||
}
|
||||
let mut stack = Vec::new();
|
||||
let mut chars = source.chars().peekable();
|
||||
let mut string = false;
|
||||
let mut escaped = false;
|
||||
while let Some(ch) = chars.next() {
|
||||
if ch == '\0' {
|
||||
return Err(LslSyntaxError::InvalidCharacter);
|
||||
}
|
||||
if string {
|
||||
if escaped {
|
||||
escaped = false;
|
||||
} else if ch == '\\' {
|
||||
escaped = true;
|
||||
} else if ch == '"' {
|
||||
string = false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if ch == '"' {
|
||||
string = true;
|
||||
continue;
|
||||
}
|
||||
if ch == '/' && chars.peek() == Some(&'/') {
|
||||
chars.next();
|
||||
for next in chars.by_ref() {
|
||||
if next == '\n' {
|
||||
break;
|
||||
}
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if ch == '/' && chars.peek() == Some(&'*') {
|
||||
chars.next();
|
||||
let mut closed = false;
|
||||
while let Some(next) = chars.next() {
|
||||
if next == '*' && chars.peek() == Some(&'/') {
|
||||
chars.next();
|
||||
closed = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if !closed {
|
||||
return Err(LslSyntaxError::UnterminatedComment);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
match ch {
|
||||
'{' | '(' | '[' => stack.push(ch),
|
||||
'}' | ')' | ']' => {
|
||||
let expected = match ch {
|
||||
'}' => '{',
|
||||
')' => '(',
|
||||
_ => '[',
|
||||
};
|
||||
if stack.pop() != Some(expected) {
|
||||
return Err(LslSyntaxError::MismatchedDelimiter);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
if string {
|
||||
Err(LslSyntaxError::UnterminatedString)
|
||||
} else if stack.is_empty() {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(LslSyntaxError::MismatchedDelimiter)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[test]
|
||||
fn lexical_and_structural_states_are_bounded() {
|
||||
assert!(validate_lsl_source("default { state_entry() { llSay(0, \"hi\"); } }").is_ok());
|
||||
assert_eq!(
|
||||
validate_lsl_source("default { \"x"),
|
||||
Err(LslSyntaxError::UnterminatedString)
|
||||
);
|
||||
assert_eq!(
|
||||
validate_lsl_source("default { /*"),
|
||||
Err(LslSyntaxError::UnterminatedComment)
|
||||
);
|
||||
assert_eq!(
|
||||
validate_lsl_source("default { ]"),
|
||||
Err(LslSyntaxError::MismatchedDelimiter)
|
||||
);
|
||||
}
|
||||
}
|
||||
3027
crates/metacrate-lsl-tools/src/parser.rs
Normal file
3027
crates/metacrate-lsl-tools/src/parser.rs
Normal file
File diff suppressed because it is too large
Load Diff
339
crates/metacrate-lsl-tools/tests/generator_compat.rs
Normal file
339
crates/metacrate-lsl-tools/tests/generator_compat.rs
Normal file
@@ -0,0 +1,339 @@
|
||||
//! Focused compatibility fixtures for issue #83's checked-in LSL generator.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use libremetaverse_types::compat::{Object, Utf16CodeUnit};
|
||||
use metacrate_lsl_tools::{
|
||||
BASE, CSymbol, CSymbolSymType, Dfa, ErrorHandler, GENERATED_PARSER_DATA, ID,
|
||||
LSL_GENERATOR_SERIALIZATION_VERSION, Lexer, Nfa, NfaNode, ObjectList,
|
||||
ObjectListOListEnumerator, Path, Regex, SCreator, Serialiser, Sfactory, SymbolType, SymbolsGen,
|
||||
TCreator, Tfactory, TokClassDef, TokensGen, generated_lexer, generated_parser,
|
||||
};
|
||||
|
||||
struct SharedWriter(Arc<Mutex<Vec<u8>>>);
|
||||
|
||||
impl std::io::Write for SharedWriter {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.0
|
||||
.lock()
|
||||
.map_err(|_| std::io::Error::other("poisoned output"))?
|
||||
.extend_from_slice(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn capture<F>(function: F) -> Vec<u8>
|
||||
where
|
||||
F: FnOnce(Box<dyn std::io::Write + Send>),
|
||||
{
|
||||
let output = Arc::new(Mutex::new(Vec::new()));
|
||||
function(Box::new(SharedWriter(output.clone())));
|
||||
Arc::try_unwrap(output)
|
||||
.expect("single output owner")
|
||||
.into_inner()
|
||||
.expect("output")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generated_lexer_preserves_tokens_reserved_words_and_eof() {
|
||||
let mut lexer = Lexer::new(generated_lexer().expect("generated lexer")).expect("runtime");
|
||||
lexer
|
||||
.start_with_string("Widget(value;):base(new Child[])".to_owned())
|
||||
.expect("source");
|
||||
let mut tokens = Vec::new();
|
||||
while let Some(token) = lexer.next_token().expect("lexing") {
|
||||
tokens.push((token.yyname(), token.yytext(), token.yynum()));
|
||||
}
|
||||
assert_eq!(
|
||||
tokens,
|
||||
[
|
||||
("ID".to_owned(), "Widget".to_owned(), 6),
|
||||
("LPAREN".to_owned(), "(".to_owned(), 12),
|
||||
("ID".to_owned(), "value".to_owned(), 6),
|
||||
("SEMICOLON".to_owned(), ";".to_owned(), 9),
|
||||
("RPAREN".to_owned(), ")".to_owned(), 13),
|
||||
("COLON".to_owned(), ":".to_owned(), 8),
|
||||
("BASE".to_owned(), "base".to_owned(), 3),
|
||||
("LPAREN".to_owned(), "(".to_owned(), 12),
|
||||
("NEW".to_owned(), "new".to_owned(), 5),
|
||||
("ID".to_owned(), "Child".to_owned(), 6),
|
||||
("LBRACK".to_owned(), "[".to_owned(), 14),
|
||||
("RBRACK".to_owned(), "]".to_owned(), 15),
|
||||
("RPAREN".to_owned(), ")".to_owned(), 13),
|
||||
("EOF".to_owned(), "EOF".to_owned(), 2),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generated_parser_accepts_the_reviewed_class_body_grammar() {
|
||||
let table = generated_parser().expect("generated parser");
|
||||
assert_eq!(table.arr, GENERATED_PARSER_DATA);
|
||||
let lexer = Lexer::new(generated_lexer().expect("generated lexer")).expect("runtime");
|
||||
let mut parser = metacrate_lsl_tools::Parser::new(table, lexer).expect("parser");
|
||||
let result = parser
|
||||
.parse_with_string("Widget(value;):base(new Child[])".to_owned())
|
||||
.expect("class body");
|
||||
assert_eq!(result.yyname(), "ClassBody");
|
||||
assert_eq!(result.yynum(), 19);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parser_and_lexer_emission_is_byte_deterministic() {
|
||||
let first_parser = capture(|output| {
|
||||
generated_parser()
|
||||
.expect("parser")
|
||||
.emit(output)
|
||||
.expect("emit");
|
||||
});
|
||||
let second_parser = capture(|output| {
|
||||
generated_parser()
|
||||
.expect("parser")
|
||||
.emit(output)
|
||||
.expect("emit");
|
||||
});
|
||||
assert_eq!(first_parser, second_parser);
|
||||
|
||||
let first_lexer = capture(|output| {
|
||||
generated_lexer()
|
||||
.expect("lexer")
|
||||
.emit_dfa(output)
|
||||
.expect("emit");
|
||||
});
|
||||
let second_lexer = capture(|output| {
|
||||
generated_lexer()
|
||||
.expect("lexer")
|
||||
.emit_dfa(output)
|
||||
.expect("emit");
|
||||
});
|
||||
assert_eq!(first_lexer, second_lexer);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn generated_token_classes_have_live_identity_and_text() {
|
||||
let mut lexer = Lexer::new(generated_lexer().expect("lexer")).expect("runtime");
|
||||
lexer
|
||||
.start_with_string("base name".to_owned())
|
||||
.expect("source");
|
||||
let base = lexer.next_token().expect("lexing").expect("base");
|
||||
let mapped_base = BASE::new(lexer.clone()).expect("mapped base");
|
||||
assert_eq!((base.yyname(), base.yynum()), ("BASE".to_owned(), 3));
|
||||
assert_eq!(
|
||||
(mapped_base.yyname(), mapped_base.yynum()),
|
||||
("BASE".to_owned(), 3)
|
||||
);
|
||||
let _ = lexer.next_token().expect("lexing").expect("name");
|
||||
let mapped_id = ID::new(lexer).expect("mapped id");
|
||||
assert_eq!(
|
||||
(mapped_id.yyname(), mapped_id.yynum()),
|
||||
("ID".to_owned(), 6)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn regex_and_nfa_build_a_functional_native_dfa() {
|
||||
let handler = ErrorHandler::default();
|
||||
let tokens = TokensGen::new(handler.clone()).expect("generator");
|
||||
let regex = Regex::new(tokens.clone(), 0, "[A-Z][a-z]+".to_owned()).expect("regex");
|
||||
assert!(
|
||||
!regex
|
||||
.match__with_char(Utf16CodeUnit(u16::from(b'Q')))
|
||||
.expect("single-unit mismatch")
|
||||
);
|
||||
assert!(
|
||||
!regex
|
||||
.match__with_char(Utf16CodeUnit(u16::from(b'q')))
|
||||
.expect("mismatch")
|
||||
);
|
||||
assert_eq!(
|
||||
regex
|
||||
.match__with_string("Zebra".to_owned())
|
||||
.expect("prefix"),
|
||||
5
|
||||
);
|
||||
|
||||
let mut nfa = Nfa::new_with_tokens_gen_regex(tokens, regex).expect("nfa");
|
||||
nfa.m_end.m_s_terminal = "CAPITAL".to_owned();
|
||||
let dfa = Dfa::new_with_nfa(nfa).expect("dfa");
|
||||
let mut table = metacrate_lsl_tools::YyLexer::new(handler).expect("table");
|
||||
table.set_start_dfa("YYINITIAL", dfa).expect("start");
|
||||
table.using_eof = true;
|
||||
let mut lexer = Lexer::new(table).expect("runtime");
|
||||
lexer.start_with_string("Rust".to_owned()).expect("source");
|
||||
let token = lexer.next_token().expect("lexing").expect("token");
|
||||
assert_eq!(
|
||||
(token.yyname(), token.yytext()),
|
||||
("CAPITAL".to_owned(), "Rust".to_owned())
|
||||
);
|
||||
|
||||
let tokens = TokensGen::new(ErrorHandler::default()).expect("manual generator");
|
||||
let mut manual = Nfa::new_with_tokens_gen(tokens.clone()).expect("manual nfa");
|
||||
manual.m_end.m_s_terminal = "PAIR".to_owned();
|
||||
let middle = NfaNode::new(tokens).expect("middle state");
|
||||
manual
|
||||
.start
|
||||
.add_arc(Utf16CodeUnit(u16::from(b'a')), middle.clone())
|
||||
.expect("first arc");
|
||||
middle
|
||||
.add_arc(Utf16CodeUnit(u16::from(b'b')), manual.m_end.clone())
|
||||
.expect("second arc");
|
||||
let dfa = Dfa::new_with_nfa(manual).expect("determinised nfa");
|
||||
let mut action = -1;
|
||||
assert_eq!(
|
||||
dfa.match_("ab".to_owned(), 0, &mut action).expect("pair"),
|
||||
2
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn serialiser_round_trips_deterministically_with_versioning() {
|
||||
let value = Object::Map(HashMap::from([
|
||||
("b".to_owned(), Object::Integer(7)),
|
||||
(
|
||||
"a".to_owned(),
|
||||
Object::Array(vec![Object::Boolean(true), Object::String("λ".to_owned())]),
|
||||
),
|
||||
]));
|
||||
let encoded = capture(|output| {
|
||||
let serialiser = Serialiser::new_with_text_writer(output).expect("writer");
|
||||
serialiser.version_check().expect("version");
|
||||
serialiser.serialise(value.clone()).expect("serialise");
|
||||
});
|
||||
let encoded_again = capture(|output| {
|
||||
let serialiser = Serialiser::new_with_text_writer(output).expect("writer");
|
||||
serialiser.version_check().expect("version");
|
||||
serialiser.serialise(value.clone()).expect("serialise");
|
||||
});
|
||||
assert_eq!(encoded, encoded_again);
|
||||
let integers = String::from_utf8(encoded)
|
||||
.expect("ASCII integers")
|
||||
.split(',')
|
||||
.filter(|value| !value.trim().is_empty())
|
||||
.map(|value| value.trim().parse::<i32>().expect("integer"))
|
||||
.collect::<Vec<_>>();
|
||||
let decoder = Serialiser::new_with_int32_array(integers).expect("reader");
|
||||
decoder
|
||||
.version_check()
|
||||
.expect(LSL_GENERATOR_SERIALIZATION_VERSION);
|
||||
assert_eq!(decoder.deserialise().expect("deserialise"), value);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compatibility_enumerator_and_generator_state_are_live() {
|
||||
let mut values = ObjectList::new().expect("list");
|
||||
values.add(Object::Integer(1)).expect("first");
|
||||
values
|
||||
.add(Object::String("two".to_owned()))
|
||||
.expect("second");
|
||||
let enumerator = ObjectListOListEnumerator::new(values).expect("enumerator");
|
||||
assert_eq!(enumerator.current(), Object::Undefined);
|
||||
assert!(enumerator.move_next().expect("first"));
|
||||
assert_eq!(enumerator.current(), Object::Integer(1));
|
||||
assert!(enumerator.move_next().expect("second"));
|
||||
assert_eq!(enumerator.current(), Object::String("two".to_owned()));
|
||||
assert!(!enumerator.move_next().expect("end"));
|
||||
enumerator.reset().expect("reset");
|
||||
assert!(enumerator.move_next().expect("first again"));
|
||||
|
||||
let generator = TokensGen::new(ErrorHandler::default()).expect("generator");
|
||||
assert_eq!(generator.new_state().expect("state one"), 1);
|
||||
assert_eq!(generator.new_state().expect("state two"), 2);
|
||||
assert_eq!(
|
||||
generator
|
||||
.fix_actions("yybegin(); yyl".to_owned())
|
||||
.expect("rewritten action"),
|
||||
"yym.yy_begin(); ((tokens)yym)"
|
||||
);
|
||||
assert!(Dfa::new_with_tokens_gen(generator).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn symbol_and_token_factories_execute_registered_rust_closures() {
|
||||
let parser = metacrate_lsl_tools::Parser::new(
|
||||
generated_parser().expect("parser table"),
|
||||
Lexer::new(generated_lexer().expect("lexer table")).expect("lexer"),
|
||||
)
|
||||
.expect("parser");
|
||||
let symbol_creator = SCreator::from_fn(|_| Ok(Object::String("symbol".to_owned())));
|
||||
let registration = Sfactory::new(
|
||||
generated_parser().expect("parser table"),
|
||||
"GeneratedSymbol".to_owned(),
|
||||
symbol_creator,
|
||||
)
|
||||
.expect("symbol registration");
|
||||
assert_eq!(registration.name(), "GeneratedSymbol");
|
||||
assert_eq!(
|
||||
Sfactory::create("GeneratedSymbol_Derived".to_owned(), parser).expect("symbol factory"),
|
||||
Object::String("symbol".to_owned())
|
||||
);
|
||||
|
||||
let lexer = Lexer::new(generated_lexer().expect("lexer table")).expect("lexer");
|
||||
let token_creator = TCreator::from_fn(|_| Ok(Object::Integer(83)));
|
||||
let registration = Tfactory::new(
|
||||
generated_lexer().expect("lexer table"),
|
||||
"GeneratedToken".to_owned(),
|
||||
token_creator,
|
||||
)
|
||||
.expect("token registration");
|
||||
assert_eq!(registration.name(), "GeneratedToken");
|
||||
assert_eq!(
|
||||
Tfactory::create("GeneratedToken_Derived".to_owned(), lexer).expect("token factory"),
|
||||
Object::Integer(83)
|
||||
);
|
||||
|
||||
let callback_ran = Arc::new(AtomicBool::new(false));
|
||||
let callback_state = callback_ran.clone();
|
||||
let async_creator = TCreator::from_fn(|_| Ok(Object::String("async".to_owned())));
|
||||
let result = async_creator
|
||||
.begin_invoke(
|
||||
Lexer::new(generated_lexer().expect("lexer table")).expect("lexer"),
|
||||
Box::new(move |_| callback_state.store(true, Ordering::SeqCst)),
|
||||
Object::Undefined,
|
||||
)
|
||||
.expect("begin invoke");
|
||||
assert!(callback_ran.load(Ordering::SeqCst));
|
||||
assert_eq!(
|
||||
async_creator.end_invoke(result).expect("end invoke"),
|
||||
Object::String("async".to_owned())
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn symbols_paths_and_token_class_definitions_keep_native_state() {
|
||||
let symbols = SymbolsGen::new(ErrorHandler::default()).expect("symbols generator");
|
||||
let declared = SymbolType::new_with_symbols_gen_string_boolean(
|
||||
symbols.clone(),
|
||||
"SemanticValue".to_owned(),
|
||||
true,
|
||||
)
|
||||
.expect("symbol type");
|
||||
assert_eq!(
|
||||
declared
|
||||
.find("SemanticValue".to_owned())
|
||||
.expect("lookup")
|
||||
.name,
|
||||
"SemanticValue"
|
||||
);
|
||||
|
||||
let identifier = CSymbol::native("ID", 6, CSymbolSymType::Terminal).expect("symbol");
|
||||
let path = Path::new_with_c_symbol_array(vec![identifier.clone()]).expect("path");
|
||||
assert_eq!(path.spelling()[0].m_yynum, 6);
|
||||
assert!(!path.valid);
|
||||
assert_eq!(path.top().m_state, 0);
|
||||
|
||||
let output = Arc::new(Mutex::new(Vec::new()));
|
||||
let generator =
|
||||
metacrate_lsl_tools::GenBase::new(ErrorHandler::default(), Box::new(SharedWriter(output)))
|
||||
.expect("generator base");
|
||||
let definition = TokClassDef::new(generator, "IDENTIFIER".to_owned(), "TOKEN".to_owned())
|
||||
.expect("token class");
|
||||
assert_eq!(definition.m_name, "IDENTIFIER");
|
||||
assert_eq!(definition.m_ref_token, "TOKEN");
|
||||
assert_eq!(definition.m_yynum, 3);
|
||||
}
|
||||
391
crates/metacrate-lsl-tools/tests/lexer_compat.rs
Normal file
391
crates/metacrate-lsl-tools/tests/lexer_compat.rs
Normal file
@@ -0,0 +1,391 @@
|
||||
//! Focused compatibility fixtures translated from the pinned lexer runtime.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use libremetaverse_types::compat::{Object, UnicodeCategory, Utf16CodeUnit};
|
||||
use metacrate_lsl_tools::{
|
||||
CSToolsException, CharacterMatcher, CsReader, Dfa, DfaAccept, DfaState, DiagnosticCategory,
|
||||
DotNetUnicodeCategory, Error, ErrorHandler, InputEncoding, Lexer, LexerAction, LineManager,
|
||||
ObjectList, ResWds, SourceLineInfo, TOKEN, TokenDefinition, YyLexer,
|
||||
};
|
||||
|
||||
fn token(name: &str, number: i32) -> TokenDefinition {
|
||||
TokenDefinition::new(name, number).expect("valid definition")
|
||||
}
|
||||
|
||||
fn accept(name: &str, number: i32, action: LexerAction, reserved_words: Option<&str>) -> DfaAccept {
|
||||
DfaAccept {
|
||||
token: token(name, number),
|
||||
action,
|
||||
action_number: number,
|
||||
reserved_words: reserved_words.map(ToOwned::to_owned),
|
||||
}
|
||||
}
|
||||
|
||||
fn language_table() -> YyLexer {
|
||||
let letters = vec![
|
||||
DotNetUnicodeCategory::UppercaseLetter,
|
||||
DotNetUnicodeCategory::LowercaseLetter,
|
||||
DotNetUnicodeCategory::TitlecaseLetter,
|
||||
DotNetUnicodeCategory::ModifierLetter,
|
||||
DotNetUnicodeCategory::OtherLetter,
|
||||
];
|
||||
let whitespace = vec![
|
||||
DotNetUnicodeCategory::Control,
|
||||
DotNetUnicodeCategory::SpaceSeparator,
|
||||
DotNetUnicodeCategory::LineSeparator,
|
||||
DotNetUnicodeCategory::ParagraphSeparator,
|
||||
];
|
||||
let states = vec![
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(letters.clone()), 1)
|
||||
.transition(CharacterMatcher::Exact(u16::from(b'_')), 1)
|
||||
.transition(
|
||||
CharacterMatcher::Category(DotNetUnicodeCategory::DecimalDigitNumber),
|
||||
2,
|
||||
)
|
||||
.transition(CharacterMatcher::Categories(whitespace.clone()), 3)
|
||||
.transition(CharacterMatcher::Exact(u16::from(b'+')), 4),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(letters), 1)
|
||||
.transition(CharacterMatcher::Exact(u16::from(b'_')), 1)
|
||||
.transition(
|
||||
CharacterMatcher::Category(DotNetUnicodeCategory::DecimalDigitNumber),
|
||||
1,
|
||||
)
|
||||
.accepting(accept("ID", 3, LexerAction::Emit, Some("keywords"))),
|
||||
DfaState::default()
|
||||
.transition(
|
||||
CharacterMatcher::Category(DotNetUnicodeCategory::DecimalDigitNumber),
|
||||
2,
|
||||
)
|
||||
.accepting(accept("INTEGER", 4, LexerAction::Emit, None)),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(whitespace), 3)
|
||||
.accepting(accept("WS", 5, LexerAction::Skip, None)),
|
||||
DfaState::default().accepting(accept("PLUS", 6, LexerAction::Emit, None)),
|
||||
];
|
||||
let mut table = YyLexer::new(ErrorHandler::default()).expect("table");
|
||||
table.using_eof = true;
|
||||
table
|
||||
.set_start_dfa("YYINITIAL", Dfa::from_states(states, 0).expect("dfa"))
|
||||
.expect("start state");
|
||||
table
|
||||
.set_reserved_words(
|
||||
"keywords",
|
||||
ResWds::from_pairs(
|
||||
[("if", token("IF", 20)), ("while", token("WHILE", 21))],
|
||||
false,
|
||||
)
|
||||
.expect("reserved words"),
|
||||
)
|
||||
.expect("reserved table");
|
||||
table
|
||||
}
|
||||
|
||||
fn lexer(source: &str) -> Lexer {
|
||||
let mut lexer = Lexer::new(language_table()).expect("lexer");
|
||||
lexer
|
||||
.start_with_string(source.to_owned())
|
||||
.expect("valid source");
|
||||
lexer
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn token_fixture_preserves_names_text_numbers_positions_and_eof() {
|
||||
let mut lexer = lexer("if café_2 + 19");
|
||||
let mut tokens = Vec::new();
|
||||
while let Some(value) = lexer.next_token().expect("tokenization") {
|
||||
tokens.push(value);
|
||||
}
|
||||
let actual = tokens
|
||||
.iter()
|
||||
.map(|token| {
|
||||
(
|
||||
token.yyname(),
|
||||
token.yytext(),
|
||||
token.yynum(),
|
||||
token.pos,
|
||||
token.end,
|
||||
)
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(
|
||||
actual,
|
||||
[
|
||||
("IF".to_owned(), "if".to_owned(), 20, 0, 2),
|
||||
("ID".to_owned(), "café_2".to_owned(), 3, 3, 9),
|
||||
("PLUS".to_owned(), "+".to_owned(), 6, 10, 11),
|
||||
("INTEGER".to_owned(), "19".to_owned(), 4, 12, 14),
|
||||
("EOF".to_owned(), "EOF".to_owned(), 2, 15, 15),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reserved_words_are_exact_unless_the_table_requests_uppercase() {
|
||||
let mut lexer = lexer("IF if");
|
||||
assert_eq!(lexer.next().expect("identifier").yyname(), "ID");
|
||||
assert_eq!(lexer.next().expect("keyword").yyname(), "IF");
|
||||
|
||||
let words = ResWds::from_pairs([("while", token("WHILE", 21))], true).expect("words");
|
||||
let mut value = TOKEN::new_with_lexer_string(lexer.clone(), "WhIlE".to_owned()).expect("token");
|
||||
words.check(lexer, &mut value).expect("check");
|
||||
assert_eq!((value.yyname(), value.yynum()), ("WHILE".to_owned(), 21));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn invalid_input_reports_stable_category_location_and_text() {
|
||||
let mut lexer = lexer("ok @ nope");
|
||||
assert_eq!(lexer.next().expect("first").yytext(), "ok");
|
||||
assert_eq!(
|
||||
lexer.next().expect_err("invalid character"),
|
||||
Error::Parse {
|
||||
position: 3,
|
||||
context: "input does not match any lexer rule"
|
||||
}
|
||||
);
|
||||
let diagnostic = &lexer.diagnostics()[0];
|
||||
assert_eq!(diagnostic.category, DiagnosticCategory::InvalidCharacter);
|
||||
assert_eq!(diagnostic.location.line_number, 1);
|
||||
assert_eq!(diagnostic.location.raw_char_position, 3);
|
||||
assert_eq!(diagnostic.input, "@");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_start_condition_reports_invalid_state_without_changing_state() {
|
||||
let mut lexer = lexer("value");
|
||||
assert_eq!(
|
||||
lexer.yy_begin("MISSING".to_owned()),
|
||||
Err(Error::Parse {
|
||||
position: 0,
|
||||
context: "unknown lexer start condition"
|
||||
})
|
||||
);
|
||||
assert_eq!(lexer.m_state, "YYINITIAL");
|
||||
assert_eq!(
|
||||
lexer.diagnostics()[0].category,
|
||||
DiagnosticCategory::InvalidState
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn source_reader_removes_both_comment_forms_and_keeps_newlines() {
|
||||
let mut reader = CsReader::new_with_string("a/*x\ny*/b//z\nc".to_owned()).expect("reader");
|
||||
assert_eq!(reader.read_line().expect("line"), "a");
|
||||
assert_eq!(reader.read_line().expect("line"), "b");
|
||||
assert_eq!(reader.read_line().expect("line"), "c");
|
||||
assert!(reader.eof().expect("eof"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn source_reader_normalizes_crlf_and_lone_cr() {
|
||||
let mut reader = CsReader::new_with_string("a\r\nb\rc\n".to_owned()).expect("reader");
|
||||
assert_eq!(reader.read_line().expect("line"), "a");
|
||||
assert_eq!(reader.read_line().expect("line"), "b");
|
||||
assert_eq!(reader.read_line().expect("line"), "c");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn source_reader_rejects_unterminated_block_comment_at_eof() {
|
||||
assert_eq!(
|
||||
CsReader::new_with_string("before /* never closed".to_owned()).expect_err("invalid"),
|
||||
Error::Parse {
|
||||
position: 22,
|
||||
context: "unterminated block comment"
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn line_directive_updates_file_and_logical_line() {
|
||||
let reader =
|
||||
CsReader::new_with_string("#line 700 \"table.lex\"\nword\n".to_owned()).expect("reader");
|
||||
assert_eq!(reader.fname, "table.lex");
|
||||
let info = SourceLineInfo::new_with_line_manager_int32(reader.lm, 1).expect("location");
|
||||
assert_eq!(info.line_number, 700);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn utf16_input_decoding_and_surrogate_validation_are_deterministic() {
|
||||
let mut reader = CsReader::from_bytes(
|
||||
&[0xFF, 0xFE, b'A', 0, 0x3D, 0xD8, 0, 0xDE],
|
||||
InputEncoding::Utf16Le,
|
||||
"source.lex",
|
||||
)
|
||||
.expect("utf16");
|
||||
assert_eq!(reader.read_line().expect("line"), "A😀");
|
||||
let mut bom_override =
|
||||
CsReader::from_bytes(&[0xFE, 0xFF, 0, b'B'], InputEncoding::Utf16Le, "source.lex")
|
||||
.expect("BOM override");
|
||||
assert_eq!(bom_override.read_line().expect("line"), "B");
|
||||
assert!(matches!(
|
||||
CsReader::from_bytes(&[0x00, 0xD8], InputEncoding::Utf16Le, "bad"),
|
||||
Err(Error::Parse {
|
||||
position: 0,
|
||||
context: "source contains an unpaired UTF-16 surrogate"
|
||||
})
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ascii_input_rejects_non_ascii_instead_of_lossily_replacing_it() {
|
||||
assert!(matches!(
|
||||
CsReader::from_bytes(&[b'a', 0x80], InputEncoding::Ascii, "bad"),
|
||||
Err(Error::Parse {
|
||||
position: 1,
|
||||
context: "source contains a non-ASCII byte"
|
||||
})
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn category_predicates_match_dotnet_values_and_groups() {
|
||||
let mut table = YyLexer::new(ErrorHandler::default()).expect("table");
|
||||
let punctuation = table.get_test("Punctuation").expect("punctuation");
|
||||
let whitespace = table.get_test("WhiteSpace").expect("whitespace");
|
||||
let states = vec![
|
||||
DfaState::default()
|
||||
.transition(punctuation, 1)
|
||||
.transition(whitespace, 2),
|
||||
DfaState::default().accepting(accept("PUNCT", 1, LexerAction::Emit, None)),
|
||||
DfaState::default().accepting(accept("SPACE", 2, LexerAction::Emit, None)),
|
||||
];
|
||||
let dfa = Dfa::from_states(states, 0).expect("dfa");
|
||||
let mut action = -1;
|
||||
assert_eq!(dfa.match_("—".to_owned(), 0, &mut action), Ok(1));
|
||||
assert_eq!(dfa.match_("\t".to_owned(), 0, &mut action), Ok(1));
|
||||
assert_eq!(dfa.match_("\0".to_owned(), 0, &mut action), Ok(-1));
|
||||
assert!(
|
||||
metacrate_lsl_tools::CatTest::new(UnicodeCategory(16))
|
||||
.expect("surrogate")
|
||||
.test(Utf16CodeUnit(0xD800))
|
||||
.expect("test")
|
||||
);
|
||||
assert_eq!(table.get_test("MissingClass"), Err(Error::Argument));
|
||||
assert_eq!(
|
||||
table.erh.diagnostics()[0].category,
|
||||
DiagnosticCategory::UnknownCharacterSet
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn dfa_uses_maximum_munch_and_exposes_action_number() {
|
||||
let states = vec![
|
||||
DfaState::default().transition(CharacterMatcher::Exact(b'a'.into()), 1),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Exact(b'b'.into()), 2)
|
||||
.accepting(accept("A", 1, LexerAction::Emit, None)),
|
||||
DfaState::default().accepting(accept("AB", 2, LexerAction::Emit, None)),
|
||||
];
|
||||
let dfa = Dfa::from_states(states, 0).expect("dfa");
|
||||
let mut action = -1;
|
||||
assert_eq!(dfa.match_("abc".to_owned(), 0, &mut action), Ok(2));
|
||||
assert_eq!(action, 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deterministic_table_output_is_independent_of_hash_iteration() {
|
||||
let table = language_table();
|
||||
let first = Arc::new(Mutex::new(Vec::new()));
|
||||
table
|
||||
.emit_dfa(Box::new(SharedWriter(first.clone())))
|
||||
.expect("emit first");
|
||||
let second = Arc::new(Mutex::new(Vec::new()));
|
||||
table
|
||||
.emit_dfa(Box::new(SharedWriter(second.clone())))
|
||||
.expect("emit second");
|
||||
let first = first.lock().expect("first output").clone();
|
||||
let second = second.lock().expect("second output").clone();
|
||||
assert_eq!(first, second);
|
||||
assert!(
|
||||
String::from_utf8(first)
|
||||
.expect("utf8")
|
||||
.starts_with("LSLLEXER 1\n")
|
||||
);
|
||||
}
|
||||
|
||||
struct SharedWriter(Arc<Mutex<Vec<u8>>>);
|
||||
|
||||
impl std::io::Write for SharedWriter {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.0
|
||||
.lock()
|
||||
.map_err(|_| std::io::Error::other("poisoned output"))?
|
||||
.extend_from_slice(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compatibility_enumerator_resets_to_the_first_token() {
|
||||
let lexer = lexer("one two");
|
||||
let mut enumerator = lexer.get_enumerator().expect("enumerator");
|
||||
assert!(enumerator.move_next().expect("first"));
|
||||
assert_eq!(enumerator.current().yytext(), "one");
|
||||
assert!(enumerator.move_next().expect("second"));
|
||||
assert_eq!(enumerator.current().yytext(), "two");
|
||||
enumerator.reset().expect("reset");
|
||||
assert!(enumerator.move_next().expect("first again"));
|
||||
assert_eq!(enumerator.current().yytext(), "one");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn error_handler_counts_reports_and_honors_throw_mode() {
|
||||
let error =
|
||||
CSToolsException::new_with_int32_string(43, "bad encoding".to_owned()).expect("diagnostic");
|
||||
let mut collecting = ErrorHandler::new_with_constructor().expect("handler");
|
||||
collecting.error(error.clone()).expect("reported");
|
||||
assert_eq!(collecting.counter, 1);
|
||||
assert_eq!(
|
||||
collecting.diagnostics()[0].category,
|
||||
DiagnosticCategory::Encoding
|
||||
);
|
||||
|
||||
let mut throwing = ErrorHandler::new_with_boolean(true).expect("handler");
|
||||
assert_eq!(throwing.error(error), Err(Error::InvalidOperation));
|
||||
assert_eq!(throwing.counter, 1);
|
||||
assert!(throwing.diagnostics().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn line_manager_backtracking_removes_future_lines() {
|
||||
let mut manager = LineManager::new().expect("manager");
|
||||
manager.newline(4).expect("line 2");
|
||||
let first_on_second =
|
||||
SourceLineInfo::new_with_line_manager_int32(manager.clone(), 4).expect("location");
|
||||
assert_eq!(first_on_second.raw_char_position, 1);
|
||||
manager.newline(8).expect("line 3");
|
||||
manager.backto(6).expect("rewind");
|
||||
assert_eq!(manager.lines, 2);
|
||||
assert_eq!(manager.end, 8);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn object_list_preserves_push_add_top_pop_and_index_order() {
|
||||
let mut values = ObjectList::new().expect("list");
|
||||
values.add(Object::Integer(2)).expect("add");
|
||||
values.push(Object::Integer(1)).expect("push");
|
||||
assert_eq!(values.count(), 2);
|
||||
assert_eq!(values.top(), Object::Integer(1));
|
||||
assert_eq!(values.item(1), Object::Integer(2));
|
||||
assert_eq!(values.pop(), Ok(Object::Integer(1)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn matcher_set_is_value_based_and_bounded() {
|
||||
let matcher = CharacterMatcher::Set(BTreeSet::from([b'x'.into(), b'y'.into()]));
|
||||
let states = vec![
|
||||
DfaState::default().transition(matcher, 1),
|
||||
DfaState::default().accepting(accept("XY", 1, LexerAction::Emit, None)),
|
||||
];
|
||||
let dfa = Dfa::from_states(states, 0).expect("dfa");
|
||||
let mut action = 0;
|
||||
assert_eq!(dfa.match_("y".to_owned(), 0, &mut action), Ok(1));
|
||||
assert_eq!(dfa.match_("z".to_owned(), 0, &mut action), Ok(-1));
|
||||
}
|
||||
473
crates/metacrate-lsl-tools/tests/parser_compat.rs
Normal file
473
crates/metacrate-lsl-tools/tests/parser_compat.rs
Normal file
@@ -0,0 +1,473 @@
|
||||
//! Focused compatibility fixtures for the native issue #82 parser boundary.
|
||||
|
||||
use std::sync::{Arc, Mutex};
|
||||
|
||||
use metacrate_lsl_tools::{
|
||||
Associativity, CSymbol, CSymbolSymType, CharacterMatcher, Dfa, DfaAccept, DfaState,
|
||||
DiagnosticCategory, Error, ErrorHandler, Grammar, Lexer, LexerAction, LslError, ParseTree,
|
||||
Parser, ParserEntry, Precedence, PrecedencePrecType, Production, ResWds, SymbolSet,
|
||||
TokenDefinition, YyLexer,
|
||||
};
|
||||
|
||||
const EOF: i32 = 2;
|
||||
const ID: i32 = 3;
|
||||
const INTEGER: i32 = 4;
|
||||
const PLUS: i32 = 5;
|
||||
const STAR: i32 = 6;
|
||||
const MINUS: i32 = 7;
|
||||
const LPAREN: i32 = 8;
|
||||
const RPAREN: i32 = 9;
|
||||
const SEMICOLON: i32 = 10;
|
||||
const COMMA: i32 = 11;
|
||||
const STRING: i32 = 12;
|
||||
const LBRACE: i32 = 13;
|
||||
const RBRACE: i32 = 14;
|
||||
const EQUAL: i32 = 15;
|
||||
const DEFAULT: i32 = 20;
|
||||
const STATE_ENTRY: i32 = 21;
|
||||
|
||||
fn definition(name: &str, number: i32) -> TokenDefinition {
|
||||
TokenDefinition::new(name, number).expect("valid token")
|
||||
}
|
||||
|
||||
fn accept(name: &str, number: i32, action: LexerAction) -> DfaAccept {
|
||||
DfaAccept {
|
||||
token: definition(name, number),
|
||||
action,
|
||||
action_number: number,
|
||||
reserved_words: (number == ID).then(|| "keywords".to_owned()),
|
||||
}
|
||||
}
|
||||
|
||||
fn parser_lexer(source: &str) -> Lexer {
|
||||
use metacrate_lsl_tools::DotNetUnicodeCategory as Category;
|
||||
|
||||
let letters = vec![
|
||||
Category::UppercaseLetter,
|
||||
Category::LowercaseLetter,
|
||||
Category::TitlecaseLetter,
|
||||
Category::ModifierLetter,
|
||||
Category::OtherLetter,
|
||||
];
|
||||
let whitespace = vec![
|
||||
Category::Control,
|
||||
Category::SpaceSeparator,
|
||||
Category::LineSeparator,
|
||||
Category::ParagraphSeparator,
|
||||
];
|
||||
let states = vec![
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(letters.clone()), 1)
|
||||
.transition(CharacterMatcher::Exact(b'_'.into()), 1)
|
||||
.transition(CharacterMatcher::Category(Category::DecimalDigitNumber), 2)
|
||||
.transition(CharacterMatcher::Categories(whitespace.clone()), 3)
|
||||
.transition(CharacterMatcher::Exact(b'+'.into()), 4)
|
||||
.transition(CharacterMatcher::Exact(b'*'.into()), 5)
|
||||
.transition(CharacterMatcher::Exact(b'-'.into()), 6)
|
||||
.transition(CharacterMatcher::Exact(b'('.into()), 7)
|
||||
.transition(CharacterMatcher::Exact(b')'.into()), 8)
|
||||
.transition(CharacterMatcher::Exact(b';'.into()), 9)
|
||||
.transition(CharacterMatcher::Exact(b','.into()), 10)
|
||||
.transition(CharacterMatcher::Exact(b'{'.into()), 11)
|
||||
.transition(CharacterMatcher::Exact(b'}'.into()), 12)
|
||||
.transition(CharacterMatcher::Exact(b'"'.into()), 13)
|
||||
.transition(CharacterMatcher::Exact(b'='.into()), 15),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(letters.clone()), 1)
|
||||
.transition(CharacterMatcher::Exact(b'_'.into()), 1)
|
||||
.transition(CharacterMatcher::Category(Category::DecimalDigitNumber), 1)
|
||||
.accepting(accept("ID", ID, LexerAction::Emit)),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Category(Category::DecimalDigitNumber), 2)
|
||||
.accepting(accept("INTEGER", INTEGER, LexerAction::Emit)),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(whitespace), 3)
|
||||
.accepting(accept("WS", 30, LexerAction::Skip)),
|
||||
DfaState::default().accepting(accept("PLUS", PLUS, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("STAR", STAR, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("MINUS", MINUS, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("LPAREN", LPAREN, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("RPAREN", RPAREN, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("SEMICOLON", SEMICOLON, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("COMMA", COMMA, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("LBRACE", LBRACE, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("RBRACE", RBRACE, LexerAction::Emit)),
|
||||
DfaState::default()
|
||||
.transition(CharacterMatcher::Categories(letters), 13)
|
||||
.transition(CharacterMatcher::Exact(b' '.into()), 13)
|
||||
.transition(CharacterMatcher::Exact(b'"'.into()), 14),
|
||||
DfaState::default().accepting(accept("STRING", STRING, LexerAction::Emit)),
|
||||
DfaState::default().accepting(accept("EQUAL", EQUAL, LexerAction::Emit)),
|
||||
];
|
||||
let mut table = YyLexer::new(ErrorHandler::default()).expect("lexer table");
|
||||
table.using_eof = true;
|
||||
table
|
||||
.set_start_dfa("YYINITIAL", Dfa::from_states(states, 0).expect("dfa"))
|
||||
.expect("start state");
|
||||
table
|
||||
.set_reserved_words(
|
||||
"keywords",
|
||||
ResWds::from_pairs(
|
||||
[
|
||||
("default", definition("DEFAULT", DEFAULT)),
|
||||
("state_entry", definition("STATE_ENTRY", STATE_ENTRY)),
|
||||
],
|
||||
false,
|
||||
)
|
||||
.expect("reserved words"),
|
||||
)
|
||||
.expect("reserved table");
|
||||
let mut lexer = Lexer::new(table).expect("lexer");
|
||||
lexer.start_with_string(source.to_owned()).expect("source");
|
||||
lexer
|
||||
}
|
||||
|
||||
fn add_common_terminals(grammar: &mut Grammar) {
|
||||
for (name, number) in [
|
||||
("EOF", EOF),
|
||||
("ID", ID),
|
||||
("INTEGER", INTEGER),
|
||||
("PLUS", PLUS),
|
||||
("STAR", STAR),
|
||||
("MINUS", MINUS),
|
||||
("LPAREN", LPAREN),
|
||||
("RPAREN", RPAREN),
|
||||
("SEMICOLON", SEMICOLON),
|
||||
("COMMA", COMMA),
|
||||
("STRING", STRING),
|
||||
("LBRACE", LBRACE),
|
||||
("RBRACE", RBRACE),
|
||||
("EQUAL", EQUAL),
|
||||
("DEFAULT", DEFAULT),
|
||||
("STATE_ENTRY", STATE_ENTRY),
|
||||
("Error", 0),
|
||||
] {
|
||||
grammar.add_symbol(name, number, true).expect("terminal");
|
||||
}
|
||||
}
|
||||
|
||||
fn expression_parser(source: &str) -> Parser {
|
||||
const EXPRESSION: i32 = 100;
|
||||
let mut grammar = Grammar::new(EXPRESSION, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
grammar
|
||||
.add_symbol("Expression", EXPRESSION, false)
|
||||
.expect("nonterminal");
|
||||
grammar
|
||||
.set_precedence(PLUS, 1, Associativity::Left)
|
||||
.expect("plus precedence");
|
||||
grammar
|
||||
.set_precedence(STAR, 2, Associativity::Left)
|
||||
.expect("star precedence");
|
||||
grammar
|
||||
.set_precedence(MINUS, 3, Associativity::Right)
|
||||
.expect("minus precedence");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![EXPRESSION, PLUS, EXPRESSION])
|
||||
.expect("plus");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![EXPRESSION, STAR, EXPRESSION])
|
||||
.expect("multiply");
|
||||
grammar
|
||||
.add_production_with_precedence(EXPRESSION, vec![MINUS, EXPRESSION], Some(MINUS))
|
||||
.expect("unary");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![LPAREN, EXPRESSION, RPAREN])
|
||||
.expect("parentheses");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![ID])
|
||||
.expect("identifier");
|
||||
Parser::new(grammar.build().expect("table"), parser_lexer(source)).expect("parser")
|
||||
}
|
||||
|
||||
fn tree(symbol: &metacrate_lsl_tools::SYMBOL) -> &ParseTree {
|
||||
symbol
|
||||
.m_dollar
|
||||
.downcast_ref::<ParseTree>()
|
||||
.expect("parse tree")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn precedence_and_associativity_choose_the_reference_tree() {
|
||||
let mut parser = expression_parser("a + b * c + d");
|
||||
let result = parser
|
||||
.parse_with_string("a + b * c + d".to_owned())
|
||||
.expect("parse");
|
||||
let root = tree(&result);
|
||||
assert_eq!(root.name, "Expression");
|
||||
assert_eq!(root.children[1].number, PLUS);
|
||||
assert_eq!(root.children[0].children[1].number, PLUS);
|
||||
assert_eq!(root.children[0].children[2].children[1].number, STAR);
|
||||
assert!(!parser.m_symbols.conflicts().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn right_associative_unary_precedence_is_deterministic() {
|
||||
let mut parser = expression_parser("- - a");
|
||||
let result = parser.parse_with_string("- - a".to_owned()).expect("parse");
|
||||
let root = tree(&result);
|
||||
assert_eq!(root.children[0].number, MINUS);
|
||||
assert_eq!(root.children[1].children[0].number, MINUS);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nonassociative_conflict_rejects_a_chained_operator() {
|
||||
const EXPRESSION: i32 = 100;
|
||||
let mut grammar = Grammar::new(EXPRESSION, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
grammar
|
||||
.add_symbol("Expression", EXPRESSION, false)
|
||||
.expect("nonterminal");
|
||||
grammar
|
||||
.set_precedence(PLUS, 1, Associativity::Nonassoc)
|
||||
.expect("precedence");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![EXPRESSION, PLUS, EXPRESSION])
|
||||
.expect("binary");
|
||||
grammar
|
||||
.add_production(EXPRESSION, vec![ID])
|
||||
.expect("identifier");
|
||||
let mut parser =
|
||||
Parser::new(grammar.build().expect("table"), parser_lexer("a+a+a")).expect("parser");
|
||||
assert!(matches!(
|
||||
parser.parse_with_string("a+a+a".to_owned()),
|
||||
Err(Error::Parse {
|
||||
context: "syntax error",
|
||||
..
|
||||
})
|
||||
));
|
||||
assert_eq!(
|
||||
parser.diagnostics().expect("diagnostics")[0].category,
|
||||
DiagnosticCategory::Syntax
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_production_accepts_an_empty_input() {
|
||||
const START: i32 = 100;
|
||||
const ITEMS: i32 = 101;
|
||||
let mut grammar = Grammar::new(START, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
grammar.add_symbol("Start", START, false).expect("start");
|
||||
grammar.add_symbol("Items", ITEMS, false).expect("items");
|
||||
grammar.add_production(START, vec![ITEMS]).expect("root");
|
||||
grammar.add_production(ITEMS, vec![]).expect("empty");
|
||||
let mut parser =
|
||||
Parser::new(grammar.build().expect("table"), parser_lexer("")).expect("parser");
|
||||
let result = parser.parse_with_string(String::new()).expect("parse");
|
||||
assert_eq!(tree(&result).children[0].name, "Items");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn lalr_lookaheads_avoid_the_classic_slr_assignment_conflict() {
|
||||
const START: i32 = 100;
|
||||
const LEFT: i32 = 101;
|
||||
const RIGHT: i32 = 102;
|
||||
let mut grammar = Grammar::new(START, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
for (name, number) in [("Start", START), ("Left", LEFT), ("Right", RIGHT)] {
|
||||
grammar
|
||||
.add_symbol(name, number, false)
|
||||
.expect("nonterminal");
|
||||
}
|
||||
grammar
|
||||
.add_production(START, vec![LEFT, EQUAL, RIGHT])
|
||||
.expect("assignment");
|
||||
grammar.add_production(START, vec![RIGHT]).expect("value");
|
||||
grammar
|
||||
.add_production(LEFT, vec![STAR, RIGHT])
|
||||
.expect("dereference");
|
||||
grammar.add_production(LEFT, vec![ID]).expect("name");
|
||||
grammar.add_production(RIGHT, vec![LEFT]).expect("right");
|
||||
let table = grammar.build().expect("LALR table");
|
||||
assert!(table.conflicts().is_empty());
|
||||
for source in ["a = b", "* a = b"] {
|
||||
let mut parser = Parser::new(table.clone(), parser_lexer(source)).expect("parser");
|
||||
let result = parser
|
||||
.parse_with_string(source.to_owned())
|
||||
.expect("assignment parse");
|
||||
assert_eq!(tree(&result).name, "Start");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn error_token_recovery_discards_input_and_returns_recovered_tree() {
|
||||
const START: i32 = 100;
|
||||
const LIST: i32 = 101;
|
||||
const STATEMENT: i32 = 102;
|
||||
let mut grammar = Grammar::new(START, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
for (name, number) in [("Start", START), ("List", LIST), ("Statement", STATEMENT)] {
|
||||
grammar
|
||||
.add_symbol(name, number, false)
|
||||
.expect("nonterminal");
|
||||
}
|
||||
grammar.add_production(START, vec![LIST]).expect("start");
|
||||
grammar
|
||||
.add_production(LIST, vec![LIST, STATEMENT])
|
||||
.expect("list");
|
||||
grammar
|
||||
.add_production(LIST, vec![STATEMENT])
|
||||
.expect("single");
|
||||
grammar
|
||||
.add_production(STATEMENT, vec![ID, SEMICOLON])
|
||||
.expect("statement");
|
||||
grammar
|
||||
.add_production(STATEMENT, vec![0, ID, SEMICOLON])
|
||||
.expect("recovery");
|
||||
let mut parser =
|
||||
Parser::new(grammar.build().expect("table"), parser_lexer("a; + b;")).expect("parser");
|
||||
let result = parser
|
||||
.parse_with_string("a; + b;".to_owned())
|
||||
.expect("recovered parse");
|
||||
assert_eq!(result.yyname(), "recoveredError");
|
||||
let diagnostics = parser.diagnostics().expect("diagnostics");
|
||||
assert_eq!(diagnostics[0].category, DiagnosticCategory::Syntax);
|
||||
assert_eq!(diagnostics[1].category, DiagnosticCategory::ParserRecovery);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn representative_lsl_script_parses_to_expected_model() {
|
||||
const SCRIPT: i32 = 100;
|
||||
const EVENT: i32 = 101;
|
||||
const CALL: i32 = 102;
|
||||
let source = r#"default { state_entry() { llSay(0, "hello"); } }"#;
|
||||
let mut grammar = Grammar::new(SCRIPT, EOF).expect("grammar");
|
||||
add_common_terminals(&mut grammar);
|
||||
for (name, number) in [("Script", SCRIPT), ("Event", EVENT), ("Call", CALL)] {
|
||||
grammar
|
||||
.add_symbol(name, number, false)
|
||||
.expect("nonterminal");
|
||||
}
|
||||
grammar
|
||||
.add_production(SCRIPT, vec![DEFAULT, LBRACE, EVENT, RBRACE])
|
||||
.expect("script");
|
||||
grammar
|
||||
.add_production(
|
||||
EVENT,
|
||||
vec![STATE_ENTRY, LPAREN, RPAREN, LBRACE, CALL, SEMICOLON, RBRACE],
|
||||
)
|
||||
.expect("event");
|
||||
grammar
|
||||
.add_production(CALL, vec![ID, LPAREN, INTEGER, COMMA, STRING, RPAREN])
|
||||
.expect("call");
|
||||
let mut parser =
|
||||
Parser::new(grammar.build().expect("table"), parser_lexer(source)).expect("parser");
|
||||
let result = parser
|
||||
.parse_with_string(source.to_owned())
|
||||
.expect("LSL source");
|
||||
let root = tree(&result);
|
||||
assert_eq!(root.name, "Script");
|
||||
assert_eq!(root.children[2].name, "Event");
|
||||
assert_eq!(root.children[2].children[4].name, "Call");
|
||||
assert_eq!(
|
||||
root.children[2].children[4].children[0].text.as_deref(),
|
||||
Some("llSay")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parser_table_emission_is_byte_deterministic() {
|
||||
let parser = expression_parser("a");
|
||||
let first = Arc::new(Mutex::new(Vec::new()));
|
||||
parser
|
||||
.m_symbols
|
||||
.emit(Box::new(SharedWriter(first.clone())))
|
||||
.expect("first");
|
||||
let second = Arc::new(Mutex::new(Vec::new()));
|
||||
parser
|
||||
.m_symbols
|
||||
.emit(Box::new(SharedWriter(second.clone())))
|
||||
.expect("second");
|
||||
assert_eq!(
|
||||
first.lock().expect("first output").as_slice(),
|
||||
second.lock().expect("second output").as_slice()
|
||||
);
|
||||
}
|
||||
|
||||
struct SharedWriter(Arc<Mutex<Vec<u8>>>);
|
||||
|
||||
impl std::io::Write for SharedWriter {
|
||||
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
|
||||
self.0
|
||||
.lock()
|
||||
.map_err(|_| std::io::Error::other("poisoned output"))?
|
||||
.extend_from_slice(bytes);
|
||||
Ok(bytes.len())
|
||||
}
|
||||
|
||||
fn flush(&mut self) -> std::io::Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_corpus_terminates_without_panics_or_unbounded_growth() {
|
||||
for source in ["+", "a +", "((a", "a * * b", "-", "a )", "@"] {
|
||||
let mut parser = expression_parser(source);
|
||||
let _ = parser.parse_with_string(source.to_owned());
|
||||
assert!(parser.diagnostics().expect("diagnostics").len() <= 2);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn mapped_symbol_set_production_precedence_and_entry_apis_are_live() {
|
||||
let terminal = CSymbol::native("PLUS", PLUS, CSymbolSymType::Terminal).expect("terminal");
|
||||
let nonterminal =
|
||||
CSymbol::native("Expression", 100, CSymbolSymType::Nonterminal).expect("nonterminal");
|
||||
let mut set = SymbolSet::native();
|
||||
assert!(set.check_in(terminal.clone()).expect("insert"));
|
||||
assert!(!set.check_in(terminal.clone()).expect("deduplicate"));
|
||||
assert!(set.contains(terminal.clone()).expect("contains"));
|
||||
assert!(terminal.matches("PLUS".to_owned()).expect("symbol text"));
|
||||
|
||||
let mut followed = nonterminal.clone();
|
||||
assert!(
|
||||
followed
|
||||
.add_follow(terminal.m_first.clone())
|
||||
.expect("first follow insertion")
|
||||
);
|
||||
assert_eq!(nonterminal.m_follow.count(), 1);
|
||||
|
||||
let nullable =
|
||||
CSymbol::native("Optional", 101, CSymbolSymType::Nonterminal).expect("nullable symbol");
|
||||
let _empty = Production::native(8, nullable.clone(), vec![]).expect("empty production");
|
||||
assert!(nullable.is_nullable().expect("registered empty production"));
|
||||
|
||||
let mut production = Production::native(7, nonterminal.clone(), vec![]).expect("production");
|
||||
production.add_to_rhs(nonterminal.clone()).expect("lhs");
|
||||
production.add_to_rhs(terminal.clone()).expect("operator");
|
||||
assert_eq!(production.prefix(1).expect("prefix")[0].m_yynum, 100);
|
||||
let target = CSymbol::native("Target", 102, CSymbolSymType::Nonterminal).expect("target");
|
||||
production
|
||||
.add_first(target.clone(), 1)
|
||||
.expect("FIRST/FOLLOW propagation");
|
||||
assert!(
|
||||
target
|
||||
.m_follow
|
||||
.contains(terminal.clone())
|
||||
.expect("shared follow mutation")
|
||||
);
|
||||
|
||||
let precedence = Precedence::first(PrecedencePrecType::Left, 5).expect("precedence");
|
||||
assert_eq!(
|
||||
Precedence::check_with_precedence_prec_type_int32(precedence, PrecedencePrecType::Left, 0,),
|
||||
Ok(5)
|
||||
);
|
||||
|
||||
let parser = expression_parser("a");
|
||||
let mut lexer = parser.m_lexer.clone();
|
||||
let symbol = lexer.next_token().expect("lexing").expect("token");
|
||||
let mut entry = ParserEntry::default();
|
||||
assert_eq!(
|
||||
symbol.pass_(parser.m_symbols.clone(), 0, &mut entry),
|
||||
Ok(true)
|
||||
);
|
||||
assert_eq!(entry.str(), "shift 1");
|
||||
|
||||
let lsl_error = LslError {
|
||||
state: 4,
|
||||
sym: metacrate_lsl_tools::SYMBOL::new_with_lexer(parser.m_lexer.clone()).expect("symbol"),
|
||||
};
|
||||
assert!(lsl_error.to_string().contains("state 4"));
|
||||
}
|
||||
Reference in New Issue
Block a user