Match Qwen Unicode number token boundaries

This commit is contained in:
Georg Bauer committed 2026-10-04 15:37:04 +02:00
1 parent 990abb4755
commit 8965e77ea1
24 files changed
+27460 -3

No files matched your search

+9
View File
@@ -0,0 +1,9 @@
running 2 tests
test engine::tokenizer::issue111::issue111_ds4_families ... {"case":"ds4_control","family":"glm","cases":19,"plain_rendered_exact":true}
{"case":"ds4_control","family":"deepseek","cases":19,"plain_rendered_exact":true}
ok
test engine::tokenizer::issue111::issue111_legacy_requests ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 533 filtered out; finished in 0.80s