Match Qwen Unicode number token boundaries
This commit is contained in:
1 parent
990abb4755
commit
8965e77ea1
24 files changed
+27460
-3
No files matched your search
@@ -0,0 +1,9 @@
|
||||
|
||||
running 2 tests
|
||||
test engine::tokenizer::issue111::issue111_ds4_families ... {"case":"ds4_control","family":"glm","cases":19,"plain_rendered_exact":true}
|
||||
{"case":"ds4_control","family":"deepseek","cases":19,"plain_rendered_exact":true}
|
||||
ok
|
||||
test engine::tokenizer::issue111::issue111_legacy_requests ... ok
|
||||
|
||||
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 533 filtered out; finished in 0.80s
|
||||
|
||||
Reference in new issue
Block a user