hugo/DS4Server
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Match Qwen Unicode number token boundaries

Browse Source
This commit is contained in:
Georg Bauer committed 2026-10-04 15:37:04 +02:00
1 parent 990abb4755
commit 8965e77ea1
24 files changed
+27460 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/issue111-tokenizer/ds4-chat.log
+28
View File
@@ -0,0 +1,28 @@
before chat-token-fixture.json
running 1 test
test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 534 filtered out; finished in 0.14s
before glm-chat-token-fixture.json
running 1 test
test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 534 filtered out; finished in 0.23s
after chat-token-fixture.json
running 1 test
test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 537 filtered out; finished in 0.15s
after glm-chat-token-fixture.json
running 1 test
test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 537 filtered out; finished in 0.23s