hugo/DS4Server
1
0 Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Apply Qwen NFC normalization before token splitting

Browse Source
This commit is contained in:
Georg Bauer committed 2026-10-04 17:14:00 +02:00
1 parent 5129c450ee
commit 6507a494bc
36 files changed
+1831 -39
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/issue116-nfc/nfc.output.log
+15
View File
@@ -0,0 +1,15 @@
running 1 test
test engine::tokenizer::issue116::issue116_nfc_tokens_and_requests ... {"effort":"direct","system":"","context_tokens":52,"exact":true}
{"effort":"direct","system":"Café 각.","context_tokens":62,"exact":true}
{"effort":"low","system":"","context_tokens":85,"exact":true}
{"effort":"low","system":"Café 각.","context_tokens":91,"exact":true}
{"effort":"medium","system":"","context_tokens":55,"exact":true}
{"effort":"medium","system":"Café 각.","context_tokens":65,"exact":true}
{"effort":"xhigh","system":"","context_tokens":97,"exact":true}
{"effort":"xhigh","system":"Café 각.","context_tokens":103,"exact":true}
{"case":"nfc","scalar_sha256":"1905b5f60f629850aa35dfc8ccb11fa6ae4d1fb0c79efd6774d36ca8e3e64eab","token_cases":13281,"requests":8,"exact":true}
ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 554 filtered out; finished in 2.30s