Apply Qwen NFC normalization before token splitting

2026-10-04 17:14:00 +02:00
1 parent 5129c450ee
commit 6507a494bc
@@ -0,0 +1,6 @@
+
+running 1 test
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 554 filtered out; finished in 0.22s
+