Apply Qwen NFC normalization before token splitting

This commit is contained in:
Georg Bauer committed 2026-10-04 17:14:00 +02:00
1 parent 5129c450ee
commit 6507a494bc
36 files changed
+1831 -39

No files matched your search

+7
View File
@@ -87,3 +87,10 @@ denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tes
(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
5 passed; the one 27B Quality-pack test skipped because that distinct model pack
is not installed. The installed Flash Next model was exercised above.
## Follow-up #116
NFC is now implemented. The component-stage test still checks every original
normalization-free split/BPE ID; it additionally checks the full tokenizer against
all `full_ids` and the normalized decode, closing the one-case gap documented
above. See `../issue116-nfc/` for full normalization and cache migration receipts.