Apply Qwen NFC normalization before token splitting
This commit is contained in:
1 parent
5129c450ee
commit
6507a494bc
36 files changed
+1831
-39
No files matched your search
@@ -87,3 +87,10 @@ denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tes
|
||||
(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
|
||||
5 passed; the one 27B Quality-pack test skipped because that distinct model pack
|
||||
is not installed. The installed Flash Next model was exercised above.
|
||||
|
||||
## Follow-up #116
|
||||
|
||||
NFC is now implemented. The component-stage test still checks every original
|
||||
normalization-free split/BPE ID; it additionally checks the full tokenizer against
|
||||
all `full_ids` and the normalized decode, closing the one-case gap documented
|
||||
above. See `../issue116-nfc/` for full normalization and cache migration receipts.
|
||||
Reference in new issue
Block a user