The installed Qwen tokenizer has an NFC normalizer, while DS4Server sends unnormalized text directly to byte BPE. Independently discovered during #111, whose explicit scope excludes normalization.
Pinned MTPLX 2.12.2 revision 9882703f3105363ddc37eca9f97aa09a1d387112, unchanged Bare-Speed artifact revision 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7. For text ́a á ́ १́b (the second a is followed by U+0301), full oracle IDs are [52033,64,18135,220,52033,220,12256,100,52033,65]; native IDs after the #111 split correction are [52033,64,264,52033,220,52033,220,12256,100,52033,65]. The full oracle decodes the composed NFC form. Removing only the normalizer from an isolated oracle tokenizer produces the native IDs, identifying the separate stage precisely. Of 5,752 #111 component cases, this one differs at the full-tokenizer stage.
Implement the artifact normalizer contract in Qwen Rust token processing, preserving special-token segmentation and the trained split/BPE sequence. Review every plain/rendered/history/continuation path and normalized context counts against the pinned oracle. Do not normalize GLM or DeepSeek; preserve exact DS4 controls. Cover canonical compositions, reordered combining marks, Hangul, already-normalized text, number categories, special boundaries and decode behavior. Changed IDs must safely miss old token-exact cache prefixes while new prefixes reuse safely. No unconditional checkpoint format bump. Run all commit gates plus isolated tokenizer/request/cache comparisons. Production remains Rust, with no Python runtime calls. Evidence will be in docs/issue111-tokenizer and tools/issue111-reference.py when #111 is committed.
The installed Qwen tokenizer has an NFC normalizer, while DS4Server sends unnormalized text directly to byte BPE. Independently discovered during #111, whose explicit scope excludes normalization.
Pinned MTPLX 2.12.2 revision `9882703f3105363ddc37eca9f97aa09a1d387112`, unchanged Bare-Speed artifact revision `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`. For text `́a á ́ १́b` (the second a is followed by U+0301), full oracle IDs are `[52033,64,18135,220,52033,220,12256,100,52033,65]`; native IDs after the #111 split correction are `[52033,64,264,52033,220,52033,220,12256,100,52033,65]`. The full oracle decodes the composed NFC form. Removing only the normalizer from an isolated oracle tokenizer produces the native IDs, identifying the separate stage precisely. Of 5,752 #111 component cases, this one differs at the full-tokenizer stage.
Implement the artifact normalizer contract in Qwen Rust token processing, preserving special-token segmentation and the trained split/BPE sequence. Review every plain/rendered/history/continuation path and normalized context counts against the pinned oracle. Do not normalize GLM or DeepSeek; preserve exact DS4 controls. Cover canonical compositions, reordered combining marks, Hangul, already-normalized text, number categories, special boundaries and decode behavior. Changed IDs must safely miss old token-exact cache prefixes while new prefixes reuse safely. No unconditional checkpoint format bump. Run all commit gates plus isolated tokenizer/request/cache comparisons. Production remains Rust, with no Python runtime calls. Evidence will be in docs/issue111-tokenizer and tools/issue111-reference.py when #111 is committed.
hugo
added the bug label 2026-10-04 13:33:42 +00:00
Implemented and pushed in 6507a494bc5cdfc373e5bc8056719e151766a373.
Qwen now enforces the artifact's NFC contract and applies the same Rust normalization implementation used by the pinned tokenizer, after added-token segmentation and before the trained split/BPE. Plain/rendered, system, history, full request and continuation paths are covered; GLM/DeepSeek processing is unchanged.
Review also found the completed-history tag fast path could bypass token-prefix migration. Qwen keys now distinguish only text fields changed by NFC; all producers/consumers (persistent/transient, live/disk, bootstrap, compaction and cleanup) use the actual model family. Already-normalized Qwen keys and both DS4 families remain byte-identical. No checkpoint-format bump.
Independent pinned-oracle verification: all 1,112,064 Unicode scalars in combining context; 13,281 normalization/plain/rendered/decode cases including all Hangul syllables and every added-token boundary; eight full/history+continuation request comparisons with exact context counts; all 5,752 prior split/BPE and full-tokenizer cases. The originally reported text now exactly matches the oracle IDs. Both DS4 families' token and chat-render controls pass.
Installed-model verification: legacy completed history misses in both live and disk paths, corrected prompt has 91 tokens, and new warm/disk continuations reuse 93 tokens with identical complete state. The unchanged compatible-cache regression retains 95-token reuse. All four commit gates pass (316 application + 8 tool tests). Receipts and reproduction commands: docs/issue116-nfc/. #117 remains separately scoped.
Implemented and pushed in `6507a494bc5cdfc373e5bc8056719e151766a373`.
Qwen now enforces the artifact's NFC contract and applies the same Rust normalization implementation used by the pinned tokenizer, after added-token segmentation and before the trained split/BPE. Plain/rendered, system, history, full request and continuation paths are covered; GLM/DeepSeek processing is unchanged.
Review also found the completed-history tag fast path could bypass token-prefix migration. Qwen keys now distinguish only text fields changed by NFC; all producers/consumers (persistent/transient, live/disk, bootstrap, compaction and cleanup) use the actual model family. Already-normalized Qwen keys and both DS4 families remain byte-identical. No checkpoint-format bump.
Independent pinned-oracle verification: all 1,112,064 Unicode scalars in combining context; 13,281 normalization/plain/rendered/decode cases including all Hangul syllables and every added-token boundary; eight full/history+continuation request comparisons with exact context counts; all 5,752 prior split/BPE and full-tokenizer cases. The originally reported text now exactly matches the oracle IDs. Both DS4 families' token and chat-render controls pass.
Installed-model verification: legacy completed history misses in both live and disk paths, corrected prompt has 91 tokens, and new warm/disk continuations reuse 93 tokens with identical complete state. The unchanged compatible-cache regression retains 95-token reuse. All four commit gates pass (316 application + 8 tool tests). Receipts and reproduction commands: `docs/issue116-nfc/`. #117 remains separately scoped.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The installed Qwen tokenizer has an NFC normalizer, while DS4Server sends unnormalized text directly to byte BPE. Independently discovered during #111, whose explicit scope excludes normalization.
Pinned MTPLX 2.12.2 revision
9882703f3105363ddc37eca9f97aa09a1d387112, unchanged Bare-Speed artifact revision74559cdf34fbfc0b593de72d17e93f37fd4f9ea7. For text́a á ́ १́b(the second a is followed by U+0301), full oracle IDs are[52033,64,18135,220,52033,220,12256,100,52033,65]; native IDs after the #111 split correction are[52033,64,264,52033,220,52033,220,12256,100,52033,65]. The full oracle decodes the composed NFC form. Removing only the normalizer from an isolated oracle tokenizer produces the native IDs, identifying the separate stage precisely. Of 5,752 #111 component cases, this one differs at the full-tokenizer stage.Implement the artifact normalizer contract in Qwen Rust token processing, preserving special-token segmentation and the trained split/BPE sequence. Review every plain/rendered/history/continuation path and normalized context counts against the pinned oracle. Do not normalize GLM or DeepSeek; preserve exact DS4 controls. Cover canonical compositions, reordered combining marks, Hangul, already-normalized text, number categories, special boundaries and decode behavior. Changed IDs must safely miss old token-exact cache prefixes while new prefixes reuse safely. No unconditional checkpoint format bump. Run all commit gates plus isolated tokenizer/request/cache comparisons. Production remains Rust, with no Python runtime calls. Evidence will be in docs/issue111-tokenizer and tools/issue111-reference.py when #111 is committed.
Implemented and pushed in
6507a494bc5cdfc373e5bc8056719e151766a373.Qwen now enforces the artifact's NFC contract and applies the same Rust normalization implementation used by the pinned tokenizer, after added-token segmentation and before the trained split/BPE. Plain/rendered, system, history, full request and continuation paths are covered; GLM/DeepSeek processing is unchanged.
Review also found the completed-history tag fast path could bypass token-prefix migration. Qwen keys now distinguish only text fields changed by NFC; all producers/consumers (persistent/transient, live/disk, bootstrap, compaction and cleanup) use the actual model family. Already-normalized Qwen keys and both DS4 families remain byte-identical. No checkpoint-format bump.
Independent pinned-oracle verification: all 1,112,064 Unicode scalars in combining context; 13,281 normalization/plain/rendered/decode cases including all Hangul syllables and every added-token boundary; eight full/history+continuation request comparisons with exact context counts; all 5,752 prior split/BPE and full-tokenizer cases. The originally reported text now exactly matches the oracle IDs. Both DS4 families' token and chat-render controls pass.
Installed-model verification: legacy completed history misses in both live and disk paths, corrected prompt has 91 tokens, and new warm/disk continuations reuse 93 tokens with identical complete state. The unchanged compatible-cache regression retains 95-token reuse. All four commit gates pass (316 application + 8 tool tests). Receipts and reproduction commands:
docs/issue116-nfc/. #117 remains separately scoped.