DS4Server/docs/issue116-nfc/README.md

Qwen NFC normalization (#116)

Qwen requires the installed artifact's NFC normalizer at load time and normalizes each ordinary text span before the trained Qwen split/BPE. Rendered added tokens are extracted first and remain literal boundaries. Plain text keeps its existing literal-marker semantics. DeepSeek JoyAI and GLM GPT4 token processing are unchanged.

The Rust dependency unicode-normalization-alignments 0.1.12 is the same normalization implementation used by tokenizers 0.22.2, the pinned oracle's installed tokenizer. Its Unicode tables are version 9.0.0; substituting newer tables would change the oracle contract. Python is only used to generate independent reference receipts, never in production.

Reference and isolated verification

Pinned MTPLX: 9882703f3105363ddc37eca9f97aa09a1d387112; Bare-Speed artifact revision: 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7. Tokenizer SHA-256: 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3. DS4 revision: ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748.

• All 1,112,064 Unicode scalars in a + scalar + acute + dot-below + separator context match the oracle normalization digest 1905b5f60f629850aa35dfc8ccb11fa6ae4d1fb0c79efd6774d36ca8e3e64eab.
• 13,281 exact normalized-text/plain-ID/rendered-ID/decode cases include canonical decompositions, reordered and leading marks, long combining sequences, all 11,172 Hangul syllables, already-normalized text, compatibility characters that NFC must preserve, and every added-token boundary. Python Unicode data only enumerates inputs; expected results come from the installed oracle normalizer/tokenizer.
• Eight three-message requests cover direct/low/medium/xhigh, with and without a decomposed system prompt. Full encoding and history plus continuation match the pinned server exactly. Context counts are 52/62, 85/91, 55/65, and 97/103 respectively.
• The original #116 text now emits [52033,64,18135,220,52033,220,12256,100,52033,65] and decodes to the oracle's composed form.
• All prior #111 component split/BPE expectations remain checked independently before normalization. The full native tokenizer now matches all 5,752 full-oracle cases, including the previously excluded NFC difference. All-number-category and exhaustive split controls remain exact.
• GLM and DeepSeek each retain exact DS4 plain/rendered IDs on all 19 shared controls. Existing DS4 chat render controls also run unchanged.

Cache and caller review

Every plain, rendered, full conversation, history, continuation, system-prompt and context-count path reaches the Qwen normalizer. Generation output bytes are not re-tokenized during a compatible completed-history continuation; its generated KV remains exact.

Token-prefix comparison alone was insufficient: a completed-conversation tag can authorize appending to old token IDs without rendering history. Qwen conversation keys now include a per-field NFC discriminator only when that field changes under the pinned normalizer. Raw histories remain distinct, appended-history keys remain prefixes, and pre-NFC affected keys cannot match new keys. All producers and consumers use the actual family: live requests, persistent checkpoint selection, transient KV lookup/store, bootstrap, compaction and failure cleanup. Both DS4 families and already-normalized Qwen keys remain byte-for-byte unchanged. No checkpoint-format bump or unconditional invalidation.

The installed-model test prefills the actual legacy completed-history IDs (92 tokens), assigns their pre-change conversation tag, and checks both live and disk request paths. Both miss safely and generate the normalized 91-token prompt with identical complete state. The new live and disk continuations both reuse 93 tokens and have identical complete snapshot hashes. The unchanged #111 cache test controls compatible histories.

This is an MTPLX behavior correction; direct-mode nonempty reasoning-history rendering remains the separate #117 issue. Direct fixtures here retain that known boundary by using empty reasoning, while every reasoning mode includes decomposed historical reasoning.

Reproduce

Generate reference files using tools/issue116-reference.py PINNED_MTPLX MODEL_DIRECTORY OUTPUT_DIRECTORY in the isolated reference Python environment. Set DS4_QWEN38_ARTIFACTS and DS4_ISSUE116_ORACLE to the model and receipt directories. Build cargo test --all-features --bin ds4-server --no-run, copy its executable outside target, then run issue116_nfc_tokens_and_requests --ignored --nocapture --test-threads=1. Set DS4_ISSUE116_DIR to a fresh writable directory and run issue116_normalized_token_cache serially under test-supervisor 114688 60 120 --command, with MTPLX_CONTEXT_COPY=0. Ordinary tests also cover composition, rejected normalizer kinds, affected/unchanged key discrimination and prefix compatibility without model assets.

All four commit gates passed: formatting, warning-free all-target/all-feature Clippy, signed release app bundle, and 316 application + 8 tool tests (239 opt-in tests ignored). Final executable isolated controls all passed; logs and supervised resource receipts are included. The compatible #111 cache retains 95-token live/disk reuse with identical state.