Apply Qwen NFC normalization before token splitting

docs/issue116-nfc/README.md

@@ -0,0 +1,32 @@
+# Qwen NFC normalization (#116)
+
+Qwen requires the installed artifact's `NFC` normalizer at load time and normalizes each ordinary text span before the trained Qwen split/BPE. Rendered added tokens are extracted first and remain literal boundaries. Plain text keeps its existing literal-marker semantics. DeepSeek JoyAI and GLM GPT4 token processing are unchanged.
+
+The Rust dependency `unicode-normalization-alignments 0.1.12` is the same normalization implementation used by [tokenizers 0.22.2](https://github.com/huggingface/tokenizers/blob/v0.22.2/tokenizers/Cargo.toml), the pinned oracle's installed tokenizer. Its Unicode tables are version 9.0.0; substituting newer tables would change the oracle contract. Python is only used to generate independent reference receipts, never in production.
+
+## Reference and isolated verification
+
+Pinned MTPLX: `9882703f3105363ddc37eca9f97aa09a1d387112`; Bare-Speed artifact revision: `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`. Tokenizer SHA-256: `0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3`. DS4 revision: `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`.
+
+- All 1,112,064 Unicode scalars in `a + scalar + acute + dot-below + separator` context match the oracle normalization digest `1905b5f60f629850aa35dfc8ccb11fa6ae4d1fb0c79efd6774d36ca8e3e64eab`.
+- 13,281 exact normalized-text/plain-ID/rendered-ID/decode cases include canonical decompositions, reordered and leading marks, long combining sequences, all 11,172 Hangul syllables, already-normalized text, compatibility characters that NFC must preserve, and every added-token boundary. Python Unicode data only enumerates inputs; expected results come from the installed oracle normalizer/tokenizer.
+- Eight three-message requests cover direct/low/medium/xhigh, with and without a decomposed system prompt. Full encoding and history plus continuation match the pinned server exactly. Context counts are 52/62, 85/91, 55/65, and 97/103 respectively.
+- The original #116 text now emits `[52033,64,18135,220,52033,220,12256,100,52033,65]` and decodes to the oracle's composed form.
+- All prior #111 component split/BPE expectations remain checked independently before normalization. The full native tokenizer now matches all 5,752 full-oracle cases, including the previously excluded NFC difference. All-number-category and exhaustive split controls remain exact.
+- GLM and DeepSeek each retain exact DS4 plain/rendered IDs on all 19 shared controls. Existing DS4 chat render controls also run unchanged.
+
+## Cache and caller review
+
+Every plain, rendered, full conversation, history, continuation, system-prompt and context-count path reaches the Qwen normalizer. Generation output bytes are not re-tokenized during a compatible completed-history continuation; its generated KV remains exact.
+
+Token-prefix comparison alone was insufficient: a completed-conversation tag can authorize appending to old token IDs without rendering history. Qwen conversation keys now include a per-field NFC discriminator only when that field changes under the pinned normalizer. Raw histories remain distinct, appended-history keys remain prefixes, and pre-NFC affected keys cannot match new keys. All producers and consumers use the actual family: live requests, persistent checkpoint selection, transient KV lookup/store, bootstrap, compaction and failure cleanup. Both DS4 families and already-normalized Qwen keys remain byte-for-byte unchanged. No checkpoint-format bump or unconditional invalidation.
+
+The installed-model test prefills the actual legacy completed-history IDs (92 tokens), assigns their pre-change conversation tag, and checks both live and disk request paths. Both miss safely and generate the normalized 91-token prompt with identical complete state. The new live and disk continuations both reuse 93 tokens and have identical complete snapshot hashes. The unchanged #111 cache test controls compatible histories.
+
+This is an MTPLX behavior correction; direct-mode nonempty reasoning-history rendering remains the separate #117 issue. Direct fixtures here retain that known boundary by using empty reasoning, while every reasoning mode includes decomposed historical reasoning.
+
+## Reproduce
+
+Generate reference files using `tools/issue116-reference.py PINNED_MTPLX MODEL_DIRECTORY OUTPUT_DIRECTORY` in the isolated reference Python environment. Set `DS4_QWEN38_ARTIFACTS` and `DS4_ISSUE116_ORACLE` to the model and receipt directories. Build `cargo test --all-features --bin ds4-server --no-run`, copy its executable outside target, then run `issue116_nfc_tokens_and_requests --ignored --nocapture --test-threads=1`. Set `DS4_ISSUE116_DIR` to a fresh writable directory and run `issue116_normalized_token_cache` serially under `test-supervisor 114688 60 120 --command`, with `MTPLX_CONTEXT_COPY=0`. Ordinary tests also cover composition, rejected normalizer kinds, affected/unchanged key discrimination and prefix compatibility without model assets.
+
+All four commit gates passed: formatting, warning-free all-target/all-feature Clippy, signed release app bundle, and 316 application + 8 tool tests (239 opt-in tests ignored). Final executable isolated controls all passed; logs and supervised resource receipts are included. The compatible #111 cache retains 95-token live/disk reuse with identical state.