Match Qwen Unicode number token boundaries

docs/issue111-tokenizer/README.md

@@ -0,0 +1,89 @@
+# Qwen Unicode number boundaries (#111)
+
+The Qwen-only splitter now implements the trained Qwen3 expression using the
+existing Rust regex dependency. Its unsupported whitespace lookahead is handled
+by retaining the last whitespace scalar for the following match. Both plain and
+rendered tokenization dispatch to it. GLM still calls the unchanged DS4 GPT4
+splitter with three-digit grouping; DeepSeek still calls unchanged JoyAI code.
+Vocabulary, merges, Unicode tables, sampling and checkpoint formats are unchanged.
+
+Oracle: MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`, installed
+Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, MLX 0.32.2.
+Artifact and environment fingerprints are in ../issue104-ar/environment.json.
+DS4 `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`; exact executable hash in ds4.json.
+Native baseline is `990abb4755f3aa38babfb819f84fe1ab9bb018da`.
+
+## Isolated results and review
+
+- All 1,112,064 Unicode scalars in letter/space context: 2,082,513 split pieces,
+  exactly the oracle digest `7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a`.
+- All 1,911 oracle number characters (Nd/Nl/No) with spaces, neighboring letters
+  and combining marks, plus language/code/punctuation/contraction/newline/tail
+  whitespace controls: 5,752 exact split and BPE cases, both entry points, exact
+  original-byte decode round trips. Original #103 regression: zero mismatches.
+- Full three-message requests and history+continuation IDs match the unmodified
+  pinned server in direct/low/medium/xhigh modes: 59/93/63/105 context tokens.
+- GLM and DeepSeek: exact DS4 IDs on 19 shared controls, plain and rendered,
+  before and after. Existing three-turn DS4 chat fixtures also pass both before
+  and after. Baseline executable used the committed tokenizer source plus only
+  the test module; legacy-requests.json contains its actual rendered request IDs.
+- Installed-model product owner: prefill/save/reopen actual old IDs (89 tokens),
+  align against corrected request (93 tokens), verify complete cache reset, then
+  generate cold. Corrected warm and disk-restored continuations each reuse 95
+  tokens and produce byte-identical complete snapshot hashes. Peak supervised
+  physical footprint about 77.2 GB; no memory or idle limit violation.
+- Caller review: only the two Qwen dispatch arms change. Both feed the existing
+  byte BPE. Shared DS4 Unicode tables and three-digit rule remain unchanged.
+  `QwenExecutor::align_prompt` retains exact token-prefix selection and clears
+  incompatible state; compatible new checkpoints continue to load normally.
+
+### Separate pre-existing contracts
+
+This issue explicitly excludes Unicode normalization. qwen.json retains both
+component `ids` and unmodified full-oracle `full_ids`. The component oracle removes
+only NFC, leaving trained split/BPE intact. 5,751 cases match the full oracle too;
+one decomposed combining-mark control differs because native NFC is missing.
+normalization-gap.json preserves that exact difference, tracked as #116. This is
+not a claim of complete Qwen tokenizer parity before #116.
+
+Direct-mode history fixtures use empty reasoning, as generated in that mode.
+Switching a reasoning conversation to direct mode exposes another existing
+renderer difference: native preserves its historical reasoning while MTPLX drops
+it. This is separately tracked as #117. Nonempty reasoning history is tested in
+all three reasoning modes here.
+
+## Reproduce
+
+Generate fixtures with the unchanged model assets and isolated oracle environment:
+
+```sh
+local-eval-results/mtplx-reference-env-0.32.2/bin/python \
+  tools/issue111-reference.py /path/to/pinned/mtplx "$DS4_QWEN38_ARTIFACTS" /tmp/issue111-oracle \
+  --ds4 /path/to/ds4/ds4 --glm /path/to/GLM.gguf --deepseek /path/to/DeepSeek.gguf
+cargo test --all-features --bin ds4-server issue111 --no-run
+cargo build --bin test-supervisor
+export DS4_ISSUE111_ORACLE="$PWD/docs/issue111-tokenizer"
+export DS4_ISSUE111_DIR=/tmp/issue111-product
+# Use the emitted test executable after compilation finishes.
+target/debug/test-supervisor 4096 60 120 --command /path/to/test-binary \
+  engine::tokenizer::issue111 --ignored --nocapture --test-threads=1
+# Run GPU work serially.
+target/debug/test-supervisor 114688 60 120 --command /path/to/test-binary \
+  issue111_token_prefix_cache --ignored --nocapture --test-threads=1
+DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/deepseek-chat.json" \
+  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
+DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/glm-chat.json" \
+  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
+```
+
+Set DS4_QWEN38_ARTIFACTS to the model directory. DS4 fixture model paths can be
+updated for another machine without changing IDs. Baseline and native logs,
+product receipt, exact oracle IDs and split digest are retained here. The small
+`qwen_split_boundaries` regression also runs without model assets in ordinary tests.
+This is an MTPLX compatibility fix; DS4/GLM behavior is intentionally preserved.
+
+Commit gates passed: rustfmt check, all-target/all-feature Clippy with warnings
+denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tests
+(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
+5 passed; the one 27B Quality-pack test skipped because that distinct model pack
+is not installed. The installed Flash Next model was exercised above.