# Qwen Unicode number boundaries (#111)

The Qwen-only splitter now implements the trained Qwen3 expression using the
existing Rust regex dependency. Its unsupported whitespace lookahead is handled
by retaining the last whitespace scalar for the following match. Both plain and
rendered tokenization dispatch to it. GLM still calls the unchanged DS4 GPT4
splitter with three-digit grouping; DeepSeek still calls unchanged JoyAI code.
Vocabulary, merges, Unicode tables, sampling and checkpoint formats are unchanged.

Oracle: MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`, installed
Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, MLX 0.32.2.
Artifact and environment fingerprints are in ../issue104-ar/environment.json.
DS4 `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`; exact executable hash in ds4.json.
Native baseline is `990abb4755f3aa38babfb819f84fe1ab9bb018da`.

## Isolated results and review

- All 1,112,064 Unicode scalars in letter/space context: 2,082,513 split pieces,
  exactly the oracle digest `7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a`.
- All 1,911 oracle number characters (Nd/Nl/No) with spaces, neighboring letters
  and combining marks, plus language/code/punctuation/contraction/newline/tail
  whitespace controls: 5,752 exact split and BPE cases, both entry points, exact
  original-byte decode round trips. Original #103 regression: zero mismatches.
- Full three-message requests and history+continuation IDs match the unmodified
  pinned server in direct/low/medium/xhigh modes: 59/93/63/105 context tokens.
- GLM and DeepSeek: exact DS4 IDs on 19 shared controls, plain and rendered,
  before and after. Existing three-turn DS4 chat fixtures also pass both before
  and after. Baseline executable used the committed tokenizer source plus only
  the test module; legacy-requests.json contains its actual rendered request IDs.
- Installed-model product owner: prefill/save/reopen actual old IDs (89 tokens),
  align against corrected request (93 tokens), verify complete cache reset, then
  generate cold. Corrected warm and disk-restored continuations each reuse 95
  tokens and produce byte-identical complete snapshot hashes. Peak supervised
  physical footprint about 77.2 GB; no memory or idle limit violation.
- Caller review: only the two Qwen dispatch arms change. Both feed the existing
  byte BPE. Shared DS4 Unicode tables and three-digit rule remain unchanged.
  `QwenExecutor::align_prompt` retains exact token-prefix selection and clears
  incompatible state; compatible new checkpoints continue to load normally.

### Separate pre-existing contracts

This issue explicitly excludes Unicode normalization. qwen.json retains both
component `ids` and unmodified full-oracle `full_ids`. The component oracle removes
only NFC, leaving trained split/BPE intact. 5,751 cases match the full oracle too;
one decomposed combining-mark control differs because native NFC is missing.
normalization-gap.json preserves that exact difference, tracked as #116. This is
not a claim of complete Qwen tokenizer parity before #116.

Direct-mode history fixtures use empty reasoning, as generated in that mode.
Switching a reasoning conversation to direct mode exposes another existing
renderer difference: native preserves its historical reasoning while MTPLX drops
it. This is separately tracked as #117. Nonempty reasoning history is tested in
all three reasoning modes here.

## Reproduce

Generate fixtures with the unchanged model assets and isolated oracle environment:

```sh
local-eval-results/mtplx-reference-env-0.32.2/bin/python \
  tools/issue111-reference.py /path/to/pinned/mtplx "$DS4_QWEN38_ARTIFACTS" /tmp/issue111-oracle \
  --ds4 /path/to/ds4/ds4 --glm /path/to/GLM.gguf --deepseek /path/to/DeepSeek.gguf
cargo test --all-features --bin ds4-server issue111 --no-run
cargo build --bin test-supervisor
export DS4_ISSUE111_ORACLE="$PWD/docs/issue111-tokenizer"
export DS4_ISSUE111_DIR=/tmp/issue111-product
# Use the emitted test executable after compilation finishes.
target/debug/test-supervisor 4096 60 120 --command /path/to/test-binary \
  engine::tokenizer::issue111 --ignored --nocapture --test-threads=1
# Run GPU work serially.
target/debug/test-supervisor 114688 60 120 --command /path/to/test-binary \
  issue111_token_prefix_cache --ignored --nocapture --test-threads=1
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/deepseek-chat.json" \
  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/glm-chat.json" \
  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
```

Set DS4_QWEN38_ARTIFACTS to the model directory. DS4 fixture model paths can be
updated for another machine without changing IDs. Baseline and native logs,
product receipt, exact oracle IDs and split digest are retained here. The small
`qwen_split_boundaries` regression also runs without model assets in ordinary tests.
This is an MTPLX compatibility fix; DS4/GLM behavior is intentionally preserved.

Commit gates passed: rustfmt check, all-target/all-feature Clippy with warnings
denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tests
(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
5 passed; the one 27B Quality-pack test skipped because that distinct model pack
is not installed. The installed Flash Next model was exercised above.

## Follow-up #116

NFC is now implemented. The component-stage test still checks every original
normalization-free split/BPE ID; it additionally checks the full tokenizer against
all `full_ids` and the normalized decode, closing the one-case gap documented
above. See `../issue116-nfc/` for full normalization and cache migration receipts.