The Qwen-only splitter now implements the trained Qwen3 expression using the existing Rust regex dependency. Its unsupported whitespace lookahead is handled by retaining the last whitespace scalar for the following match. Both plain and rendered tokenization dispatch to it. GLM still calls the unchanged DS4 GPT4 splitter with three-digit grouping; DeepSeek still calls unchanged JoyAI code. Vocabulary, merges, Unicode tables, sampling and checkpoint formats are unchanged.

Oracle: MTPLX 2.12.2 9882703f3105363ddc37eca9f97aa09a1d387112, installed Bare-Speed artifact 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7, MLX 0.32.2. Artifact and environment fingerprints are in ../issue104-ar/environment.json. DS4 ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748; exact executable hash in ds4.json. Native baseline is 990abb4755f3aa38babfb819f84fe1ab9bb018da.

Isolated results and review

All 1,112,064 Unicode scalars in letter/space context: 2,082,513 split pieces, exactly the oracle digest 7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a.
All 1,911 oracle number characters (Nd/Nl/No) with spaces, neighboring letters and combining marks, plus language/code/punctuation/contraction/newline/tail whitespace controls: 5,752 exact split and BPE cases, both entry points, exact original-byte decode round trips. Original #103 regression: zero mismatches.
Full three-message requests and history+continuation IDs match the unmodified pinned server in direct/low/medium/xhigh modes: 59/93/63/105 context tokens.
GLM and DeepSeek: exact DS4 IDs on 19 shared controls, plain and rendered, before and after. Existing three-turn DS4 chat fixtures also pass both before and after. Baseline executable used the committed tokenizer source plus only the test module; legacy-requests.json contains its actual rendered request IDs.
Installed-model product owner: prefill/save/reopen actual old IDs (89 tokens), align against corrected request (93 tokens), verify complete cache reset, then generate cold. Corrected warm and disk-restored continuations each reuse 95 tokens and produce byte-identical complete snapshot hashes. Peak supervised physical footprint about 77.2 GB; no memory or idle limit violation.
Caller review: only the two Qwen dispatch arms change. Both feed the existing byte BPE. Shared DS4 Unicode tables and three-digit rule remain unchanged. QwenExecutor::align_prompt retains exact token-prefix selection and clears incompatible state; compatible new checkpoints continue to load normally.

Separate pre-existing contracts

This issue explicitly excludes Unicode normalization. qwen.json retains both component ids and unmodified full-oracle full_ids. The component oracle removes only NFC, leaving trained split/BPE intact. 5,751 cases match the full oracle too; one decomposed combining-mark control differs because native NFC is missing. normalization-gap.json preserves that exact difference, tracked as #116. This is not a claim of complete Qwen tokenizer parity before #116.

Direct-mode history fixtures use empty reasoning, as generated in that mode. Switching a reasoning conversation to direct mode exposes another existing renderer difference: native preserves its historical reasoning while MTPLX drops it. This is separately tracked as #117. Nonempty reasoning history is tested in all three reasoning modes here.

Reproduce

Generate fixtures with the unchanged model assets and isolated oracle environment:

local-eval-results/mtplx-reference-env-0.32.2/bin/python \
  tools/issue111-reference.py /path/to/pinned/mtplx "$DS4_QWEN38_ARTIFACTS" /tmp/issue111-oracle \
  --ds4 /path/to/ds4/ds4 --glm /path/to/GLM.gguf --deepseek /path/to/DeepSeek.gguf
cargo test --all-features --bin ds4-server issue111 --no-run
cargo build --bin test-supervisor
export DS4_ISSUE111_ORACLE="$PWD/docs/issue111-tokenizer"
export DS4_ISSUE111_DIR=/tmp/issue111-product
# Use the emitted test executable after compilation finishes.
target/debug/test-supervisor 4096 60 120 --command /path/to/test-binary \
  engine::tokenizer::issue111 --ignored --nocapture --test-threads=1
# Run GPU work serially.
target/debug/test-supervisor 114688 60 120 --command /path/to/test-binary \
  issue111_token_prefix_cache --ignored --nocapture --test-threads=1
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/deepseek-chat.json" \
  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/glm-chat.json" \
  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture

Set DS4_QWEN38_ARTIFACTS to the model directory. DS4 fixture model paths can be updated for another machine without changing IDs. Baseline and native logs, product receipt, exact oracle IDs and split digest are retained here. The small qwen_split_boundaries regression also runs without model assets in ordinary tests. This is an MTPLX compatibility fix; DS4/GLM behavior is intentionally preserved.

Commit gates passed: rustfmt check, all-target/all-feature Clippy with warnings denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tests (228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests: 5 passed; the one 27B Quality-pack test skipped because that distinct model pack is not installed. The installed Flash Next model was exercised above.

Follow-up #116

NFC is now implemented. The component-stage test still checks every original normalization-free split/BPE ID; it additionally checks the full tokenizer against all full_ids and the normalized decode, closing the one-case gap documented above. See ../issue116-nfc/ for full normalization and cache migration receipts.