97 lines
5.7 KiB
Markdown
97 lines
5.7 KiB
Markdown
# Qwen Unicode number boundaries (#111)
|
|
|
|
The Qwen-only splitter now implements the trained Qwen3 expression using the
|
|
existing Rust regex dependency. Its unsupported whitespace lookahead is handled
|
|
by retaining the last whitespace scalar for the following match. Both plain and
|
|
rendered tokenization dispatch to it. GLM still calls the unchanged DS4 GPT4
|
|
splitter with three-digit grouping; DeepSeek still calls unchanged JoyAI code.
|
|
Vocabulary, merges, Unicode tables, sampling and checkpoint formats are unchanged.
|
|
|
|
Oracle: MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`, installed
|
|
Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, MLX 0.32.2.
|
|
Artifact and environment fingerprints are in ../issue104-ar/environment.json.
|
|
DS4 `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`; exact executable hash in ds4.json.
|
|
Native baseline is `990abb4755f3aa38babfb819f84fe1ab9bb018da`.
|
|
|
|
## Isolated results and review
|
|
|
|
- All 1,112,064 Unicode scalars in letter/space context: 2,082,513 split pieces,
|
|
exactly the oracle digest `7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a`.
|
|
- All 1,911 oracle number characters (Nd/Nl/No) with spaces, neighboring letters
|
|
and combining marks, plus language/code/punctuation/contraction/newline/tail
|
|
whitespace controls: 5,752 exact split and BPE cases, both entry points, exact
|
|
original-byte decode round trips. Original #103 regression: zero mismatches.
|
|
- Full three-message requests and history+continuation IDs match the unmodified
|
|
pinned server in direct/low/medium/xhigh modes: 59/93/63/105 context tokens.
|
|
- GLM and DeepSeek: exact DS4 IDs on 19 shared controls, plain and rendered,
|
|
before and after. Existing three-turn DS4 chat fixtures also pass both before
|
|
and after. Baseline executable used the committed tokenizer source plus only
|
|
the test module; legacy-requests.json contains its actual rendered request IDs.
|
|
- Installed-model product owner: prefill/save/reopen actual old IDs (89 tokens),
|
|
align against corrected request (93 tokens), verify complete cache reset, then
|
|
generate cold. Corrected warm and disk-restored continuations each reuse 95
|
|
tokens and produce byte-identical complete snapshot hashes. Peak supervised
|
|
physical footprint about 77.2 GB; no memory or idle limit violation.
|
|
- Caller review: only the two Qwen dispatch arms change. Both feed the existing
|
|
byte BPE. Shared DS4 Unicode tables and three-digit rule remain unchanged.
|
|
`QwenExecutor::align_prompt` retains exact token-prefix selection and clears
|
|
incompatible state; compatible new checkpoints continue to load normally.
|
|
|
|
### Separate pre-existing contracts
|
|
|
|
This issue explicitly excludes Unicode normalization. qwen.json retains both
|
|
component `ids` and unmodified full-oracle `full_ids`. The component oracle removes
|
|
only NFC, leaving trained split/BPE intact. 5,751 cases match the full oracle too;
|
|
one decomposed combining-mark control differs because native NFC is missing.
|
|
normalization-gap.json preserves that exact difference, tracked as #116. This is
|
|
not a claim of complete Qwen tokenizer parity before #116.
|
|
|
|
Direct-mode history fixtures use empty reasoning, as generated in that mode.
|
|
Switching a reasoning conversation to direct mode exposes another existing
|
|
renderer difference: native preserves its historical reasoning while MTPLX drops
|
|
it. This is separately tracked as #117. Nonempty reasoning history is tested in
|
|
all three reasoning modes here.
|
|
|
|
## Reproduce
|
|
|
|
Generate fixtures with the unchanged model assets and isolated oracle environment:
|
|
|
|
```sh
|
|
local-eval-results/mtplx-reference-env-0.32.2/bin/python \
|
|
tools/issue111-reference.py /path/to/pinned/mtplx "$DS4_QWEN38_ARTIFACTS" /tmp/issue111-oracle \
|
|
--ds4 /path/to/ds4/ds4 --glm /path/to/GLM.gguf --deepseek /path/to/DeepSeek.gguf
|
|
cargo test --all-features --bin ds4-server issue111 --no-run
|
|
cargo build --bin test-supervisor
|
|
export DS4_ISSUE111_ORACLE="$PWD/docs/issue111-tokenizer"
|
|
export DS4_ISSUE111_DIR=/tmp/issue111-product
|
|
# Use the emitted test executable after compilation finishes.
|
|
target/debug/test-supervisor 4096 60 120 --command /path/to/test-binary \
|
|
engine::tokenizer::issue111 --ignored --nocapture --test-threads=1
|
|
# Run GPU work serially.
|
|
target/debug/test-supervisor 114688 60 120 --command /path/to/test-binary \
|
|
issue111_token_prefix_cache --ignored --nocapture --test-threads=1
|
|
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/deepseek-chat.json" \
|
|
/path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
|
|
DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/glm-chat.json" \
|
|
/path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
|
|
```
|
|
|
|
Set DS4_QWEN38_ARTIFACTS to the model directory. DS4 fixture model paths can be
|
|
updated for another machine without changing IDs. Baseline and native logs,
|
|
product receipt, exact oracle IDs and split digest are retained here. The small
|
|
`qwen_split_boundaries` regression also runs without model assets in ordinary tests.
|
|
This is an MTPLX compatibility fix; DS4/GLM behavior is intentionally preserved.
|
|
|
|
Commit gates passed: rustfmt check, all-target/all-feature Clippy with warnings
|
|
denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tests
|
|
(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
|
|
5 passed; the one 27B Quality-pack test skipped because that distinct model pack
|
|
is not installed. The installed Flash Next model was exercised above.
|
|
|
|
## Follow-up #116
|
|
|
|
NFC is now implemented. The component-stage test still checks every original
|
|
normalization-free split/BPE ID; it additionally checks the full tokenizer against
|
|
all `full_ids` and the normalized decode, closing the one-case gap documented
|
|
above. See `../issue116-nfc/` for full normalization and cache migration receipts.
|