Publication status: analysis commit dcc4c516f0e030566c67472f8f1be0baa21ab629 is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under /Users/gb/Projects/DS4Server/docs/issue103-analysis/ and the committed test/tool paths, not yet at the remote file URLs.
Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the independent review and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
Qwen 3.8 Flash Next: behavioral/numerical parity with the pinned MTPLX 2.12.2 revision 9882703f3105363ddc37eca9f97aa09a1d387112, including token processing, RNG/sampling, context accounting, inference and cache/session state.
DSv4 Flash and GLM 5.3 Flash: preserve DS4 behavior. A Qwen fix must not change their token IDs, sampler/RNG sequences, numerical inference, context accounting or cache behavior.
Keep changes in the Qwen family/phase/dispatch where applicable. Before changing any shared helper, trace every caller; preserve DS4-backed callers exactly and provide DS4 regression controls. A helper shared by Qwen and GLM must not silently impose MTPLX semantics on GLM. Do not globally change the DS4 sampler, shared RNG, Unicode tables, sigmoid/fast-math rules, or non-Qwen checkpoint formats for this ticket.
This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
Broken behavior and evidence
Confirmed native Qwen tokenization mismatch. The shared handwritten Unicode tables misclassify trained Qwen3 number boundaries in two ways: ①(U+2460) is explicitly classified as punctuation by unicode_punctuation range 0x2460..=0x24ff; Ⅷ/²/𝟘 fall into the non-ASCII letter fallback because unicode_number omits their categories/ranges. The former joins a leading space via the punctuation branch; the latter via a letter span. Both differ from the trained Qwen3 Split contract, which uses letter/mark spans and single Unicode number-category matches.
The current shared tokenize_gpt4/char_info code is also used by GLM with max_digits=3; Qwen uses max_digits=1, including tokenize and tokenize_plain/rendered paths. The GLM tables mirror DS4's glm4_unicode_number/glm4_char_at, so globally correcting them to Unicode/Qwen semantics would break the required GLM/DS4 token IDs. DeepSeek uses its separate JoyAI path and remains a DS4 regression control.
Exact unchanged Qwen assets: " ①" actual[2765,239,254], oracle[220,46701,254]; " Ⅷ" actual[2765,88608], oracle[220,68086,100]; " ²" actual[2071,110], oracle[220,28495]; " 𝟘" actual[78449,253,246], oracle[220,54362,253,246]. Five mismatches including mixed-script sample; Hindi/Thai/vowelled Arabic/English/code/Chinese/mixed punctuation controls match. This is a native Qwen category bug, not a port of the transformers loader implementation.
Native implementation scope
Implement the trained artifact's Qwen3 pretokenizer contract exclusively for ModelFamily::Qwen in both plain and rendered dispatch. Preserve exact vocabulary/merges, letter+mark spans and single \p{N} number matching; no Unicode normalization. Reuse existing regex dependency for supported Unicode categories where practical; account explicitly for unsupported lookahead (for example \s+(?!\S)) using minimal native handling rather than accepting a different split law or adding a crate by default.
Do not change shared char_info/unicode_number/unicode_punctuation/tokenize_gpt4 behavior observed by ModelFamily::Glm, its three-digit rule, or DeepSeek's JoyAI path. A Qwen-specific split/classification branch is required wherever shared behavior differs. Protect both entry points (tokenize, tokenize_plain reached by tokenize_rendered). Preserve safe token-exact cache matching: changed Qwen IDs cause a cache miss against old prefixes, not an unsafe warm restore. No checkpoint-format bump is justified by this tokenizer fix alone; do not change GLM tokenization to make a Unicode-standard test pass.
Acceptance
Retain exact Qwen oracle IDs for all four minimal numerals/mixed sample, combining-mark scripts and existing language/code/punctuation controls. Cover Unicode Nd/Nl/No boundaries with spaces, neighboring marks/letters, contractions/newlines/whitespace-tail behavior and both plain/rendered entry points. Compare Qwen full rendered request IDs/context lengths against pinned MTPLX with unchanged assets; verify decode roundtrip and safe old-prefix miss/new-prefix warm reuse.
Mandatory DS4 controls: GLM 5.3 Flash exact token IDs for the same numeral examples plus standard DS4 chat fixtures must remain identical before/after and to DS4; DeepSeek/DSv4 Flash JoyAI/plain/rendered fixtures must likewise remain unchanged. Use exact IDs, not only decoded text, and prove the Qwen correction cannot route GLM/DeepSeek through the Qwen split. Existing shared tables are retained for their DS4-backed callers. No checkpoint invalidation or unrelated family behavior change.
Severity
Medium: deterministic Qwen prompt-ID/context mismatch. Existing token-exact restore selection yields safe misses on changed IDs; no corrupt-checkpoint consequence is claimed.
Shared evidence and reproduction
Analysis parent: #103. Affected DS4Server revision: cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983. Compared MTPLX old e652d55e2652137a4abcf1312357abbf3eb9d692 (2.10.2) with refreshed 9882703f3105363ddc37eca9f97aa09a1d387112 (2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7; unchanged hashes, compiler/OS/hardware and effective options in environment.json. Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.
Durable report and instructions, native test, oracle probe. The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
cargo test --all-features --bin ds4-server issue103_tokenizer_receipts --no-run
cargo build --bin test-supervisor
exportDS4_QWEN38_ARTIFACTS='/path/to/unchanged/qwen3.8-flash-next'exportDS4_ISSUE103_ORACLE="$PWD/docs/issue103-analysis/latest-oracle.jsonl"# Resolve the emitted test executable; do not pass cargo to the watchdog.
target/debug/test-supervisor 409660120 --command /path/to/ds4_server-test-binary issue103_tokenizer_receipts --ignored --nocapture --test-threads=1
Immutable upstream change and latest test. The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
Validation and review boundary
Implement production behavior in Rust with existing Metal/Objective-C bridge only; Python remains an isolated oracle.
Run cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, make bundle, cargo test --all-features before every future commit, plus the owned native/oracle and product regressions above.
Implementation awaits the user's independent-agent review and subsequent instruction. Creating this issue or closing #103 does not authorize a fix. Review the corrected model-specific oracle, active-route applicability and state ownership first; preserve DS4-backed families.
No arbitrary dependencies on sibling issues: coordinate shared source changes where needed; each issue owns its concrete failure and acceptance.
Publication status: analysis commit `dcc4c516f0e030566c67472f8f1be0baa21ab629` is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under `/Users/gb/Projects/DS4Server/docs/issue103-analysis/` and the committed test/tool paths, not yet at the remote file URLs.
## Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the [independent review](https://git.rfc1437.de/hugo/DS4Server/issues/111#issuecomment-3074) and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
## Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
- Qwen 3.8 Flash Next: behavioral/numerical parity with the pinned MTPLX 2.12.2 revision `9882703f3105363ddc37eca9f97aa09a1d387112`, including token processing, RNG/sampling, context accounting, inference and cache/session state.
- DSv4 Flash and GLM 5.3 Flash: preserve DS4 behavior. A Qwen fix must not change their token IDs, sampler/RNG sequences, numerical inference, context accounting or cache behavior.
- Keep changes in the Qwen family/phase/dispatch where applicable. Before changing any shared helper, trace every caller; preserve DS4-backed callers exactly and provide DS4 regression controls. A helper shared by Qwen and GLM must not silently impose MTPLX semantics on GLM. Do not globally change the DS4 sampler, shared RNG, Unicode tables, sigmoid/fast-math rules, or non-Qwen checkpoint formats for this ticket.
This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
## Broken behavior and evidence
Confirmed native Qwen tokenization mismatch. The shared handwritten Unicode tables misclassify trained Qwen3 number boundaries in two ways: ①(U+2460) is explicitly classified as punctuation by unicode_punctuation range 0x2460..=0x24ff; Ⅷ/²/𝟘 fall into the non-ASCII letter fallback because unicode_number omits their categories/ranges. The former joins a leading space via the punctuation branch; the latter via a letter span. Both differ from the trained Qwen3 Split contract, which uses letter/mark spans and single Unicode number-category matches.
The current shared `tokenize_gpt4`/char_info code is also used by GLM with max_digits=3; Qwen uses max_digits=1, including tokenize and tokenize_plain/rendered paths. The GLM tables mirror DS4's glm4_unicode_number/glm4_char_at, so globally correcting them to Unicode/Qwen semantics would break the required GLM/DS4 token IDs. DeepSeek uses its separate JoyAI path and remains a DS4 regression control.
Exact unchanged Qwen assets: " ①" actual[2765,239,254], oracle[220,46701,254]; " Ⅷ" actual[2765,88608], oracle[220,68086,100]; " ²" actual[2071,110], oracle[220,28495]; " 𝟘" actual[78449,253,246], oracle[220,54362,253,246]. Five mismatches including mixed-script sample; Hindi/Thai/vowelled Arabic/English/code/Chinese/mixed punctuation controls match. This is a native Qwen category bug, not a port of the transformers loader implementation.
## Native implementation scope
Implement the trained artifact's Qwen3 pretokenizer contract exclusively for ModelFamily::Qwen in both plain and rendered dispatch. Preserve exact vocabulary/merges, letter+mark spans and single \p{N} number matching; no Unicode normalization. Reuse existing regex dependency for supported Unicode categories where practical; account explicitly for unsupported lookahead (for example \s+(?!\S)) using minimal native handling rather than accepting a different split law or adding a crate by default.
Do not change shared char_info/unicode_number/unicode_punctuation/tokenize_gpt4 behavior observed by ModelFamily::Glm, its three-digit rule, or DeepSeek's JoyAI path. A Qwen-specific split/classification branch is required wherever shared behavior differs. Protect both entry points (`tokenize`, `tokenize_plain` reached by tokenize_rendered). Preserve safe token-exact cache matching: changed Qwen IDs cause a cache miss against old prefixes, not an unsafe warm restore. No checkpoint-format bump is justified by this tokenizer fix alone; do not change GLM tokenization to make a Unicode-standard test pass.
## Acceptance
Retain exact Qwen oracle IDs for all four minimal numerals/mixed sample, combining-mark scripts and existing language/code/punctuation controls. Cover Unicode Nd/Nl/No boundaries with spaces, neighboring marks/letters, contractions/newlines/whitespace-tail behavior and both plain/rendered entry points. Compare Qwen full rendered request IDs/context lengths against pinned MTPLX with unchanged assets; verify decode roundtrip and safe old-prefix miss/new-prefix warm reuse.
Mandatory DS4 controls: GLM 5.3 Flash exact token IDs for the same numeral examples plus standard DS4 chat fixtures must remain identical before/after and to DS4; DeepSeek/DSv4 Flash JoyAI/plain/rendered fixtures must likewise remain unchanged. Use exact IDs, not only decoded text, and prove the Qwen correction cannot route GLM/DeepSeek through the Qwen split. Existing shared tables are retained for their DS4-backed callers. No checkpoint invalidation or unrelated family behavior change.
## Severity
Medium: deterministic Qwen prompt-ID/context mismatch. Existing token-exact restore selection yields safe misses on changed IDs; no corrupt-checkpoint consequence is claimed.
## Shared evidence and reproduction
Analysis parent: [#103](https://git.rfc1437.de/hugo/DS4Server/issues/103). Affected DS4Server revision: `cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983`. Compared MTPLX old `e652d55e2652137a4abcf1312357abbf3eb9d692` (2.10.2) with refreshed `9882703f3105363ddc37eca9f97aa09a1d387112` (2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`; unchanged hashes, compiler/OS/hardware and effective options in [environment.json](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/docs/issue103-analysis/environment.json). Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.
Durable [report and instructions](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/docs/issue103-analysis/README.md), [native test](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/src/engine/metal/qwen_mtplx/issue103_analysis.rs), [oracle probe](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/tools/issue103-reference.py). The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
```sh
cargo test --all-features --bin ds4-server issue103_tokenizer_receipts --no-run
cargo build --bin test-supervisor
export DS4_QWEN38_ARTIFACTS='/path/to/unchanged/qwen3.8-flash-next'
export DS4_ISSUE103_ORACLE="$PWD/docs/issue103-analysis/latest-oracle.jsonl"
# Resolve the emitted test executable; do not pass cargo to the watchdog.
target/debug/test-supervisor 4096 60 120 --command /path/to/ds4_server-test-binary issue103_tokenizer_receipts --ignored --nocapture --test-threads=1
```
Immutable upstream [change](https://github.com/youssofal/mtplx/commit/54b1063a6bd98e077eea2421bfed7e634cf5f459) and [latest test](https://github.com/youssofal/mtplx/blob/9882703f3105363ddc37eca9f97aa09a1d387112/tests/test_tokenizer_pretokenizer_repair.py). The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
## Validation and review boundary
- Implement production behavior in Rust with existing Metal/Objective-C bridge only; Python remains an isolated oracle.
- Run `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `make bundle`, `cargo test --all-features` before every future commit, plus the owned native/oracle and product regressions above.
- Implementation awaits the user's independent-agent review and subsequent instruction. Creating this issue or closing #103 does not authorize a fix. Review the corrected model-specific oracle, active-route applicability and state ownership first; preserve DS4-backed families.
- No arbitrary dependencies on sibling issues: coordinate shared source changes where needed; each issue owns its concrete failure and acceptance.
hugo
added the bug label 2026-10-03 09:20:33 +00:00
Reviewed against DS4Server dcc4c51 and MTPLX 2.12.2 9882703f3105363ddc37eca9f97aa09a1d387112. Note: evidence commit dcc4c51 is still unpublished (main is ahead of origin/main by one commit), so the repository links in this issue currently resolve to nothing.
Defect claim: confirmed in substance; one mechanism detail is wrong; scope must be family-restricted.
Mechanism detail. Not all mismatches come from the letter heuristic. ① (U+2460) is classified as punctuation by unicode_punctuation (range 0x2460..=0x24ff), so " ①" merges through the ?[^\s\p{L}\p{N}]+ branch. Ⅷ, ², 𝟘 do fall into the non-ASCII letter fallback. Both are consequences of the hand-written unicode_number/unicode_punctuation tables, and the token IDs are wrong either way.
Scope must be Qwen-only (important).char_info and tokenize_gpt4 are shared with GLM (tokenizer.rs:328-329, ModelFamily::Glm => tokenize_gpt4(.., 3)), and our tables mirror DS4's glm4_unicode_number/glm4_char_at (ds4.c). Per AGENTS.md, DS4 is the behavioral oracle for those families. Changing the shared classification globally would break GLM parity with DS4. The fix must apply only to ModelFamily::Qwen, with GLM/DeepSeek tokenization covered as unchanged regression controls.
Implementation hint. The artifact's own tokenizer.json carries the Qwen3 Split regex ([\p{L}\p{M}]+, single \p{N}). The existing regex dependency supports these Unicode classes; only the \s+(?!\S) lookahead needs manual handling. This would avoid maintaining another hand table.
Severity Medium is reasonable. Changed IDs only cause cache misses against old prefixes (safe), so no checkpoint invalidation is needed.
## Independent review (2026-10-03)
Reviewed against DS4Server `dcc4c51` and MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`. Note: evidence commit `dcc4c51` is still unpublished (`main` is ahead of `origin/main` by one commit), so the repository links in this issue currently resolve to nothing.
**Defect claim: confirmed in substance; one mechanism detail is wrong; scope must be family-restricted.**
1. **Mechanism detail.** Not all mismatches come from the letter heuristic. ① (U+2460) is classified as *punctuation* by `unicode_punctuation` (range `0x2460..=0x24ff`), so " ①" merges through the ` ?[^\s\p{L}\p{N}]+` branch. Ⅷ, ², 𝟘 do fall into the non-ASCII letter fallback. Both are consequences of the hand-written `unicode_number`/`unicode_punctuation` tables, and the token IDs are wrong either way.
2. **Scope must be Qwen-only (important).** `char_info` and `tokenize_gpt4` are shared with GLM (tokenizer.rs:328-329, `ModelFamily::Glm => tokenize_gpt4(.., 3)`), and our tables mirror DS4's `glm4_unicode_number`/`glm4_char_at` (ds4.c). Per AGENTS.md, DS4 is the behavioral oracle for those families. Changing the shared classification globally would break GLM parity with DS4. The fix must apply only to `ModelFamily::Qwen`, with GLM/DeepSeek tokenization covered as unchanged regression controls.
3. **Implementation hint.** The artifact's own `tokenizer.json` carries the Qwen3 Split regex (`[\p{L}\p{M}]+`, single `\p{N}`). The existing `regex` dependency supports these Unicode classes; only the `\s+(?!\S)` lookahead needs manual handling. This would avoid maintaining another hand table.
Severity Medium is reasonable. Changed IDs only cause cache misses against old prefixes (safe), so no checkpoint invalidation is needed.
Description corrected after independent review (2026-10-03)
Updated the issue body in response to your review. Changes:
Corrected the mechanism:① is punctuation;Ⅷ/²/𝟘 take the letter fallback.
Restricted the correction explicitly to ModelFamily::Qwen at both plain and rendered tokenizer entry points.
Protected shared GLM/DS4 Unicode tables, GLM digit grouping and DeepSeek JoyAI behavior with exact-ID regression controls.
Added existing-regex/lookahead guidance without normalization/new asset dependency, and documented safe old-prefix misses/no checkpoint bump for this issue alone.
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the two classification mechanisms, exact artifact regex semantics, every plain/rendered family dispatch, Qwen oracle IDs and unchanged GLM/DeepSeek IDs against DS4. Reject any proposal that edits shared Unicode tables globally.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit dcc4c51 remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.
## Description corrected after independent review (2026-10-03)
Updated the issue body in response to [your review](https://git.rfc1437.de/hugo/DS4Server/issues/111#issuecomment-3074). Changes:
- Corrected the mechanism:① is punctuation;Ⅷ/²/𝟘 take the letter fallback.
- Restricted the correction explicitly to ModelFamily::Qwen at both plain and rendered tokenizer entry points.
- Protected shared GLM/DS4 Unicode tables, GLM digit grouping and DeepSeek JoyAI behavior with exact-ID regression controls.
- Added existing-regex/lookahead guidance without normalization/new asset dependency, and documented safe old-prefix misses/no checkpoint bump for this issue alone.
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the two classification mechanisms, exact artifact regex semantics, every plain/rendered family dispatch, Qwen oracle IDs and unchanged GLM/DeepSeek IDs against DS4. Reject any proposal that edits shared Unicode tables globally.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit dcc4c51 remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.
Implemented in 8965e77ea173c932e00f70759d0a83ac1c5072f0 and pushed to main.
Qwen now uses the trained Unicode-aware split for both plain and rendered input, including all Nd/Nl/No categories, letter/mark spans and exact whitespace lookahead behavior. Existing regex/BPE dependencies are reused. DS4-backed GLM and DeepSeek tokenization and all checkpoint formats remain unchanged.
Reviewed the issue against pinned MTPLX and DS4. Isolated verification: all 1,112,064 Unicode scalars match the oracle split digest; 5,752 split/BPE/roundtrip cases cover every one of the oracle's 1,911 number characters; all four reasoning-mode request IDs/context lengths match; all five original failing samples are fixed. GLM/DeepSeek exact IDs on 19 shared controls and their standard three-turn chat fixtures match DS4 both before and after. Installed-model cache test rejects the actual legacy 89-token prefix for the corrected 93-token request, then reuses 95 tokens from both live and disk checkpoints with byte-identical resulting snapshot state.
All commit gates passed: rustfmt, warning-free all-target/all-feature Clippy, release app bundle, 310 application and 8 tool tests. Pinned tokenizer tests: 5 passed, one distinct uninstalled 27B pack skipped; installed Flash Next was tested separately.
Scope is complete. The explicitly excluded NFC stage differs on one decomposed-mark component control and is tracked separately as #116; full-oracle IDs are retained alongside component IDs. A pre-existing direct-mode reasoning-history mismatch found during review is tracked as #117. Neither is hidden as full tokenizer/template parity by this closure.
Implemented in `8965e77ea173c932e00f70759d0a83ac1c5072f0` and pushed to main.
Qwen now uses the trained Unicode-aware split for both plain and rendered input, including all Nd/Nl/No categories, letter/mark spans and exact whitespace lookahead behavior. Existing regex/BPE dependencies are reused. DS4-backed GLM and DeepSeek tokenization and all checkpoint formats remain unchanged.
Reviewed the issue against pinned MTPLX and DS4. Isolated verification: all 1,112,064 Unicode scalars match the oracle split digest; 5,752 split/BPE/roundtrip cases cover every one of the oracle's 1,911 number characters; all four reasoning-mode request IDs/context lengths match; all five original failing samples are fixed. GLM/DeepSeek exact IDs on 19 shared controls and their standard three-turn chat fixtures match DS4 both before and after. Installed-model cache test rejects the actual legacy 89-token prefix for the corrected 93-token request, then reuses 95 tokens from both live and disk checkpoints with byte-identical resulting snapshot state.
All commit gates passed: rustfmt, warning-free all-target/all-feature Clippy, release app bundle, 310 application and 8 tool tests. Pinned tokenizer tests: 5 passed, one distinct uninstalled 27B pack skipped; installed Flash Next was tested separately.
Evidence, exact IDs, receipts and reproduction: [docs/issue111-tokenizer](https://git.rfc1437.de/hugo/DS4Server/src/commit/8965e77ea173c932e00f70759d0a83ac1c5072f0/docs/issue111-tokenizer).
Scope is complete. The explicitly excluded NFC stage differs on one decomposed-mark component control and is tracked separately as #116; full-oracle IDs are retained alongside component IDs. A pre-existing direct-mode reasoning-history mismatch found during review is tracked as #117. Neither is hidden as full tokenizer/template parity by this closure.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Publication status: analysis commit
dcc4c516f0e030566c67472f8f1be0baa21ab629is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under/Users/gb/Projects/DS4Server/docs/issue103-analysis/and the committed test/tool paths, not yet at the remote file URLs.Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the independent review and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
9882703f3105363ddc37eca9f97aa09a1d387112, including token processing, RNG/sampling, context accounting, inference and cache/session state.This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
Broken behavior and evidence
Confirmed native Qwen tokenization mismatch. The shared handwritten Unicode tables misclassify trained Qwen3 number boundaries in two ways: ①(U+2460) is explicitly classified as punctuation by unicode_punctuation range 0x2460..=0x24ff; Ⅷ/²/𝟘 fall into the non-ASCII letter fallback because unicode_number omits their categories/ranges. The former joins a leading space via the punctuation branch; the latter via a letter span. Both differ from the trained Qwen3 Split contract, which uses letter/mark spans and single Unicode number-category matches.
The current shared
tokenize_gpt4/char_info code is also used by GLM with max_digits=3; Qwen uses max_digits=1, including tokenize and tokenize_plain/rendered paths. The GLM tables mirror DS4's glm4_unicode_number/glm4_char_at, so globally correcting them to Unicode/Qwen semantics would break the required GLM/DS4 token IDs. DeepSeek uses its separate JoyAI path and remains a DS4 regression control.Exact unchanged Qwen assets: " ①" actual[2765,239,254], oracle[220,46701,254]; " Ⅷ" actual[2765,88608], oracle[220,68086,100]; " ²" actual[2071,110], oracle[220,28495]; " 𝟘" actual[78449,253,246], oracle[220,54362,253,246]. Five mismatches including mixed-script sample; Hindi/Thai/vowelled Arabic/English/code/Chinese/mixed punctuation controls match. This is a native Qwen category bug, not a port of the transformers loader implementation.
Native implementation scope
Implement the trained artifact's Qwen3 pretokenizer contract exclusively for ModelFamily::Qwen in both plain and rendered dispatch. Preserve exact vocabulary/merges, letter+mark spans and single \p{N} number matching; no Unicode normalization. Reuse existing regex dependency for supported Unicode categories where practical; account explicitly for unsupported lookahead (for example \s+(?!\S)) using minimal native handling rather than accepting a different split law or adding a crate by default.
Do not change shared char_info/unicode_number/unicode_punctuation/tokenize_gpt4 behavior observed by ModelFamily::Glm, its three-digit rule, or DeepSeek's JoyAI path. A Qwen-specific split/classification branch is required wherever shared behavior differs. Protect both entry points (
tokenize,tokenize_plainreached by tokenize_rendered). Preserve safe token-exact cache matching: changed Qwen IDs cause a cache miss against old prefixes, not an unsafe warm restore. No checkpoint-format bump is justified by this tokenizer fix alone; do not change GLM tokenization to make a Unicode-standard test pass.Acceptance
Retain exact Qwen oracle IDs for all four minimal numerals/mixed sample, combining-mark scripts and existing language/code/punctuation controls. Cover Unicode Nd/Nl/No boundaries with spaces, neighboring marks/letters, contractions/newlines/whitespace-tail behavior and both plain/rendered entry points. Compare Qwen full rendered request IDs/context lengths against pinned MTPLX with unchanged assets; verify decode roundtrip and safe old-prefix miss/new-prefix warm reuse.
Mandatory DS4 controls: GLM 5.3 Flash exact token IDs for the same numeral examples plus standard DS4 chat fixtures must remain identical before/after and to DS4; DeepSeek/DSv4 Flash JoyAI/plain/rendered fixtures must likewise remain unchanged. Use exact IDs, not only decoded text, and prove the Qwen correction cannot route GLM/DeepSeek through the Qwen split. Existing shared tables are retained for their DS4-backed callers. No checkpoint invalidation or unrelated family behavior change.
Severity
Medium: deterministic Qwen prompt-ID/context mismatch. Existing token-exact restore selection yields safe misses on changed IDs; no corrupt-checkpoint consequence is claimed.
Shared evidence and reproduction
Analysis parent: #103. Affected DS4Server revision:
cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983. Compared MTPLX olde652d55e2652137a4abcf1312357abbf3eb9d692(2.10.2) with refreshed9882703f3105363ddc37eca9f97aa09a1d387112(2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision74559cdf34fbfc0b593de72d17e93f37fd4f9ea7; unchanged hashes, compiler/OS/hardware and effective options in environment.json. Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.Durable report and instructions, native test, oracle probe. The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
Immutable upstream change and latest test. The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
Validation and review boundary
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings,make bundle,cargo test --all-featuresbefore every future commit, plus the owned native/oracle and product regressions above.Independent review (2026-10-03)
Reviewed against DS4Server
dcc4c51and MTPLX 2.12.29882703f3105363ddc37eca9f97aa09a1d387112. Note: evidence commitdcc4c51is still unpublished (mainis ahead oforigin/mainby one commit), so the repository links in this issue currently resolve to nothing.Defect claim: confirmed in substance; one mechanism detail is wrong; scope must be family-restricted.
unicode_punctuation(range0x2460..=0x24ff), so " ①" merges through the?[^\s\p{L}\p{N}]+branch. Ⅷ, ², 𝟘 do fall into the non-ASCII letter fallback. Both are consequences of the hand-writtenunicode_number/unicode_punctuationtables, and the token IDs are wrong either way.char_infoandtokenize_gpt4are shared with GLM (tokenizer.rs:328-329,ModelFamily::Glm => tokenize_gpt4(.., 3)), and our tables mirror DS4'sglm4_unicode_number/glm4_char_at(ds4.c). Per AGENTS.md, DS4 is the behavioral oracle for those families. Changing the shared classification globally would break GLM parity with DS4. The fix must apply only toModelFamily::Qwen, with GLM/DeepSeek tokenization covered as unchanged regression controls.tokenizer.jsoncarries the Qwen3 Split regex ([\p{L}\p{M}]+, single\p{N}). The existingregexdependency supports these Unicode classes; only the\s+(?!\S)lookahead needs manual handling. This would avoid maintaining another hand table.Severity Medium is reasonable. Changed IDs only cause cache misses against old prefixes (safe), so no checkpoint invalidation is needed.
Description corrected after independent review (2026-10-03)
Updated the issue body in response to your review. Changes:
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the two classification mechanisms, exact artifact regex semantics, every plain/rendered family dispatch, Qwen oracle IDs and unchanged GLM/DeepSeek IDs against DS4. Reject any proposal that edits shared Unicode tables globally.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit
dcc4c51remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.Implemented in
8965e77ea173c932e00f70759d0a83ac1c5072f0and pushed to main.Qwen now uses the trained Unicode-aware split for both plain and rendered input, including all Nd/Nl/No categories, letter/mark spans and exact whitespace lookahead behavior. Existing regex/BPE dependencies are reused. DS4-backed GLM and DeepSeek tokenization and all checkpoint formats remain unchanged.
Reviewed the issue against pinned MTPLX and DS4. Isolated verification: all 1,112,064 Unicode scalars match the oracle split digest; 5,752 split/BPE/roundtrip cases cover every one of the oracle's 1,911 number characters; all four reasoning-mode request IDs/context lengths match; all five original failing samples are fixed. GLM/DeepSeek exact IDs on 19 shared controls and their standard three-turn chat fixtures match DS4 both before and after. Installed-model cache test rejects the actual legacy 89-token prefix for the corrected 93-token request, then reuses 95 tokens from both live and disk checkpoints with byte-identical resulting snapshot state.
All commit gates passed: rustfmt, warning-free all-target/all-feature Clippy, release app bundle, 310 application and 8 tool tests. Pinned tokenizer tests: 5 passed, one distinct uninstalled 27B pack skipped; installed Flash Next was tested separately.
Evidence, exact IDs, receipts and reproduction: docs/issue111-tokenizer.
Scope is complete. The explicitly excluded NFC stage differs on one decomposed-mark component control and is tracked separately as #116; full-oracle IDs are retained alongside component IDs. A pre-existing direct-mode reasoning-history mismatch found during review is tracked as #117. Neither is hidden as full tokenizer/template parity by this closure.