Match Qwen Unicode number token boundaries

docs/issue111-tokenizer/README.md

@@ -0,0 +1,89 @@
+# Qwen Unicode number boundaries (#111)
+
+The Qwen-only splitter now implements the trained Qwen3 expression using the
+existing Rust regex dependency. Its unsupported whitespace lookahead is handled
+by retaining the last whitespace scalar for the following match. Both plain and
+rendered tokenization dispatch to it. GLM still calls the unchanged DS4 GPT4
+splitter with three-digit grouping; DeepSeek still calls unchanged JoyAI code.
+Vocabulary, merges, Unicode tables, sampling and checkpoint formats are unchanged.
+
+Oracle: MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`, installed
+Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, MLX 0.32.2.
+Artifact and environment fingerprints are in ../issue104-ar/environment.json.
+DS4 `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`; exact executable hash in ds4.json.
+Native baseline is `990abb4755f3aa38babfb819f84fe1ab9bb018da`.
+
+## Isolated results and review
+
+- All 1,112,064 Unicode scalars in letter/space context: 2,082,513 split pieces,
+  exactly the oracle digest `7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a`.
+- All 1,911 oracle number characters (Nd/Nl/No) with spaces, neighboring letters
+  and combining marks, plus language/code/punctuation/contraction/newline/tail
+  whitespace controls: 5,752 exact split and BPE cases, both entry points, exact
+  original-byte decode round trips. Original #103 regression: zero mismatches.
+- Full three-message requests and history+continuation IDs match the unmodified
+  pinned server in direct/low/medium/xhigh modes: 59/93/63/105 context tokens.
+- GLM and DeepSeek: exact DS4 IDs on 19 shared controls, plain and rendered,
+  before and after. Existing three-turn DS4 chat fixtures also pass both before
+  and after. Baseline executable used the committed tokenizer source plus only
+  the test module; legacy-requests.json contains its actual rendered request IDs.
+- Installed-model product owner: prefill/save/reopen actual old IDs (89 tokens),
+  align against corrected request (93 tokens), verify complete cache reset, then
+  generate cold. Corrected warm and disk-restored continuations each reuse 95
+  tokens and produce byte-identical complete snapshot hashes. Peak supervised
+  physical footprint about 77.2 GB; no memory or idle limit violation.
+- Caller review: only the two Qwen dispatch arms change. Both feed the existing
+  byte BPE. Shared DS4 Unicode tables and three-digit rule remain unchanged.
+  `QwenExecutor::align_prompt` retains exact token-prefix selection and clears
+  incompatible state; compatible new checkpoints continue to load normally.
+
+### Separate pre-existing contracts
+
+This issue explicitly excludes Unicode normalization. qwen.json retains both
+component `ids` and unmodified full-oracle `full_ids`. The component oracle removes
+only NFC, leaving trained split/BPE intact. 5,751 cases match the full oracle too;
+one decomposed combining-mark control differs because native NFC is missing.
+normalization-gap.json preserves that exact difference, tracked as #116. This is
+not a claim of complete Qwen tokenizer parity before #116.
+
+Direct-mode history fixtures use empty reasoning, as generated in that mode.
+Switching a reasoning conversation to direct mode exposes another existing
+renderer difference: native preserves its historical reasoning while MTPLX drops
+it. This is separately tracked as #117. Nonempty reasoning history is tested in
+all three reasoning modes here.
+
+## Reproduce
+
+Generate fixtures with the unchanged model assets and isolated oracle environment:
+
+```sh
+local-eval-results/mtplx-reference-env-0.32.2/bin/python \
+  tools/issue111-reference.py /path/to/pinned/mtplx "$DS4_QWEN38_ARTIFACTS" /tmp/issue111-oracle \
+  --ds4 /path/to/ds4/ds4 --glm /path/to/GLM.gguf --deepseek /path/to/DeepSeek.gguf
+cargo test --all-features --bin ds4-server issue111 --no-run
+cargo build --bin test-supervisor
+export DS4_ISSUE111_ORACLE="$PWD/docs/issue111-tokenizer"
+export DS4_ISSUE111_DIR=/tmp/issue111-product
+# Use the emitted test executable after compilation finishes.
+target/debug/test-supervisor 4096 60 120 --command /path/to/test-binary \
+  engine::tokenizer::issue111 --ignored --nocapture --test-threads=1
+# Run GPU work serially.
+target/debug/test-supervisor 114688 60 120 --command /path/to/test-binary \
+  issue111_token_prefix_cache --ignored --nocapture --test-threads=1
+DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/deepseek-chat.json" \
+  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
+DS4SERVER_CHAT_REFERENCE="$PWD/docs/issue111-tokenizer/glm-chat.json" \
+  /path/to/test-binary ds4_chat_matches_original_session_tokens --ignored --nocapture
+```
+
+Set DS4_QWEN38_ARTIFACTS to the model directory. DS4 fixture model paths can be
+updated for another machine without changing IDs. Baseline and native logs,
+product receipt, exact oracle IDs and split digest are retained here. The small
+`qwen_split_boundaries` regression also runs without model assets in ordinary tests.
+This is an MTPLX compatibility fix; DS4/GLM behavior is intentionally preserved.
+
+Commit gates passed: rustfmt check, all-target/all-feature Clippy with warnings
+denied, release app bundle with ad-hoc signing, and 310 application + 8 tool tests
+(228 opt-in tests ignored by the ordinary gate). Pinned upstream tokenizer tests:
+5 passed; the one 27B Quality-pack test skipped because that distinct model pack
+is not installed. The installed Flash Next model was exercised above.

docs/issue111-tokenizer/baseline.log

@@ -0,0 +1,9 @@
+
+running 2 tests
+test engine::tokenizer::issue111::issue111_ds4_families ... {"case":"ds4_control","family":"glm","cases":19,"plain_rendered_exact":true}
+{"case":"ds4_control","family":"deepseek","cases":19,"plain_rendered_exact":true}
+ok
+test engine::tokenizer::issue111::issue111_legacy_requests ... ok
+
+test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 533 filtered out; finished in 0.80s
+

docs/issue111-tokenizer/bundle.log

@@ -0,0 +1,8 @@
+cargo build --release --all-targets --all-features
+   Compiling ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
+    Finished `release` profile [optimized] target(s) in 38.38s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
+cargo packager --release --formats app
+codesign --force --deep --sign - --timestamp=none target/release/DS4Server.app
+target/release/DS4Server.app: replacing existing signature

docs/issue111-tokenizer/clippy.log

@@ -0,0 +1,4 @@
+    Checking ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
+    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.95s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`

docs/issue111-tokenizer/ds4-chat.log

@@ -0,0 +1,28 @@
+before chat-token-fixture.json
+
+running 1 test
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 534 filtered out; finished in 0.14s
+
+before glm-chat-token-fixture.json
+
+running 1 test
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 534 filtered out; finished in 0.23s
+
+after chat-token-fixture.json
+
+running 1 test
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 537 filtered out; finished in 0.15s
+
+after glm-chat-token-fixture.json
+
+running 1 test
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 537 filtered out; finished in 0.23s
+

docs/issue111-tokenizer/ds4.json

@@ -0,0 +1 @@
+{"revision": "ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748", "binary_sha256": "53eae11b7d98f4f73435d28753870b6338a8740cd6084e028d8ed9fdd45d3111", "families": [{"family": "glm", "model": "/Users/gb/Library/Application Support/de.rfc1437.ds4server/models/glm-5.3-flash/GLM-5.3-Flash-Q2.gguf", "cases": [{"text": " ①", "ids": [111229]}, {"text": " Ⅷ", "ids": [2858, 91001]}, {"text": " ²", "ids": [2139, 110]}, {"text": " 𝟘", "ids": [80590, 253, 246]}, {"text": "Item ①, Ⅷ, ², 𝟘 and 123.", "ids": [1234, 111229, 11, 2858, 91001, 11, 2139, 110, 11, 80590, 253, 246, 323, 220, 108714, 13]}, {"text": "भारत एक विशाल देश है जिसकी संस्कृति बहुत पुरानी है।", "ids": [151387, 31280, 108, 78601, 14913, 237, 64234, 14913, 113, 42069, 114, 31280, 110, 14913, 99, 54418, 114, 83615, 151404, 14913, 250, 42069, 116, 64234, 43397, 67662, 71759, 77740, 30366, 243, 151400, 78601, 38643, 14913, 105, 93158, 72094, 78601, 82946, 72094, 43923, 31280, 101, 43397, 83615, 151404, 151430]}, {"text": "ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก", "ids": [54318, 22230, 71712, 27796, 35705, 153221, 49438, 35705, 34339, 26201, 28221, 37019, 23208, 153214, 20144, 153215, 22230, 22230, 26201, 35705, 46880, 27796, 25122, 64270, 70767, 25122, 18592, 70767, 31649, 71712, 35470, 37019, 34339, 30317, 20728, 26201, 26201, 20728, 25122]}, {"text": "الْعَرَبِيَّةُ", "ids": [31251, 58985, 23161, 27815, 11066, 27815, 21310, 52357, 14548, 132158, 25790, 64137]}, {"text": "The quick brown fox jumps over the lazy dog.", "ids": [785, 3974, 13867, 38627, 34041, 916, 279, 15666, 5562, 13]}, {"text": "中文，标点。", "ids": [102169, 3837, 98593, 98428, 1773]}, {"text": "def f(x):\n    return x + 123\n", "ids": [750, 282, 2075, 982, 262, 470, 856, 488, 220, 108714, 198]}, {"text": "I'm WE'RE it's he'd I'll you've can't 'ſx", "ids": [40, 2776, 19640, 93362, 432, 594, 566, 4172, 358, 3278, 498, 3003, 646, 944, 364, 153000, 87]}, {"text": "́a á ́ १́b", "ids": [53481, 64, 264, 53481, 220, 53481, 220, 151433, 53481, 65]}, {"text": "a\r\n  b\t\t", "ids": [64, 319, 220, 293, 298]}, {"text": "a   ", "ids": [64, 262]}, {"text": "a\t  b", "ids": [64, 3677, 103245, 65]}, {"text": "a… b", "ids": [64, 126, 227, 4102, 65]}, {"text": "x \n \r\n   y", "ids": [87, 715, 2529, 256, 379]}, {"text": "😀 ①🙂Ⅷ ²+𝟘", "ids": [152336, 111229, 152402, 69942, 100, 2139, 110, 10, 55873, 253, 246]}]}, {"family": "deepseek", "model": "/Users/gb/Library/Application Support/de.rfc1437.ds4server/models/deepseek-v4-flash-0731/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf", "cases": [{"text": " ①", "ids": [1327, 242, 257]}, {"text": " Ⅷ", "ids": [1327, 230, 103]}, {"text": " ²", "ids": [1203, 113]}, {"text": " 𝟘", "ids": [223, 70958, 249]}, {"text": "Item ①, Ⅷ, ², 𝟘 and 123.", "ids": [7519, 1327, 242, 257, 14, 1327, 230, 103, 14, 1203, 113, 14, 223, 70958, 249, 305, 223, 6895, 16]}, {"text": "भारत एक विशाल देश है जिसकी संस्कृति बहुत पुरानी है।", "ids": [34952, 17989, 8422, 66862, 48957, 117, 31639, 26227, 79882, 35674, 27039, 64689, 11018, 8714, 58501, 11931, 61515, 32774, 52058, 28568, 18359, 11634, 8422, 14410, 56275, 22336, 8714, 35674, 4361]}, {"text": "ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก", "ids": [113424, 9289, 125977, 5689, 13307, 15316, 64120, 8429, 2952, 22401, 1979, 6565, 24208]}, {"text": "الْعَرَبِيَّةُ", "ids": [1712, 9821, 1183, 52748, 61079, 47029, 106817, 7680]}, {"text": "The quick brown fox jumps over the lazy dog.", "ids": [671, 4787, 13769, 46012, 54994, 1060, 270, 41638, 6397, 16]}, {"text": "中文，标点。", "ids": [21134, 303, 1701, 985, 320]}, {"text": "def f(x):\n    return x + 123\n", "ids": [3465, 285, 4042, 3395, 361, 1354, 1527, 940, 223, 6895, 201]}, {"text": "I'm WE'RE it's he'd I'll you've can't 'ſx", "ids": [43, 4571, 33418, 9, 3827, 436, 734, 468, 7485, 342, 5922, 440, 5270, 588, 1664, 905, 75929, 90]}, {"text": "́a á ́ १́b", "ids": [17793, 67, 260, 17793, 223, 17793, 223, 26020, 17793, 68]}, {"text": "a\r\n  b\t\t", "ids": [67, 204, 201, 223, 291, 998]}, {"text": "a   ", "ids": [67, 361]}, {"text": "a\t  b", "ids": [67, 200, 663, 228, 68]}, {"text": "a… b", "ids": [67, 129, 230, 2162, 68]}, {"text": "x \n \r\n   y", "ids": [90, 539, 223, 204, 201, 262, 383]}, {"text": "😀 ①🙂Ⅷ ²+𝟘", "ids": [28927, 225, 1327, 242, 257, 80300, 227, 23603, 103, 1203, 113, 13, 70958, 249]}]}]}

docs/issue111-tokenizer/legacy-requests.json

@@ -0,0 +1 @@
+[[248045,846,198,46701,254,2765,88608,2071,110,78449,253,246,199049,220,52033,64,248046,198,248045,74455,198,248068,271,248069,271,3833,2765,239,254,13,248046,198,248045,846,198,22791,2765,88608,321,78449,253,246,13,198,256,830,248046,198,248045,74455,198,248068,271,248069,271],[248045,8678,198,24342,286,4879,369,716,310,3238,13,13262,678,7047,9522,321,10419,11,6992,5774,310,279,16198,1973,24366,24150,362,13,248046,198,248045,846,198,46701,254,2765,88608,2071,110,78449,253,246,199049,220,52033,64,248046,198,248045,74455,198,248068,198,83017,2071,110,3272,13,198,248069,271,3833,2765,239,254,13,248046,198,248045,846,198,22791,2765,88608,321,78449,253,246,13,198,256,830,248046,198,248045,74455,198,248068,198],[248045,846,198,46701,254,2765,88608,2071,110,78449,253,246,199049,220,52033,64,248046,198,248045,74455,198,248068,198,83017,2071,110,3272,13,198,248069,271,3833,2765,239,254,13,248046,198,248045,846,198,22791,2765,88608,321,78449,253,246,13,198,256,830,248046,198,248045,74455,198,248068,198],[248045,8678,198,24342,286,4879,369,716,310,830,11553,13,5044,1683,15060,1472,279,3274,11,9307,1328,30800,11,2814,47675,25605,11,321,60445,55404,11,27224,11,321,30246,303,279,1534,4087,13,248046,198,248045,846,198,46701,254,2765,88608,2071,110,78449,253,246,199049,220,52033,64,248046,198,248045,74455,198,248068,198,83017,2071,110,3272,13,198,248069,271,3833,2765,239,254,13,248046,198,248045,846,198,22791,2765,88608,321,78449,253,246,13,198,256,830,248046,198,248045,74455,198,248068,198]]

docs/issue111-tokenizer/native.log

@@ -0,0 +1,16 @@
+
+running 3 tests
+test engine::tokenizer::issue111::issue111_ds4_families ... {"case":"ds4_control","family":"glm","cases":19,"plain_rendered_exact":true}
+{"case":"ds4_control","family":"deepseek","cases":19,"plain_rendered_exact":true}
+ok
+test engine::tokenizer::issue111::issue111_every_unicode_scalar_split ... {"case":"unicode_split","scalars":1112064,"pieces":2082513,"sha256":"7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a","exact":true}
+ok
+test engine::tokenizer::issue111::issue111_qwen_tokens_and_requests ... {"case":"request","effort":"direct","context_tokens":59,"exact":true}
+{"case":"request","effort":"low","context_tokens":93,"exact":true}
+{"case":"request","effort":"medium","context_tokens":63,"exact":true}
+{"case":"request","effort":"xhigh","context_tokens":105,"exact":true}
+{"case":"tokenizer","number_count":1911,"cases":5752,"plain_rendered_roundtrip_exact":true}
+ok
+
+test result: ok. 3 passed; 0 failed; 0 ignored; 0 measured; 534 filtered out; finished in 6.13s
+

docs/issue111-tokenizer/normalization-gap.json

@@ -0,0 +1,38 @@
+[
+  {
+    "text": "́a á ́ १́b",
+    "ids": [
+      52033,
+      64,
+      264,
+      52033,
+      220,
+      52033,
+      220,
+      12256,
+      100,
+      52033,
+      65
+    ],
+    "full_ids": [
+      52033,
+      64,
+      18135,
+      220,
+      52033,
+      220,
+      12256,
+      100,
+      52033,
+      65
+    ],
+    "pieces": [
+      "́a",
+      " á",
+      " ́",
+      " ",
+      "१",
+      "́b"
+    ]
+  }
+]

docs/issue111-tokenizer/original-regression.log

@@ -0,0 +1,19 @@
+
+running 1 test
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_tokenizer_receipts ... {"case":"native_tokenizer","text":"भारत एक विशाल देश है जिसकी संस्कृति बहुत पुरानी है।","actual_ids":[149755,148504,76519,171549,165997,151224,197734,170046,184864,160357,172281,237627,158827,153570,199638,80752,151859,181117,170046,154845],"expected_ids":[149755,148504,76519,171549,165997,151224,197734,170046,184864,160357,172281,237627,158827,153570,199638,80752,151859,181117,170046,154845],"matches":true}
+{"case":"native_tokenizer","text":"ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก","actual_ids":[156150,148450,186319,148285,178118,155545,149119,162566,149196],"expected_ids":[156150,148450,186319,148285,178118,155545,149119,162566,149196],"matches":true}
+{"case":"native_tokenizer","text":"إِنَّ اللُّغَةَ الْعَرَبِيَّةَ جَمِيلَةٌ","actual_ids":[88792,164372,148784,159613,78958,191556,205629,153458,150765,151003,178402,27004,79333,152397,216808,233790],"expected_ids":[88792,164372,148784,159613,78958,191556,205629,153458,150765,151003,178402,27004,79333,152397,216808,233790],"matches":true}
+{"case":"native_tokenizer","text":"The quick brown fox jumps over the lazy dog near the river bank.","actual_ids":[760,3841,13477,37550,33075,888,279,15217,5388,3043,279,14367,5883,13],"expected_ids":[760,3841,13477,37550,33075,888,279,15217,5388,3043,279,14367,5883,13],"matches":true}
+{"case":"native_tokenizer","text":"def foo(x):\n    return x + 1\n","actual_ids":[727,14785,2007,1590,198,262,460,830,478,220,16,198],"expected_ids":[727,14785,2007,1590,198,262,460,830,478,220,16,198],"matches":true}
+{"case":"native_tokenizer","text":"中国的文化历史悠久。","actual_ids":[108733,96655,125188,1710],"expected_ids":[108733,96655,125188,1710],"matches":true}
+{"case":"native_tokenizer","text":"a🙂b ©c ∑d —e …f €g ™h\n","actual_ids":[64,169171,65,7013,66,11654,239,67,1892,68,4444,69,12612,70,27717,95,71,198],"expected_ids":[64,169171,65,7013,66,11654,239,67,1892,68,4444,69,12612,70,27717,95,71,198],"matches":true}
+{"case":"native_tokenizer","text":"x́y १२३ ١٢٣ ①Ⅷ²‍Z","actual_ids":[87,52033,88,220,12256,100,12256,101,12256,102,220,149,94,149,95,149,96,220,46701,254,68086,100,28495,373,235,57],"expected_ids":[87,52033,88,220,12256,100,12256,101,12256,102,220,149,94,149,95,149,96,220,46701,254,68086,100,28495,373,235,57],"matches":true}
+{"case":"native_tokenizer","text":" ①","actual_ids":[220,46701,254],"expected_ids":[220,46701,254],"matches":true}
+{"case":"native_tokenizer","text":" Ⅷ","actual_ids":[220,68086,100],"expected_ids":[220,68086,100],"matches":true}
+{"case":"native_tokenizer","text":" ²","actual_ids":[220,28495],"expected_ids":[220,28495],"matches":true}
+{"case":"native_tokenizer","text":" 𝟘","actual_ids":[220,54362,253,246],"expected_ids":[220,54362,253,246],"matches":true}
+{"case":"tokenizer_summary","mismatches":0}
+ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 537 filtered out; finished in 0.53s
+

docs/issue111-tokenizer/product.jsonl

@@ -0,0 +1,2 @@
+{"case":"token_cache","old_tokens":89,"new_tokens":93,"old_prefix_miss":true,"warm_cached":95,"disk_cached":95,"warm_disk_state_exact":true}
+{"event":"test_resource_summary","elapsed_ms":18595,"peak":{"cpu_time_ns":19116354416,"resident_bytes":25394003968,"physical_bytes":77175615456,"peak_physical_bytes":77175648272,"disk_read_bytes":73502404608,"disk_write_bytes":470114304},"error":null}

docs/issue111-tokenizer/qwen.json

@@ -0,0 +1 @@
+{"number_count": 1911, "cases": [{"text": " ①", "ids": [220, 46701, 254], "full_ids": [220, 46701, 254], "pieces": [" ", "①"]}, {"text": " Ⅷ", "ids": [220, 68086, 100], "full_ids": [220, 68086, 100], "pieces": [" ", "Ⅷ"]}, {"text": " ²", "ids": [220, 28495], "full_ids": [220, 28495], "pieces": [" ", "²"]}, {"text": " 𝟘", "ids": [220, 54362, 253, 246], "full_ids": [220, 54362, 253, 246], "pieces": [" ", "𝟘"]}, {"text": "Item ①, Ⅷ, ², 𝟘 and 123.", "ids": [1192, 220, 46701, 254, 11, 220, 68086, 100, 11, 220, 28495, 11, 220, 54362, 253, 246, 321, 220, 16, 17, 18, 13], "full_ids": [1192, 220, 46701, 254, 11, 220, 68086, 100, 11, 220, 28495, 11, 220, 54362, 253, 246, 321, 220, 16, 17, 18, 13], "pieces": ["Item", " ", "①", ",", " ", "Ⅷ", ",", " ", "²", ",", " ", "𝟘", " and", " ", "1", "2", "3", "."]}, {"text": "भारत एक विशाल देश है जिसकी संस्कृति बहुत पुरानी है।", "ids": [149755, 148504, 76519, 171549, 165997, 151224, 197734, 170046, 184864, 160357, 172281, 237627, 158827, 153570, 199638, 80752, 151859, 181117, 170046, 154845], "full_ids": [149755, 148504, 76519, 171549, 165997, 151224, 197734, 170046, 184864, 160357, 172281, 237627, 158827, 153570, 199638, 80752, 151859, 181117, 170046, 154845], "pieces": ["भारत", " एक", " विशाल", " देश", " है", " जिसकी", " संस्कृति", " बहुत", " पुरानी", " है", "।"]}, {"text": "ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก", "ids": [156150, 148450, 186319, 148285, 178118, 155545, 149119, 162566, 149196], "full_ids": [156150, 148450, 186319, 148285, 178118, 155545, 149119, 162566, 149196], "pieces": ["ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก"]}, {"text": "الْعَرَبِيَّةُ", "ids": [30350, 187581, 153458, 150765, 151003, 178402, 62422], "full_ids": [30350, 187581, 153458, 150765, 151003, 178402, 62422], "pieces": ["الْعَرَبِيَّةُ"]}, {"text": "The quick brown fox jumps over the lazy dog.", "ids": [760, 3841, 13477, 37550, 33075, 888, 279, 15217, 5388, 13], "full_ids": [760, 3841, 13477, 37550, 33075, 888, 279, 15217, 5388, 13], "pieces": ["The", " quick", " brown", " fox", " jumps", " over", " the", " lazy", " dog", "."]}, {"text": "中文，标点。", "ids": [99986, 3709, 141111, 1710], "full_ids": [99986, 3709, 141111, 1710], "pieces": ["中文", "，标点", "。"]}, {"text": "def f(x):\n    return x + 123\n", "ids": [727, 281, 2007, 1590, 198, 262, 460, 830, 478, 220, 16, 17, 18, 198], "full_ids": [727, 281, 2007, 1590, 198, 262, 460, 830, 478, 220, 16, 17, 18, 198], "pieces": ["def", " f", "(x", "):\n", "   ", " return", " x", " +", " ", "1", "2", "3", "\n"]}, {"text": "I'm WE'RE it's he'd I'll you've can't 'ſx", "ids": [40, 2688, 19076, 90902, 424, 579, 551, 4035, 353, 3172, 488, 2908, 628, 914, 359, 129, 123, 87], "full_ids": [40, 2688, 19076, 90902, 424, 579, 551, 4035, 353, 3172, 488, 2908, 628, 914, 359, 129, 123, 87], "pieces": ["I", "'m", " WE", "'RE", " it", "'s", " he", "'d", " I", "'ll", " you", "'ve", " can", "'t", " '", "ſx"]}, {"text": "́a á ́ १́b", "ids": [52033, 64, 264, 52033, 220, 52033, 220, 12256, 100, 52033, 65], "full_ids": [52033, 64, 18135, 220, 52033, 220, 12256, 100, 52033, 65], "pieces": ["́a", " á", " ́", " ", "१", "́b"]}, {"text": "a\r\n  b\t\t", "ids": [64, 317, 220, 292, 297], "full_ids": [64, 317, 220, 292, 297], "pieces": ["a", "\r\n", " ", " b", "\t\t"]}, {"text": "a   ", "ids": [64, 262], "full_ids": [64, 262], "pieces": ["a", "   "]}, {"text": "a\t  b", "ids": [64, 3551, 373, 225, 65], "full_ids": [64, 3551, 373, 225, 65], "pieces": ["a", "\t ", " b"]}, {"text": "a… b", "ids": [64, 126, 227, 3966, 65], "full_ids": [64, 126, 227, 3966, 65], "pieces": ["a", "…", " b"]}, {"text": "x \n \r\n   y", "ids": [87, 695, 2449, 256, 374], "full_ids": [87, 695, 2449, 256, 374], "pieces": ["x", " \n \r\n", "  ", " y"]}, {"text": "😀 ①🙂Ⅷ ²+𝟘", "ids": [169706, 220, 46701, 254, 169171, 68086, 100, 220, 28495, 10, 54362, 253, 246], "full_ids": [169706, 220, 46701, 254, 169171, 68086, 100, 220, 28495, 10, 54362, 253, 246], "pieces": ["😀", " ", "①", "🙂", "Ⅷ", " ", "²", "+", "𝟘"]}, {"text": " 0", "ids": [220, 15], "full_ids": [220, 15], "pieces": [" ", "0"]}, {"text": "a0b", "ids": [64, 15, 65], "full_ids": [64, 15, 65], "pieces": ["a", "0", "b"]}, {"text": " ́0́x", "ids": [220, 52033, 15, 52033, 87], "full_ids": [220, 52033, 15, 52033, 87], "pieces": [" ́", "0", "́x"]}, {"text": " 1", "ids": [220, 16], "full_ids": [220, 16], "pieces": [" ", "1"]}, {"text": "a1b", "ids": [64, 16, 65], "full_ids": [64, 16, 65], "pieces": ["a", "1", "b"]}, {"text": " ́1́x", "ids": [220, 52033, 16, 52033, 87], "full_ids": [220, 52033, 16, 52033,Line truncated

docs/issue111-tokenizer/requests.json

@@ -0,0 +1 @@
+[{"effort": "direct", "messages": [{"role": "user", "content": " ① Ⅷ ² 𝟘 भारत ́a\n "}, {"role": "assistant", "content": "One ①.", "reasoning_content": ""}, {"role": "user", "content": "Continue Ⅷ and 𝟘.\n   x"}], "ids": [248045, 846, 198, 46701, 254, 220, 68086, 100, 220, 28495, 220, 54362, 253, 246, 199049, 220, 52033, 64, 248046, 198, 248045, 74455, 198, 248068, 271, 248069, 271, 3833, 220, 46701, 254, 13, 248046, 198, 248045, 846, 198, 22791, 220, 68086, 100, 321, 220, 54362, 253, 246, 13, 198, 256, 830, 248046, 198, 248045, 74455, 198, 248068, 271, 248069, 271], "rendered": "<|im_start|>user\n① Ⅷ ² 𝟘 भारत ́a<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\nOne ①.<|im_end|>\n<|im_start|>user\nContinue Ⅷ and 𝟘.\n   x<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"}, {"effort": "low", "messages": [{"role": "user", "content": " ① Ⅷ ² 𝟘 भारत ́a\n "}, {"role": "assistant", "content": "One ①.", "reasoning_content": "Brief ² thought."}, {"role": "user", "content": "Continue Ⅷ and 𝟘.\n   x"}], "ids": [248045, 8678, 198, 24342, 286, 4879, 369, 716, 310, 3238, 13, 13262, 678, 7047, 9522, 321, 10419, 11, 6992, 5774, 310, 279, 16198, 1973, 24366, 24150, 362, 13, 248046, 198, 248045, 846, 198, 46701, 254, 220, 68086, 100, 220, 28495, 220, 54362, 253, 246, 199049, 220, 52033, 64, 248046, 198, 248045, 74455, 198, 248068, 198, 83017, 220, 28495, 3272, 13, 198, 248069, 271, 3833, 220, 46701, 254, 13, 248046, 198, 248045, 846, 198, 22791, 220, 68086, 100, 321, 220, 54362, 253, 246, 13, 198, 256, 830, 248046, 198, 248045, 74455, 198, 248068, 198], "rendered": "<|im_start|>system\nReasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.<|im_end|>\n<|im_start|>user\n① Ⅷ ² 𝟘 भारत ́a<|im_end|>\n<|im_start|>assistant\n<think>\nBrief ² thought.\n</think>\n\nOne ①.<|im_end|>\n<|im_start|>user\nContinue Ⅷ and 𝟘.\n   x<|im_end|>\n<|im_start|>assistant\n<think>\n"}, {"effort": "medium", "messages": [{"role": "user", "content": " ① Ⅷ ² 𝟘 भारत ́a\n "}, {"role": "assistant", "content": "One ①.", "reasoning_content": "Brief ² thought."}, {"role": "user", "content": "Continue Ⅷ and 𝟘.\n   x"}], "ids": [248045, 846, 198, 46701, 254, 220, 68086, 100, 220, 28495, 220, 54362, 253, 246, 199049, 220, 52033, 64, 248046, 198, 248045, 74455, 198, 248068, 198, 83017, 220, 28495, 3272, 13, 198, 248069, 271, 3833, 220, 46701, 254, 13, 248046, 198, 248045, 846, 198, 22791, 220, 68086, 100, 321, 220, 54362, 253, 246, 13, 198, 256, 830, 248046, 198, 248045, 74455, 198, 248068, 198], "rendered": "<|im_start|>user\n① Ⅷ ² 𝟘 भारत ́a<|im_end|>\n<|im_start|>assistant\n<think>\nBrief ² thought.\n</think>\n\nOne ①.<|im_end|>\n<|im_start|>user\nContinue Ⅷ and 𝟘.\n   x<|im_end|>\n<|im_start|>assistant\n<think>\n"}, {"effort": "xhigh", "messages": [{"role": "user", "content": " ① Ⅷ ² 𝟘 भारत ́a\n "}, {"role": "assistant", "content": "One ①.", "reasoning_content": "Brief ² thought."}, {"role": "user", "content": "Continue Ⅷ and 𝟘.\n   x"}], "ids": [248045, 8678, 198, 24342, 286, 4879, 369, 716, 310, 830, 11553, 13, 5044, 1683, 15060, 1472, 279, 3274, 11, 9307, 1328, 30800, 11, 2814, 47675, 25605, 11, 321, 60445, 55404, 11, 27224, 11, 321, 30246, 303, 279, 1534, 4087, 13, 248046, 198, 248045, 846, 198, 46701, 254, 220, 68086, 100, 220, 28495, 220, 54362, 253, 246, 199049, 220, 52033, 64, 248046, 198, 248045, 74455, 198, 248068, 198, 83017, 220, 28495, 3272, 13, 198, 248069, 271, 3833, 220, 46701, 254, 13, 248046, 198, 248045, 846, 198, 22791, 220, 68086, 100, 321, 220, 54362, 253, 246, 13, 198, 256, 830, 248046, 198, 248045, 74455, 198, 248068, 198], "rendered": "<|im_start|>system\nReasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.<|im_end|>\n<|im_start|>user\n① Ⅷ ² 𝟘 भारत ́a<|im_end|>\n<|im_start|>assistant\n<think>\nBrief ² thought.\n</think>\n\nOne ①.<|im_end|>\n<|im_start|>user\nContinue Ⅷ and 𝟘.\n   x<|im_end|>\n<|im_start|>assistant\n<think>\n"}]

docs/issue111-tokenizer/scalar-split.json

@@ -0,0 +1 @@
+{"scalars": 1112064, "pieces": 2082513, "sha256": "7ef8755fb56062d9695b38bc132f325ea59c3c65f198e1b1ab20bff1afef632a", "pattern": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+"}

docs/issue111-tokenizer/tests.log

@@ -0,0 +1,582 @@
+   Compiling ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 5.89s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
+     Running unittests src/main.rs (target/debug/deps/ds4_server-5679c54d8848b708)
+
+running 538 tests
+test agent::tests::ai_risk_response_requires_structured_risk_and_reason ... ok
+test agent::tests::anchored_edits_require_unique_head_and_tail ... ok
+test a2ui::tests::streaming_parser_only_accepts_complete_a2ui_jsonl_lines ... ok
+test agent::tests::ai_shell_risk_assessment_receives_the_dev_brain_root ... ok
+test agent::tests::dev_brain_paths_are_trusted_by_shell_risk_assessment ... ok
+test a2ui::tests::basic_formatters_follow_catalog_examples ... ok
+test a2ui::tests::embedded_catalog_and_validator_cover_the_same_components ... ok
+test a2ui::tests::catalog_validation_rejects_invalid_variants_and_function_arguments ... ok
+test a2ui::tests::catalog_accepts_pie_donut_and_heatmap_charts ... ok
+test a2ui::tests::latest_created_surface_is_the_active_surface ... ok
+test a2ui::tests::lifecycle_updates_incrementally_and_json_pointer_upserts ... ok
+test a2ui::tests::dismissal_boundaries_replay_history_and_a_fresh_active_epoch ... ok
+test a2ui::tests::catalog_validation_and_actions_use_current_local_data ... ok
+test agent::tests::ralph_loop_stops_on_cap_blocker_failure_and_cancellation ... ok
+test a2ui::tests::all_basics_compose_into_one_surface ... ok
+test a2ui::tests::v1_action_and_function_round_trips_follow_current_schema ... ok
+test agent::tests::ralph_report_semantics_and_handoff_bound_are_enforced ... ok
+test agent::tests::present_svg_parsers_recover_exact_source_for_both_model_transports ... ok
+test agent::tests::model_adapters_preserve_primitives_and_report_recoverable_syntax_errors ... ok
+test agent::tests::risky_shell_commands_require_one_time_approval ... ok
+test agent::tests::generated_schemas_match_the_executable_tool_contracts ... ok
+test agent::tests::scripted_ralph_backend_gets_fresh_prompts_and_only_validated_handoffs ... ok
+test agent::tests::project_boundary_rejects_parent_paths ... ok
+test agent::tests::repeated_call_advisories_use_canonical_arguments_and_reset_per_user_turn ... ok
+test agent::tests::ralph_report_schemas_are_child_only_and_parse_for_both_models ... ok
+test a2ui::export::tests::static_svg_renderer_covers_the_complete_catalog ... ok
+test agent::tests::stored_tool_cards_restore_ai_approval_reasons_by_call ... ok
+test agent::tests::tool_output_keeps_terminal_lines_but_parameter_summaries_do_not ... ok
+test agent::tests::present_svg_executes_without_side_effects_and_maps_only_completed_cards ... ok
+test agent::web::tests::browser_helpers_preserve_ds4_wire_values ... ok
+test agent::web::tests::headless_browser_extracts_a_local_page ... ignored, requires an installed Chrome browser
+test agent::tests::glob_confinement_skips_symlinks_and_honors_cancellation ... ok
+test agent::tests::validation_is_precise_and_prevents_side_effects_until_corrected ... ok
+test agent::tests::present_svg_validation_enforces_the_static_vector_boundary ... ok
+test agent::tests::prompts_and_parsers_expose_the_reference_tool_set ... ok
+test app::generation::tests::attached_image_marker_is_model_only ... ok
+test agent::tests::successful_structured_file_tools_report_project_paths ... ok
+test agent::tests::glob_finds_all_regular_file_names_deterministically_without_reading_them ... ok
+test agent::tests::standard_agent_skills_are_discovered_prompted_and_read_only ... ok
+test agent::tests::shell_commands_skip_user_startup_files ... ok
+test app::generation::tests::plain_chat_uses_the_same_unmodified_prompt_in_ui_and_headless ... ok
+test app::generation::tests::queued_guidance_is_one_reference_style_user_turn ... ok
+test app::generation::tests::title_context_starts_at_the_first_user_and_ignores_system_rows ... ok
+test app::generation::tests::turn_summary_accumulates_model_continuations ... ok
+test agent::tests::standard_file_tools_can_maintain_only_managed_dev_brain_pages ... ok
+test agent::tests::shell_environment_is_loaded_once_from_login_and_interactive_startup_files ... ok
+test agent::tests::ralph_round_contexts_are_unique_and_child_jobs_are_cleaned_up ... ok
+test app::git::tests::unified_diff_parser_aligns_replacements_for_split_view ... ok
+test app::git::tests::configured_git_diff_options_change_the_rendered_patch ... ok
+test app::git::tests::unstage_works_before_the_first_commit ... ok
+test app::preferences::tests::runtime_config_updates_after_lock_poisoning ... ok
+test app::git::tests::libgit2_status_merges_staged_and_worktree_changes ... ok
+test app::projects::tests::discarding_a_session_removes_every_checkpoint_stage ... ok
+test app::projects::tests::draft_titles_follow_the_stored_session_count ... ok
+test agent::tests::denial_and_stop_cancel_commands_awaiting_approval ... ok
+test app::projects::tests::session_states_are_exclusive_and_ordered ... ok
+test app::projects::tests::sweeping_removes_checkpoints_of_sessions_that_no_longer_exist ... ok
+test app::projects::tests::git_state_lists_and_switches_local_branches ... ok
+test app::tests::background_activity_drives_polling_indicators_and_quit_confirmation ... ok
+test agent::tests::shell_environment_probe_stops_blocked_startup_files ... ok
+test app::git::tests::origin_fetch_pull_and_push_use_the_current_branch ... ok
+test app::tests::chat_html_export_embeds_loaded_a2ui_images_as_data_urls ... ok
+test agent::tests::oversized_shell_output_is_bounded_and_retained_until_cleanup ... ok
+test app::tests::chat_only_follows_updates_while_near_the_tail ... ok
+test app::tests::ds4_gib_and_streaming_cache_inputs_are_typed ... ok
+test app::tests::one_shot_titles_are_reduced_to_a_single_sidebar_line ... ok
+test app::tests::owned_runtime_paths_stay_in_application_support ... ok
+test app::tests::preferences_shortcut_and_dspark_support_are_explicit ... ok
+test app::tests::source_does_not_depend_on_a_sibling_ds4_checkout ... ok
+test app::tests::stored_session_always_selects_its_model ... ok
+test app::tests::tab_scrolls_a_field_back_into_the_preferences_viewport ... ok
+test app::view::a2ui::tests::renderer_input_helpers_cover_dates_numbers_choices_and_weights ... ok
+test app::view::chat::tests::chat_divider_groups_token_counts_by_thousands ... ok
+test app::view::chat::tests::composer_uses_shift_enter_for_line_breaks ... ok
+test app::view::git::tests::split_diff_width_covers_the_longest_side_and_tabs ... ok
+test app::tests::chat_export_preserves_completed_svg_and_keeps_failures_generic ... ok
+test app::view::tests::action_buttons_share_geometry ... ok
+test app::view::tests::context_pie_wedge_follows_the_fill_grade ... ok
+test app::view::tests::download_measurements_are_readable ... ok
+test app::view::tests::raised_surfaces_stay_dark ... ok
+test compaction::tests::a_new_summary_replaces_the_previous_rebuild_summary ... ok
+test compaction::tests::hard_trigger_and_retry_reserve_match_the_reference ... ok
+test compaction::tests::soft_trigger_matches_reference_thresholds ... ok
+test compaction::tests::summary_budget_keeps_answer_room_and_rejects_exhausted_contexts ... ok
+test compaction::tests::summary_control_markup_is_private ... ok
+test compaction::tests::tail_prefers_the_first_user_boundary_after_target ... ok
+test config::tests::defaults_write_an_empty_file_and_reload_unchanged ... ok
+test config::tests::deprecated_flash_config_migrates_and_persists_only_0731 ... ok
+test config::tests::explicit_0731_dspark_opt_out_survives_the_default ... ok
+test config::tests::invalid_files_are_reported_instead_of_reset ... ok
+test app::preferences::tests::selectors_rehydrate_their_generation_and_model_profiles ... ok
+test config::tests::model_profiles_expose_only_their_upstream_thinking_modes ... ok
+test config::tests::legacy_flat_model_settings_are_dropped_but_the_prompt_is_kept ... ok
+test config::tests::only_changed_values_are_written_and_read_back ... ok
+test config::tests::qwen_selection_and_native_defaults_survive_restart ... ok
+test app::preferences::tests::multiline_system_prompt_round_trips_through_the_editor ... ok
+test app::generation::tests::forced_compaction_requires_new_visible_chat ... ok
+test app::tests::assistant_stream_keeps_reasoning_separate_from_the_answer ... ok
+test app::generation::tests::legacy_generation_rows_collapse_into_their_user_turn ... ok
+test app::tests::chat_export_formats_visible_turns_reasoning_and_bash_tools ... ok
+test app::generation::tests::model_history_uses_persisted_content_without_changing_the_transcript ... ok
+test app::view::stats::tests::current_session_summary_uses_persisted_chat_facts ... ok
+test app::generation::tests::last_compaction_selects_its_tail_without_hiding_history ... ok
+test app::generation::tests::assistant_markdown_keeps_llm_tables ... ok
+test app::generation::tests::a2ui_auto_switch_waits_for_a_renderable_surface ... ok
+test app::generation::tests::complete_tool_call_in_reasoning_gets_one_correction ... ok
+test app::generation::tests::extension_context_deduplicates_session_start_and_rearms_after_compaction ... ok
+test app::tests::chat_html_export_is_safe_standalone_and_renders_embedded_surfaces ... ok
+test dev_brain::tests::skill_index_migration_does_not_overwrite_an_unrelated_root_note ... ok
+test dev_brain::tests::path_confinement_and_unrelated_notes_are_preserved ... ok
+test dsml::tests::parses_reference_and_legacy_dsml_syntaxes ... ok
+test dsml::tests::recovers_complete_invokes_before_an_incomplete_one ... ok
+test dsml::tests::recovers_complete_invokes_without_the_outer_closing_tag ... ok
+test engine::ds4_sampling::tests::heap_fallback_preserves_rng_and_signed_zero_ties ... ok
+test engine::gguf::tests::accepts_exact_mxfp4_block_layout ... ok
+test engine::gguf::tests::keeps_unknown_tensor_kinds_rejected ... ok
+test engine::gguf::tests::maps_metadata_and_tensor_payload_without_copying_weights ... ok
+test engine::gguf::tests::mxfp4_scalar_dot_matches_independent_values ... ok
+test engine::gguf::tests::rejects_mxfp4_dimension_overflow ... ok
+test engine::gguf::tests::rejects_mxfp4_rows_that_are_not_block_aligned ... ok
+test engine::gguf::tests::rejects_truncated_mxfp4_payload ... ok
+test engine::kvstore::tests::budget_evicts_the_least_valuable_unprotected_entry ... ok
+test engine::kvstore::tests::longest_compatible_prefix_wins_and_hits_survive_reopen ... ok
+test engine::kvstore::tests::opening_reclaims_checkpoints_the_index_cannot_reach ... ok
+test engine::metal::glm::tests::dsa_indexer_schedule_matches_the_reference ... ok
+test engine::metal::glm::tests::glm53_m5_text_performance_gate ... ignored, requires the installed GLM 5.3 Flash Q2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm53_m5_vision_performance_and_oracle_gate ... ignored, requires GLM 5.3 Q2, its vision sidecar, a fixed image, and Apple Metal
+test engine::metal::glm::tests::glm53_reference_logits_replay_separates_sampling_from_execution ... ignored, requires the installed GLM 5.3 Flash Q2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm53_reference_prompt_tokens_match_shared_runtime ... ignored, installed GLM and DS4SERVER_GLM_REFERENCE_LOG from the standalone reference
+test engine::metal::glm::tests::glm53_two_row_verifier_restores_recurrent_state ... ignored, installed GLM 5.3 and Metal; run under test-supervisor
+test engine::metal::glm::tests::glm_byte_budget_reserves_prefill_before_dynamic_experts ... ok
+test engine::metal::glm::tests::glm_mtp_preserves_target_tokens_and_drafts ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm_prefill_schedule_matches_ds4_boundaries ... ok
+test engine::metal::glm::tests::glm_resident_decode_flushes_periodically_without_a_final_or_ssd_flush ... ok
+test engine::metal::glm::tests::repeated_glm_prompt_rewinds_one_token_for_logits ... ok
+test engine::metal::glm::tests::resident_and_streamed_glm_match_ds4_decode_oracles ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::glm::tests::streamed_glm_uses_ds4_indexed_prefill_for_long_prompts ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::gpu::qwen_mtplx_allocation_tests::backend_allocation_policy_is_explicit_inside_and_outside_qwen_scope ... ignored, requires Apple Metal; allocation only, no GPU dispatch or model
+test engine::metal::gpu::qwen_mtplx_allocation_tests::dispatch_retains_allocation_without_changing_data_donation ... ignored, requires Apple Metal, no model weights
+test engine::metal::gpu::qwen_mtplx_allocation_tests::empty_storage_preserves_data_identity_views_and_zero_allocator_usage ... ignored, requires Apple Metal context; empty descriptors, no GPU allocation
+test engine::metal::gpu::qwen_mtplx_allocation_tests::mtplx_tensor_cpu_access_releases_native_storage ... ignored, requires Apple Metal; CPU buffer access only, no GPU dispatch or model
+test engine::metal::markov::tests::installed_markov_worker_dispatch ... ignored, requires installed 0731 DSpark support; CPU dispatch diagnostic, not parity acceptance
+test engine::metal::markov::tests::markov_workers_reuse_threads_and_buffers_with_ordered_ties ... ok
+test engine::metal::power_throttle_keeps_ds4_ewma_and_full_power_bypass ... ok
+test database::tests::context_work_reactivates_an_archived_session ... ok
+test engine::metal::qwen::tests::affine_split_k_matches_reference_shapes_and_scratch_bound ... ok
+test engine::metal::qwen::tests::dense_attention_crossover_is_phase_specific ... ok
+test engine::metal::qwen::tests::qsa_selection_and_capacity_cover_the_full_native_window ... ok
+test engine::metal::qwen::tests::qwen_affine_qmv_and_qmm_match_mlx_reference ... ignored, requires the pinned Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_affine_qmv_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_core_boundary_and_checkpoint_are_stable ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_fused_gdn_input_rows_match_separate_projections ... ignored, requires the pinned Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_gdn_conv_norm_rows_matches_mtplx ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_gdn_delta_rows_matches_mtplx ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_layer_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_lazy_target_probabilities_match_batched_sampling ... ignored, requires the pinned Qwen metadata and Apple Metal; no model inference
+test engine::metal::qwen::tests::qwen_m5_performance_gate ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_metal_primitives_match_reference_vectors ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_mtp_matches_target_and_restores_its_cache ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_ple_gather_copies_only_the_requested_row ... ok
+test database::tests::a2ui_dismissal_persists_a_fresh_surface_boundary ... ok
+test engine::metal::qwen::tests::qwen_ple_hash_contract_matches_golden_boundaries ... ok
+test engine::metal::qwen::tests::qwen_qsa_dense_mask_accepts_unsorted_blocks ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_full_context_preallocation_succeeds ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_logical_depth_oracles_cover_prefill_decode_and_state ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_real_layer_executes_the_first_sparse_token ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_selection_and_attention_remain_bounded_at_native_depths ... ignored, requires Apple Metal and allocates the full QSA score window
+test engine::metal::qwen::tests::qwen_reference_layer_trace ... ignored, diagnostic trace against the pinned MLX-LM reference
+test engine::metal::qwen::tests::qwen_reference_sdpa_trace ... ignored, diagnostic comparison with captured MTPLX SDPA inputs
+test engine::metal::qwen::tests::qwen_reference_stateful_trace ... ignored, diagnostic trace against the pinned MLX-LM reference
+test engine::metal::qwen::tests::qwen_reference_verify_trace ... ignored, diagnostic trace against the pinned MTPLX verifier
+test engine::metal::qwen::tests::qwen_sorted_routed_prefill_matches_grouped_output ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_stage_counters_preserve_async_batches ... ignored, requires pinned Qwen embeddings and Apple Metal; stage profiler opt-in
+test database::tests::sessions_are_ordered_by_recent_use_within_sidebar_groups ... ok
+test engine::metal::qwen::tests::qwen_target_submission_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_verify_layer_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen_mtplx::allocator::mtplx_allocations_preserve_logical_bounds_with_reference_rounding ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::allocator::mtplx_native_allocator_heap_limits_and_gpu_lifetime ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx::allocator::mtplx_native_allocator_replays_reference_allocations ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::ar::mtplx_ar_driver_matches_reference ... ignored, requires Apple Metal; unchanged MTPLX generate_ar orchestration oracle
+test engine::metal::qwen_mtplx::array::mtplx_array_detach_clears_every_siblings_inputs_and_primitive ... ok
+test engine::metal::qwen_mtplx::array::mtplx_array_materialization_keeps_geometry_and_rejects_short_storage ... ignored, requires Apple Metal; typed materialized descriptors without model weights
+test engine::metal::qwen_mtplx::array::mtplx_array_layout_preserves_dtype_shape_stride_and_materialization_flags ... ok
+test engine::metal::qwen_mtplx::array::mtplx_dtype_promotion_matches_all_reference_pairs ... ok
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_dense_mask_graph_matches_reference ... ignored, requires Apple Metal; original MTPLX block mask including sentinel and tails
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_eager_array_graph_matches_reference ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_prefill_gather_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX tiled prefill gather
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_rows_gather_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX QSA rows-gather attention
+test engine::metal::qwen_mtplx::attention::mtplx_sdpa_fallback_graph_matches_reference ... ignored, requires Apple M5 Max; actual Qwen SDPA fallback reference
+test engine::metal::qwen_mtplx::binary::mtplx_graph_binary_preserves_reference_branches ... ignored, requires Apple Metal; binary graph broadcast/layout and donation branches
+test engine::metal::qwen_mtplx::binding::mtplx_installed_module_binding_matches_reference ... ignored, requires installed DS4_QWEN38_ARTIFACTS and SSD policy; only small norms and four table rows evaluated
+test engine::metal::qwen_mtplx::cache::mtplx_kv_cache_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX KVCache growth, trim and state receipts
+test engine::metal::qwen_mtplx::cache::mtplx_qsa_cache_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX positional cache, mirror and held state aliases
+test engine::metal::qwen_mtplx::compile_graph::mtplx_compile_scalar_bits_and_custom_kernel_identity ... ignored, requires Metal buffers and external supervisor; scalar ownership only, no GPU dispatch
+test engine::metal::qwen_mtplx::allocator::mtplx_allocator_cache_matches_live_reference_trace ... ok
+test engine::metal::qwen_mtplx::compiled::mtplx_replay_replaces_descriptors_without_rebuilding_primitives ... ok
+test engine::metal::qwen_mtplx::context_copy::prompt_copy_match_and_backoff_preserve_reference_policy ... ok
+test engine::metal::qwen_mtplx::decoder::mtplx_decoder_connected_attention_and_moe_advance_caches ... ignored, requires Apple Metal and external test-supervisor; complete decoder/cache smoke
+test engine::metal::qwen_mtplx::decoder::mtplx_decoder_residual_preserves_graph_order ... ok
+test engine::metal::qwen_mtplx::dense::mtplx_dense_attention_batches_match_reference ... ignored, requires Apple M5 Max; actual BF16 batched matmul reference
+test engine::metal::qwen_mtplx::dense::mtplx_dense_float_graph_matches_reference ... ignored, requires Apple M5 Max; floating Matmul factory and original dispatch
+test engine::metal::qwen_mtplx::dense::mtplx_qsa_score_array_graph_matches_reference ... ignored, requires Apple M5 Max; canonical QSA F32 graph with the original TF32 setting
+test engine::metal::qwen_mtplx::encoder::mtplx_empty_bindings_preserve_nil_dependencies_without_dummy_storage ... ignored, requires Apple Metal; empty original-kernel grids and nil bindings
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_barrier_epochs_and_commit_thresholds ... ok
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_commits_after_primitives_and_balances_completion ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_scope_routes_operators_and_allocator ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_shared_events_and_error_completion_order ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_untracked_dependencies_across_encoders_and_commands ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_drains_failure_preserves_cause_and_retries_only_failed_nodes ... ignored, requires Apple Metal; supervised evaluator failure and retry
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_releases_intermediates_and_preserves_descriptor_donation ... ignored, requires Apple Metal; graph ownership and last-consumer release
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_retains_tracer_topology_without_input_donation ... ignored, requires Apple Metal; evaluator tracer donation and retained topology
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_runs_cpu_gpu_siblings_and_async_continuations ... ignored, requires Apple Metal; tiny supervised CPU/GPU evaluator graph
+test database::tests::inline_svg_tool_call_survives_session_switch_compaction_and_reopen ... ok
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_preserves_bfs_width_and_sibling_choice ... ok
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_preserves_scheduled_detach_and_tracer_errors ... ok
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_ar_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_AR_REFERENCE required
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_mtp_prefill_matches_reference ... ignored, full installed MTP model; external guard and DS4_QWEN38_MTP_PREFILL_REFERENCE required
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_tracks_stream_fences_per_leaf_and_cross_device_union ... ok
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_ping_forward_matches_reference ... ignored, full installed model; requires external memory guard and DS4_QWEN38_FORWARD_REFERENCE
+test engine::metal::qwen_mtplx::execution::qwen_power_throttle_drains_before_sleep_and_bypasses_at_full_power ... ignored, requires Apple Metal; run under test-supervisor
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_constant_spelling ... ok
+test database::tests::session_model_migration_preserves_data_and_accepts_catalog ... ok
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_fusion_gpu_matches_reference ... ignored, requires Metal and external supervisor; generated fusion vs pinned SiLU plus multi-output/scalar dispatch
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_forward_matches_model_reference ... ignored, requires Apple Metal; actual MTPLX GDN forward and cache receipts
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_prefill_and_verify_match_model_reference ... ignored, requires Apple Metal; complete GDN Conv1d prefill and fused verify graphs
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_staged_primitives_match_reference ... ignored, requires Apple Metal; 80 actual delta-update receipts and six conv-normalization kernels
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_step_preserves_views_and_siblings ... ignored, requires Apple Metal; GDN custom primitive strided copies and sibling roots
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_event_snapshots_preserve_values_streams_and_shared_errors ... ignored, requires Apple Metal events, no model weights
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_fence_coherence_preserves_empty_and_partial_word_spans ... ignored, requires Apple Metal3, model-free empty coherent dispatch
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_fences_transfer_cpu_gpu_gpu_cpu_in_both_modes ... ignored, requires Apple Metal3, tiny model-free coherent fence kernels
+test engine::metal::qwen_mtplx::hyper::mtplx_graph_hyper_forward_matches_reference ... ignored, requires Apple M5 Max; actual MTPLX GatedResidual reference receipts
+test engine::metal::qwen_mtplx::hyper::mtplx_graph_hyper_pack_and_reads_match_reference ... ignored, requires Apple Metal; original hyper primitives and runtime pack
+test engine::metal::qwen_mtplx::indexing::mtplx_graph_indexing_preserves_layouts_and_offsets ... ignored, requires Apple Metal; Select layout branches, GatherAxis views and empty arguments
+test engine::metal::qwen_mtplx::indexing::mtplx_scatter_axis_graph_matches_reference ... ignored, requires Apple Metal; pinned Bool ScatterAxis layout and broadcast paths
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_numeric_receipts ... ignored, #103 diagnostic; Metal required, eager versus compiled numeric operands
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_sampling_receipts ... ignored, #103 diagnostic; Metal required, pins currently broken sampling law and invalid rows
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_sorted_gather_receipts ... ignored, #103 diagnostic; M5 and pinned runtime shader required; initialized sentinels
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_tokenizer_receipts ... ignored, #103 diagnostic; installed tokenizer and latest oracle JSONL; no GPU
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_vision_position_receipts ... ignored, #103 diagnostic; small active attention module on Metal; mixed position boundary
+test engine::metal::qwen_mtplx::issue104::issue104_installed_ar_stream_and_state ... ignored, full pinned Bare-Speed model; external memory guard required
+test engine::metal::qwen_mtplx::issue105::issue105_batched_greedy_rejects_each_invalid_row ... ignored, Metal required; run serially under test-supervisor
+test engine::metal::qwen_mtplx::issue106::issue106_complete_state_and_checkpoint_recovery ... ignored, installed model and actual pre-fix checkpoint; external memory guard required
+test engine::metal::qwen_mtplx::issue106::issue106_upstream_exit_matrix ... ignored, installed model, scripted logits on real caches; external memory guard required
+test engine::metal::qwen_mtplx::issue108::issue108_attention_chunk_positions ... ignored, Metal and pinned MTPLX chunk fixtures; external supervisor required
+test engine::metal::qwen_mtplx::issue109::issue109_gate_phase_and_dtype_laws ... ignored, Metal and exhaustive pinned gate fixtures; external supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_actual_projection_rows_match_oracle ... ignored, installed layer weights and pinned oracle receipts; external memory supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_guard_allocation_refusal_releases_graph_and_retries ... ignored, Metal allocation refusal and retry; external supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_large_cold_and_warm_prefill ... ignored, full installed model and large partial chunks; serial memory supervisor required
+test engine::metal::qwen_mtplx::linear::mtplx_graph_affine_dequantize_matches_reference ... ignored, requires Apple Metal; original table dequantize dtype/group/bit/layout receipts
+test engine::metal::qwen_mtplx::linear::mtplx_graph_affine_quantize_matches_reference ... ignored, requires Apple Metal; original affine quantize dtype/group/bit/layout receipts
+test engine::metal::qwen_mtplx::linear::mtplx_graph_gather_qmm_matches_reference ... ignored, requires Apple Metal; original GatherQMM array layouts and expert routing
+test engine::metal::qwen_mtplx::linear::mtplx_graph_quantized_linear_batched_identity ... ignored, requires Apple Metal and external supervisor; strided/broadcast batched affine dispatch
+test engine::metal::qwen_mtplx::linear::mtplx_graph_quantized_linear_matches_reference ... ignored, requires Apple M5 Max Metal; 78 pinned affine fixtures through lazy graph
+test engine::metal::qwen_mtplx::linear::sorted_gather_guard_matches_pinned_call_predicates ... ok
+test engine::metal::qwen_mtplx::load::mtplx_safetensors_load_laziness_boundaries_and_lifetime ... ignored, requires Apple Metal; bounded CPU Load and dependent GPU graph
+test engine::metal::qwen_mtplx::mlp::mtplx_graph_shared_mlp_matches_reference ... ignored, requires Apple M5 Max; actual MTPLX shared MLP reference receipts
+test engine::metal::qwen_mtplx::mlp::mtplx_graph_split_and_compiled_silu_layouts ... ignored, requires Apple Metal; Split sibling ownership and compiled SiLU layouts
+test database::tests::migrations_and_crud_work ... ok
+test engine::metal::qwen_mtplx::model::mtplx_model_compile_policy_and_run_boundaries ... ok
+test engine::metal::qwen_mtplx::moe::mtplx_graph_sparse_moe_matches_reference ... ignored, requires Apple M5 Max; full MoE Array graph, no model weights; use test-supervisor
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_embedding_matches_reference ... ignored, requires Apple Metal; full resident NGram graph and lazy decode history
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_gpu_rows_and_history_match_reference ... ignored, requires Apple Metal; original lazy NGram GPU row graph, no table
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_resident_matches_reference ... ignored, requires Apple Metal; original NGramTable resident gather/dequantize receipts
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_table_priority ... ignored, requires Apple Metal; NGram table routing and missing-sidecar behavior
+test engine::metal::qwen_mtplx::ngram_sidecar::mtplx_ngram_sidecar_matches_reference ... ignored, requires Apple Metal; actual file-backed MTPLX sidecar/LRU/dequantization receipts
+test engine::metal::qwen_mtplx::ngram_stage::mtplx_ngram_stage_allclose_branches ... ignored, requires Apple Metal; stage verification special values and original reduction branches
+test engine::metal::qwen_mtplx::ngram_stage::mtplx_ngram_stage_matches_reference ... ignored, requires Apple Metal; original complete stage lifecycle and file-backed tables
+test engine::metal::qwen_mtplx::normalization::mtplx_graph_norm_modules_match_reference ... ignored, requires Apple Metal; connected Qwen normalization graph golden fixtures
+test engine::metal::qwen_mtplx::normalization::mtplx_graph_rms_norm_preserves_reference_layout_and_values ... ignored, requires Apple Metal; graph RMSNorm donation, copy, broadcast and looped kernels
+test engine::metal::qwen_mtplx::normalization::mtplx_logsumexp_graph_matches_reference ... ignored, requires Apple Metal; original sampler LogSumExp branches
+test engine::metal::qwen_mtplx::normalization::mtplx_softmax_graph_storage_matches_reference ... ignored, requires Apple Metal; original Softmax type and storage branches
+test engine::metal::qwen_mtplx::ops::mtplx_concatenate_preserves_general_general_copy_semantics ... ignored, requires Apple Metal; Concatenate promotion, offsets, strided writers and validation
+test engine::metal::qwen_mtplx::ops::mtplx_owned_operations_build_and_execute_qsa_layout_graph ... ignored, requires Apple Metal; self-contained lazy QSA-layout recipe, no callback map
+test engine::metal::qwen_mtplx::ops::mtplx_slice_and_contiguous_preserve_cache_window_semantics ... ignored, requires Apple Metal; normalized Slice and Contiguous cache-window graph
+test engine::metal::qwen_mtplx::fused::mtplx_compile_fusion_preserves_parent_and_output_boundaries ... ok
+test engine::metal::qwen_mtplx::owner::issue105_product::issue105_product_ar_cleanup_and_http_retry ... ignored, installed Qwen model; serial test-supervisor with memory guard required
+test engine::metal::qwen_mtplx::owner::issue105_product::issue105_product_mtp_cleanup_and_http_retry ... ignored, installed Qwen MTP model; serial test-supervisor with memory guard required
+test engine::metal::qwen_mtplx::owner::issue106_product::issue106_product_discards_old_checkpoint_and_recovers_cold ... ignored, installed model and actual pre-fix checkpoint; external memory guard required
+test engine::metal::qwen_mtplx::owner::issue107_product::issue107_vision_product ... ignored, installed model plus vision tower; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::issue109_product::issue109_verify_rows_and_warm_checkpoint ... ignored, installed model; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::issue111_product::issue111_token_prefix_cache ... ignored, installed model; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::qwen_product_generator_chat_matches_reference ... ignored, full installed model through product Generator; external supervisor and chat reference required
+test engine::metal::qwen_mtplx::owner::qwen_vision_chat_checkpoint_preserves_image_identity ... ignored, requires installed Qwen, Metal, and DS4_QWEN_VISION_IMAGE; run under test-supervisor
+test engine::metal::qwen_mtplx::ple::mtplx_graph_ple_connected_matches_reference ... ignored, requires Apple M5 Max; complete original PLELayer with real lookup
+test engine::metal::qwen_mtplx::ple::mtplx_graph_ple_projection_and_cache_match_reference ... ignored, requires Apple M5 Max; actual MTPLX PLELayer after embedding lookup
+test engine::metal::qwen_mtplx::owner::qwen_mtp_progress_publishes_without_turn_completion_or_double_counting ... ok
+test engine::metal::qwen_mtplx::policy::qwen_server_policy_matches_original_functions ... ignored, requires DS4_QWEN38_POLICY_REFERENCE from pinned MTPLX; CPU-only
+test engine::metal::qwen_mtplx::prefill::mtplx_mtp_prefill_driver_matches_reference ... ignored, requires Apple Metal; original streaming MTP history prefill
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_emitter_matches_all_pinned_silu_bodies ... ok
+test engine::metal::qwen_mtplx::prefill::mtplx_prefill_driver_matches_reference ... ignored, requires Apple Metal; original _prefill control flow and raw cache roots
+test engine::metal::qwen_mtplx::qsa_attention::mtplx_qsa_attention_module_matches_reference ... ignored, requires Metal and unchanged whole-Attention receipts; no model download
+test engine::metal::qwen_mtplx::qsa_compile::mtplx_compiled_indexer_array_replay_matches_reference ... ignored, requires Metal and original compiled indexer receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_core::mtplx_qsa_core_host_matches_reference ... ignored, requires Metal and original host-core receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_cache_donation_requires_descriptor_data_and_allocation_contract ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_completion_owns_buffers_until_gpu_finishes ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_evaluator_retains_inputs_and_siblings_not_primary_data ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_indexer::issue107::issue107_real_geometry_query_and_block_start_positions ... ignored, Metal and independently verified pinned indexer fixtures; external supervisor required
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_canonical_indexer_routes_match_reference ... ignored, requires Metal and original ongoing indexer routing receipts; no weights
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_indexer_compiled_commit_matches_reference ... ignored, requires Metal and original indexer lifecycle receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_indexer_prefix_matches_reference ... ignored, requires Apple Metal, original stateful QSAIndexer receipts, no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_schedule_preserves_the_selected_sibling ... ok
+test engine::metal::qwen_mtplx::qsa_prefill::mtplx_qsa_prefill_graph_matches_reference ... ignored, requires Metal and original two-stage prefill receipts, no weights
+test engine::metal::qwen_mtplx::qsa_prepare::mtplx_qsa_prepare_graph_matches_reference ... ignored, requires Apple Metal; parameterized original QSA preparation and traced positions
+test engine::metal::qwen_mtplx::policy::retired_ar_pipeline_flag_cannot_change_server_or_turn_policy ... ok
+test engine::metal::qwen_mtplx::qsa_select::mtplx_qsa_fused_graph_matches_reference ... ignored, requires Apple Metal and original QSA selector receipts; no model weights
+test engine::metal::qwen_mtplx::random::mtplx_sampler_prefix_matches_reference ... ignored, requires Apple Metal; original target-prefix sampler and default RNG
+test engine::metal::qwen_mtplx::random::mtplx_sampler_resident_matches_reference ... ignored, requires Apple Metal; original MTPLX resident sampling and keyed RNG
+test engine::metal::qwen_mtplx::reduce::mtplx_sampler_argmax_matches_reference ... ignored, requires Apple Metal; original MTPLX greedy ArgReduce and storage
+test engine::metal::qwen_mtplx::reduce::mtplx_sum_graph_matches_reference ... ignored, requires Apple Metal; original shared sum dispatch and QSA mean
+test engine::metal::qwen_mtplx::residency::mtplx_residency_matches_reference_lifecycle_cases ... ignored, requires Apple Metal 3 / macOS 15, no model weights
+test engine::metal::qwen_mtplx::rope::mtplx_rope_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX RoPE, static YaRN and multi-axis receipts
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_batched_matches_reference ... ignored, requires Apple Metal; original MTPLX bound and serial batch distributions
+test engine::metal::qwen_mtplx::request::qwen_request_settings_override_backend_policy ... ok
+test engine::metal::qwen_mtplx::request::qwen_request_progress_uses_ui_frontiers_and_decode_timer ... ok
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_entry_matches_reference ... ignored, requires Apple Metal; actual MTPLX generator sampling entry
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_fallback_matches_reference ... ignored, requires Apple Metal; original sparse fallback and host dense sampler
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_penalties_match_reference ... ignored, requires Apple Metal; original GPU and CPU raw-logit penalties
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_selection_matches_reference ... ignored, requires Apple Metal; original MTPLX sampler selection, no model weights
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_serial_support_matches_reference ... ignored, requires Apple Metal; actual MTPLX serial candidate route, no weights
+test engine::metal::qwen_mtplx::compile_graph::mtplx_compile_simplify_preserves_reference_parent_rules ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_cpu_primitive_holds_data_and_temporaries_in_fifo_order ... ignored, requires Apple Metal shared storage, no GPU dispatch or model weights
+test engine::metal::qwen_mtplx::scheduler::mtplx_cpu_dispatch_counts_tenth_and_completes_after_failed_work ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_cpu_events_propagate_errors_without_consuming_source ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_streams_preserve_fifo_errors_and_shutdown ... ok
+test engine::metal::qwen_mtplx::sdpa::mtplx_sdpa_vector_matches_reference ... ignored, requires Apple M5 Max; actual pinned fused SDPA kernels
+test engine::metal::qwen_mtplx::session_cache::mtplx_full_installed_chat_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_CHAT_REFERENCE required
+test engine::metal::qwen_mtplx::session_cache::mtplx_session_boundary_policy ... ok
+test engine::metal::qwen_mtplx::session_cache::mtplx_session_final_state_requires_capture_and_safe_cache ... ok
+test engine::metal::qwen_mtplx::session_checkpoint::qwen_checkpoint_roundtrip_and_failed_restore_preserve_live_state ... ignored, model-free Metal checkpoint; external supervisor and DS4_QWEN38_CHECKPOINT_DIR required
+test engine::metal::qwen_mtplx::session_codec::mtplx_session_codec_host_import_matches_reference ... ignored, model-free Metal imports; external supervisor and DS4_HOST_IMPORT_REFERENCE required
+test engine::metal::qwen_mtplx::session_codec::mtplx_session_codec_matches_original_blobs_and_restore ... ignored, tiny GPU arrays; external supervisor and DS4_SESSION_CODEC_REFERENCE required
+test engine::metal::qwen_mtplx::sort::mtplx_sort_graph_matches_reference ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::sparse_attention::mtplx_sparse_attention_graph_matches_reference ... ignored, requires Apple M5; actual MTPLX sparse Decode and TensorOps Prefill consumers
+test engine::metal::qwen_mtplx::speculative::mtplx_full_installed_mtp_generation_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_MTP_GENERATION_REFERENCE required
+test engine::metal::profile::tests::profile_reports_locality_and_lru_hits ... ok
+test engine::metal::qwen_mtplx::stream::mtplx_stream_routes_queues_events_and_finalizes_all_under_pressure ... ignored, requires Apple Metal, tiny model-free two-queue copy
+test engine::metal::qwen_mtplx::stream::mtplx_stream_registry_preserves_thread_defaults_templates_and_clear ... ok
+test engine::metal::qwen_mtplx::submission::mtplx_submission_attaches_at_flush_eval_finish_and_drop ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::unary::mtplx_graph_unary_preserves_reference_layout_and_values ... ignored, requires Apple Metal; graph unary layout, donation and kernel boundaries
+test engine::metal::qwen_mtplx::views::mtplx_dtype_conversion_and_views_preserve_values_layout_and_donation ... ignored, requires Apple Metal; original conversion/view kernels and donation in lazy eval
+test engine::metal::qwen_mtplx::views::mtplx_dynamic_slice_graph_matches_reference ... ignored, requires Metal and original dynamic slice receipts; no model weights
+test engine::metal::qwen_mtplx::views::mtplx_reshape_shares_or_copies_in_the_connected_evaluator ... ignored, requires Apple Metal; original Reshape copy kernels and shared backing
+test engine::metal::qwen_mtplx::views::mtplx_views_preserve_aliases_offsets_strides_and_gpu_output ... ignored, requires Apple Metal; connected zero-copy views and offset-aware original copy kernel
+test engine::metal::qwen_mtplx::vision::qwen_vision_tower_matches_mtplx_image ... ignored, requires Metal and pinned Qwen vision artifacts/reference values
+test engine::metal::qwen_mtplx::weights::mtplx_checkpoint_projection_fusions_match_reference ... ignored, requires Apple Metal; bounded lazy checkpoint Load and fusion
+test engine::metal::qwen_mtplx::weights::mtplx_installed_checkpoint_fusion_metadata_matches_reference ... ignored, requires installed DS4_QWEN38_ARTIFACTS; headers only, never evaluate weights
+test engine::metal::qwen_mtplx_tests::mtplx_affine_selection_preserves_reference_boundaries ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_canonical_hyper_and_gdn_kernels_match_reference ... ignored, requires Apple Metal, no model weights
+test database::tests::chat_and_compaction_history_survive_reopen_and_session_deletion ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_connected_delta_update_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gather_sort_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_decode_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_forward_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_staged_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_norm_modules_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_router_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_sparse_moe_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_switch_glu_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_gate_up_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_cache_lifecycle_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_backings_and_dynamic_copies_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_cache_routing_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_cache_window_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_capacity_matches_reference ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_indexer_modes_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_scalar_sources_match_reference ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_connected_cache_selection_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_eager_topk_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_f32_scores_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_indexer_preparation_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_indexer_routes_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_producer_dispatch_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_canonical_source_bodies_preserve_pinned_hashes ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_selector_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_affine_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_argsort_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_gather_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx::submission::mtplx_all_native_command_commits_pass_the_submission_hook ... ok
+test engine::metal::tests::batch_raw_span_includes_the_chunk_and_previous_window ... ok
+test engine::metal::tests::compression_schedule_tracks_the_deepseek_model_shape ... ok
+test engine::metal::tests::decode_feature_policy_keeps_device_generations_and_rollbacks_separate ... ok
+test engine::metal::tests::decode_raw_span_tracks_the_logical_sliding_window ... ok
+test engine::metal::tests::directional_steering_matches_the_ds4_token_oracle ... ignored, requires the installed Flash GGUF, DS4 steering fixture, and a Metal device
+test engine::metal::tests::dspark_confidence_quantizes_q8_input_like_ds4 ... ok
+test engine::metal::tests::dspark_markov_activation_matches_q8_rounding ... ok
+test engine::metal::tests::dspark_matches_original_summary_cycles ... ignored, requires installed 0731/DSpark and DS4SERVER_DSPARK_REFERENCE cycle recording
+test engine::metal::tests::dspark_runs_a_target_owned_greedy_cycle ... ignored, requires the installed 0731 Flash and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::dspark_scheduler_matches_the_ds4_default_window ... ok
+test engine::metal::tests::dspark_verifier_selects_only_the_committed_row ... ok
+test engine::metal::tests::flash_0731_long_context_crosses_indexed_prefill_boundary ... ignored, requires a 0731 Flash GGUF and an Apple M5 device
+test engine::metal::tests::flash_0731_m5_decode_performance_gate ... ignored, requires a 0731 Flash GGUF and an Apple M5 device
+test engine::metal::tests::flash_0731_resident_and_ssd_streaming_choose_the_same_tokens ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::flash_0731_runs_exact_sampled_dspark ... ignored, requires the installed 0731 target and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::flash_0731_ssd_streaming_maps_and_seeds_batched_prefill_layers ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::generated_hotlists_match_each_model_shape ... ok
+test engine::metal::tests::half_conversion_handles_normal_and_subnormal_values ... ok
+test engine::metal::tests::pro_q4_model_spans_remain_isolated ... ok
+test engine::metal::tests::pro_resident_admission_includes_weights_context_and_scratch ... ok
+test engine::metal::tests::pro_ssd_streaming_full_layer_and_selected_batch_agree ... ignored, requires the installed DeepSeek V4 Pro GGUF and Apple Metal
+test engine::metal::tests::resident_multi_session_switching_preserves_each_kv_frontier ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::ssd_streaming_supports_dspark ... ignored, requires the installed 0731 Flash and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::streaming_memory_plan_matches_ds4_graph_formulas ... ok
+test engine::metal::vision::tests::glm53_resize_matches_reference_token_grids ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_activity_preserves_reference_pressure_and_wait_rules ... ok
+test engine::qwen::tests::metadata_and_safetensors_boundaries_fail_closed ... ok
+test engine::qwen::tests::pinned_artifact_set_loads_and_renders_goldens ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test engine::qwen::tests::pinned_qwen_metadata_matches_product_without_executor_state ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test engine::qwen_vision::qwen_image_cache_keys_distinguish_pixels_and_preserve_prior_text ... ok
+test engine::qwen::tests::frozen_inventory_and_memory_categories_are_exact ... ok
+test engine::qwen_vision::qwen_vision_positions_and_resize_contract ... ok
+test engine::sampling_tests::checkpoint_headers_identify_the_exact_model ... ok
+test engine::sampling_tests::checkpoint_rebuilds_explain_compatibility_and_history_misses ... ok
+test engine::sampling_tests::checkpoint_tag_accepts_an_unchanged_prefix_before_reminders ... ok
+test engine::sampling_tests::checkpoint_tag_covers_the_canonical_chat_state ... ok
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_distribution_and_rng_match_reference ... ok
+test engine::sampling_tests::ds4_sampler_real_logits_cpu_benchmark ... ignored, CPU-only; requires DS4_SAMPLER_RECORDING with the recorded GLM logits
+test engine::sampling_tests::exact_delta_sampling_accepts_or_corrects_the_draft ... ok
+test engine::sampling_tests::generated_text_drains_incomplete_bytes_before_phase_changes_and_finish ... ok
+test engine::sampling_tests::generated_text_preserves_thinking_utf8_and_split_stops ... ok
+test engine::sampling_tests::metal_executes_real_flash_token ... ignored, requires the 80 GiB Flash checkpoint and Apple Metal
+test dev_brain::tests::parser_and_index_rebuild_ignore_links_in_code ... ok
+test engine::sampling_tests::qwen_rng_matches_numpy_pcg64 ... ok
+test engine::sampling_tests::qwen_sampling_uses_mtplx_global_nucleus_mass_before_top_k ... ok
+test engine::sampling_tests::resumed_prefill_crossover_matches_each_ds4_graph ... ok
+test engine::sampling_tests::sampling_probabilities_match_ds4_filter_order ... ok
+test engine::sampling_tests::split_utf8_token_bytes_are_joined_before_decoding ... ok
+test engine::sampling_tests::top_k_and_stream_stops_are_applied_before_output ... ok
+test engine::sampling_tests::workspace_instruction_identity_participates_in_append_only_kv_keys ... ok
+test engine::sampling_tests::zero_temperature_is_greedy ... ok
+test engine::tokenizer::issue111::issue111_ds4_families ... ignored, CPU-only; installed DS4 GGUFs and exact oracle IDs
+test engine::tokenizer::issue111::issue111_every_unicode_scalar_split ... ignored, CPU-only; exhaustive pinned Qwen split receipt
+test engine::tokenizer::issue111::issue111_qwen_tokens_and_requests ... ignored, CPU-only; installed Qwen tokenizer and pinned exact IDs
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_workspace_matches_reference ... ok
+test engine::tokenizer::tests::character_helpers_are_total_at_every_byte_offset ... ok
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ignored, CPU-only; requires DS4SERVER_CHAT_REFERENCE and its installed GGUF
+test engine::tokenizer::tests::qwen_checkpoint_continuation_matches_full_conversation ... ignored, requires the installed Qwen tokenizer, but no GPU or model weights
+test engine::tokenizer::tests::qwen_plain_chat_matches_mtplx_server_tokens ... ignored, requires local tokenizer and tools/qwen-chat-reference.py output; no inference
+test engine::tokenizer::tests::reasoning_prefixes_match_each_model_family_contract ... ok
+test engine::validation::tests::configured_glm53_main_fixture_passes_the_exact_layout ... ok
+test engine::validation::tests::configured_glm53_vision_fixture_passes_the_exact_layout ... ok
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_cpu_rows_and_raw_history_match_reference ... ok
+test engine::validation::tests::installed_dspark_fixture_passes_the_target_layout ... ok
+test engine::validation::tests::mxfp4_is_limited_to_routed_experts ... ok
+test engine::tokenizer::issue111::qwen_split_boundaries ... ok
+test extensions::tests::duplicate_skill_names_and_update_rollback_are_deterministic ... ok
+test engine::sampling_tests::probability_ratio_speculation_preserves_the_target_distribution ... ok
+test extensions::tests::hook_output_is_bounded_typed_and_event_matched ... ok
+test dev_brain::tests::validation_names_every_drifted_source_and_requires_semantic_refresh ... ok
+test engine::metal::qwen_mtplx::array::mtplx_array_releases_sibling_cycles_and_deep_input_graphs ... ok
+test extensions::tests::pinned_ponytail_runs_all_lifecycle_modes ... ignored, downloads the pinned Ponytail reference integration
+test extensions::tests::manifest_and_paths_are_validated_without_following_escapes ... ok
+test extensions::tests::symlink_and_variable_escapes_are_rejected ... ok
+test dev_brain::tests::skill_validation_and_default_guide_restore_are_strict ... ok
+test extensions::tests::registry_persists_enable_disable_errors_and_uninstall ... ok
+test instructions::tests::changes_removals_resume_and_compaction_are_append_only ... ok
+test instructions::tests::empty_and_repaired_baselines_do_not_repeat_prompt_content ... ok
+test dev_brain::tests::source_hash_drift_excludes_page_from_authoritative_queries ... ok
+test metrics::tests::cache_report_splits_buckets_and_folds_index_files_into_their_checkpoint ... ok
+test metrics::tests::decode_timer_excludes_later_work_and_resets_between_requests ... ok
+test metrics::tests::prompt_timing_keeps_unmeasured_components_distinct_from_zero ... ok
+test instructions::tests::discovery_is_global_to_local_deduplicated_and_confined ... ok
+test metrics::tests::removed_preview_metric_code_stays_decodable ... ok
+test metrics::tests::snapshots_track_runtime_and_server_counters ... ok
+test model::qwen::tests::embedded_manifest_is_the_frozen_complete_set ... ok
+test model::qwen::tests::verification_rejects_hash_mismatch_and_honors_cancellation ... ok
+test instructions::tests::newly_crossed_scope_is_added_once_and_unavailable_content_is_retained ... ok
+test model::transfer::tests::cancellation_keeps_the_partial_file_for_the_next_run ... ok
+test model::transfer::tests::catalog_and_checksum_verification_are_explicit ... ok
+test model::transfer::tests::qwen_set_verifies_reloads_and_removes_as_one_managed_artifact ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test model::transfer::tests::managed_artifact_inventory_and_delete_include_partial_files ... ok
+test model::qwen::qwen_vision_is_an_independent_managed_artifact_set ... ok
+test model::transfer::tests::verification_reports_bytes_read ... ok
+test model_eval::tests::acceleration_receipt_reports_effective_sampling_contract ... ok
+test model_eval::tests::acceleration_switch_uses_each_models_native_path ... ok
+test model_eval::tests::canary_cross_phase_samples_keep_their_start_phase ... ok
+test model::transfer::tests::restart_resumes_at_the_existing_partial_byte ... ok
+test model_eval::tests::evaluations_exclude_overlapping_workers_and_release_on_exit ... ok
+test model_eval::tests::external_canary_observes_phases_and_finishes_on_eof ... ignored, Metal probe only; DS4SERVER_CANARY_BINARY must point to the current release binary
+test model_eval::tests::external_canary_protocol_accepts_only_known_phases ... ok
+test model_eval::tests::parses_safe_defaults_and_overrides ... ok
+test model_eval::tests::rejects_unbounded_or_invalid_limits ... ok
+test instructions::tests::frames_escape_control_tags_and_enforce_utf8_byte_limits ... ok
+test model_eval::tests::warmup_timing_separates_loading_without_inventing_missing_durations ... ok
+test model_eval::tests::worker_progress_events_advance_the_watchdog ... ok
+test native_edit::tests::edit_history_supports_undo_and_redo ... ok
+test native_edit::tests::paste_replays_a_complete_command_shortcut ... ok
+test model_eval::tests::repeated_prompts_form_one_ordered_conversation ... ok
+test native_menu::tests::edit_items_are_custom_menu_events ... ok
+test native_media::tests::media_player_only_accepts_remote_http_urls ... ok
+test native_menu::tests::user_guide_uses_unmodified_f1 ... ok
+test native_menu::tests::view_items_keep_the_selected_pane_checked ... ok
+test runtime::tests::dropping_active_generation_cancels_it ... ok
+test runtime::tests::operation_failures_use_the_matching_event ... ok
+test runtime::tests::operation_tracking_is_consistent ... ok
+test runtime::tests::runtime_panic_recovery_accepts_the_next_operation ... ok
+test server::request::tests::reasoning_requests_preserve_each_models_exact_modes ... ok
+test server::tests::canonical_tool_replay_preserves_argument_types ... ok
+test server::tests::connection_slots_enforce_the_limit_and_release_on_drop ... ok
+test server::tests::content_arrays_match_ds4_text_projection ... ok
+test server::tests::cors_headers_are_opt_in_and_cover_preflight ... ok
+test server::tests::context_limit_errors_use_protocol_standard_fields ... ok
+test server::tests::dsml_tool_stream_projects_incremental_json_arguments ... ok
+test server::tests::generated_dsml_becomes_openai_tool_calls ... ok
+test server::tests::generated_tool_parser_accepts_ds4_recovery_syntaxes ... ok
+test server::tests::model_metadata_uses_the_requested_alias_and_default_token_limit ... ok
+test server::tests::protocol_tool_results_require_live_or_replayed_call_ids ... ok
+test server::tests::responses_parser_rejects_dropped_state_and_preserves_hosted_calls ... ok
+test server::tests::sampled_tool_replay_survives_server_restart ... ok
+test server::tests::streaming_errors_match_ds4_server_framing ... ok
+test dev_brain::tests::validation_keeps_broken_links_editable_and_reports_provenance_errors ... ok
+test server::tests::tool_prompt_matches_ds4_server_text ... ok
+test server::tests::tool_stream_projects_ds4_recovery_syntaxes ... ok
+test settings::tests::captured_ds4_cli_options_are_uniquely_classified ... ok
+test settings::tests::effective_settings_build_one_engine_and_turn_configuration ... ok
+test settings::tests::effective_settings_preserve_unset_model_defaults ... ok
+test settings::tests::execution_settings_keep_ds4_automatic_values_and_glm_rules ... ok
+test settings::tests::kv_cache_gates_follow_the_ds4_store_rules ... ok
+test settings::tests::reasoning_selection_is_exact_and_model_specific ... ok
+test settings::tests::runtime_settings_preserve_automatic_values_and_cross_field_rules ... ok
+test settings::tests::speculative_settings_match_acceleration_defaults_and_dependencies ... ok
+test server::tests::streaming_prefill_keeps_the_client_connection_alive ... ok
+test dev_brain::tests::verified_skills_are_indexed_and_injected_on_demand ... ok
+test model_eval::tests::cpu_accounting_matches_getrusage_units ... ok
+test dev_brain::tests::source_versions_follow_that_file_instead_of_the_whole_worktree ... ok
+test engine::validation::tests::installed_ds4_fixture_opens_and_renders_a_prompt ... ok
+test extensions::tests::direct_hook_runner_isolated_environment_and_handles_failures ... ok
+test model::transfer::tests::stalled_download_times_out_and_keeps_the_partial_file ... ok
+test server::tests::request_deadline_stops_slow_clients ... ok
+test engine::sampling_tests::ds4_sampler_matches_original_reference_tokens_and_rng ... ok
+test agent::tests::execution_failures_and_timeouts_have_distinct_codes ... ok
+kill: 82281: No such process
+test agent::tests::dropping_session_tools_stops_live_jobs_and_removes_output ... ok
+test agent::tests::local_file_and_bash_tools_execute_in_the_project ... ok
+test extensions::tests::lifecycle_matchers_preserve_declared_event_order ... ok
+test engine::qwen_vision::qwen_image_preprocessing_matches_pillow_mtplx ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_shaders_are_pinned ... ok
+test extensions::tests::hook_failures_are_bounded_actionable_and_non_shell ... ok
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_mixed_distributions_match_reference ... ok
+
+test result: ok. 310 passed; 0 failed; 228 ignored; 0 measured; 0 filtered out; finished in 6.63s
+
+     Running unittests tools/qwen38-artifacts.rs (target/debug/deps/qwen38_artifacts-1288d410762a1a77)
+
+running 4 tests
+test tests::artifact_names_cannot_escape_the_output_directory ... ok
+test tests::affine_sample_digest_uses_packed_values_scales_and_biases ... ok
+test tests::safetensors_validation_rejects_duplicate_names_and_invalid_ranges ... ok
+test tests::artifact_download_resumes_an_authenticated_partial ... ok
+
+test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
+
+     Running unittests tools/test-supervisor.rs (target/debug/deps/test_supervisor-df80f419c8443a12)
+
+running 4 tests
+test supervisor::tests::direct_worker_arguments_are_not_rust_test_arguments ... ok
+test supervisor::tests::split_records_cannot_be_interleaved_with_watchdog_samples ... ok
+test supervisor::tests::split_probe_records_do_not_reset_progress ... ok
+ready{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+ready
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+progressprogressprogressprogressprogressprogressprogressprogresstest supervisor::tests::watchdog_stops_memory_and_silence_but_not_total_runtime ... ok
+
+test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.38s
+

docs/issue111-tokenizer/upstream.log

@@ -0,0 +1 @@
+.....s                                                                   [100%]

src/engine/metal/qwen_mtplx/issue103_analysis.rs

@@ -332,7 +332,10 @@ fn issue103_tokenizer_receipts() {
         "{}",
         serde_json::json!({"case":"tokenizer_summary","mismatches":mismatches})
     );
-    assert_eq!(mismatches, 5, "diagnosis changed; inspect exact token IDs");
+    assert_eq!(
+        mismatches, 0,
+        "Qwen IDs must match the trained pretokenizer"
+    );
 }
 
 #[test]

src/engine/metal/qwen_mtplx/issue111_product.rs

@@ -0,0 +1,126 @@
+//! Token-exact cache compatibility across the Qwen split correction.
+use super::*;
+use crate::settings::{
+    GenerationPreferences, ReasoningMode, RuntimePreferences, SpeculativePreferences,
+    effective_settings,
+};
+use std::path::PathBuf;
+
+#[test]
+#[ignore = "installed model; serial external memory supervisor required"]
+fn issue111_token_prefix_cache() {
+    let root = PathBuf::from(std::env::var_os("DS4_QWEN38_ARTIFACTS").unwrap());
+    let oracle = PathBuf::from(std::env::var_os("DS4_ISSUE111_ORACLE").unwrap());
+    let directory = PathBuf::from(std::env::var_os("DS4_ISSUE111_DIR").unwrap());
+    std::fs::create_dir_all(&directory).unwrap();
+    let requests: serde_json::Value =
+        serde_json::from_slice(&std::fs::read(oracle.join("requests.json")).unwrap()).unwrap();
+    let legacy: Vec<Vec<i32>> =
+        serde_json::from_slice(&std::fs::read(oracle.join("legacy-requests.json")).unwrap())
+            .unwrap();
+    let case = &requests[1];
+    let ids: Vec<i32> = serde_json::from_value(case["ids"].clone()).unwrap();
+    let settings = effective_settings(
+        crate::model::ModelChoice::Qwen38FlashNext,
+        &GenerationPreferences {
+            context_tokens: 8192,
+            max_generated_tokens: 2,
+            system_prompt: String::new(),
+            temperature: Some(0.),
+            top_p: Some(0.95),
+            min_p: Some(0.),
+            seed: Some(1),
+            reasoning_mode: ReasoningMode::Low,
+        },
+        &RuntimePreferences {
+            speculative: SpeculativePreferences {
+                glm_mtp: true,
+                ..Default::default()
+            },
+            ..Default::default()
+        },
+        root.parent().unwrap(),
+    )
+    .unwrap();
+    let messages = case["messages"]
+        .as_array()
+        .unwrap()
+        .iter()
+        .map(|m| ChatTurn {
+            user: m["role"] == "user",
+            tool: false,
+            system: false,
+            skip_previous_eos: false,
+            reasoning: m["reasoning_content"].as_str().map(str::to_owned),
+            reasoning_complete: true,
+            content: m["content"].as_str().unwrap().into(),
+        })
+        .collect::<Vec<_>>();
+    let mut owner = QwenExecutor::open(&settings.engine).unwrap();
+    owner.policy.store_min_suffix = Some(0);
+    assert_eq!(
+        owner
+            .metadata
+            .tokenizer
+            .encode_conversation("", &messages, ReasoningMode::Low),
+        ids
+    );
+    assert_ne!(legacy[1], ids);
+    owner.prefill(&legacy[1], |_| true).unwrap();
+    owner
+        .save_checkpoint(&directory.join("old-prefix.bin"), [111; 32], &mut |_| {})
+        .unwrap();
+    owner.reset().unwrap();
+    assert!(
+        owner
+            .load_checkpoint(&directory.join("old-prefix.bin"), &mut |_| {})
+            .unwrap()
+    );
+    assert_eq!(owner.tokens(), legacy[1]);
+    assert_eq!(owner.align_prompt(&ids).unwrap(), 0);
+    assert!(owner.live.current.is_none());
+    let metrics = Metrics::new(&directory.join("metrics"));
+    let run = |owner: &mut QwenExecutor, messages: &[ChatTurn]| {
+        let (out, complete) = owner
+            .generate(
+                messages,
+                &settings.turn,
+                &AtomicBool::new(false),
+                &metrics,
+                |_, _, _| {},
+                |_, _| {},
+            )
+            .unwrap();
+        assert!(complete);
+        out
+    };
+    let cold = run(&mut owner, &messages);
+    assert_eq!(cold.cached_tokens, 0);
+    assert_eq!(cold.prompt_tokens as usize, ids.len());
+    assert!(owner.tokens().starts_with(&ids));
+    owner
+        .save_checkpoint(&directory.join("new-prefix.bin"), [111; 32], &mut |_| {})
+        .unwrap();
+    let mut next = messages.clone();
+    next.push(cold.message.clone());
+    next.push(ChatTurn {
+        content: "Continue with Ⅷ and ².".into(),
+        ..messages[0].clone()
+    });
+    let warm = run(&mut owner, &next);
+    assert!(warm.cached_tokens >= cold.prompt_tokens);
+    let expected = super::issue107_product::state(&owner);
+    owner.reset().unwrap();
+    assert!(
+        owner
+            .load_checkpoint(&directory.join("new-prefix.bin"), &mut |_| {})
+            .unwrap()
+    );
+    let reopened = run(&mut owner, &next);
+    assert_eq!(warm.cached_tokens, reopened.cached_tokens);
+    assert_eq!(expected, super::issue107_product::state(&owner));
+    println!(
+        "{}",
+        serde_json::json!({"case":"token_cache", "old_tokens":legacy[1].len(),"new_tokens":ids.len(),"old_prefix_miss":true,"warm_cached":warm.cached_tokens,"disk_cached":reopened.cached_tokens,"warm_disk_state_exact":true})
+    );
+}

src/engine/metal/qwen_mtplx/owner.rs

@@ -1076,3 +1076,7 @@ mod issue107_product;
 #[cfg(test)]
 #[path = "issue109_product.rs"]
 mod issue109_product;
+
+#[cfg(test)]
+#[path = "issue111_product.rs"]
+mod issue111_product;

src/engine/tokenizer.rs

@@ -326,7 +326,7 @@ impl Tokenizer {
         match self.family {
             ModelFamily::DeepSeek => self.tokenize_joyai(text, &mut output),
             ModelFamily::Glm => self.tokenize_gpt4(text, &mut output, 3),
-            ModelFamily::Qwen => self.tokenize_gpt4(text, &mut output, 1),
+            ModelFamily::Qwen => self.tokenize_qwen(text, &mut output),
         }
         output
     }
@@ -374,7 +374,7 @@ impl Tokenizer {
         match self.family {
             ModelFamily::DeepSeek => self.tokenize_joyai(text, output),
             ModelFamily::Glm => self.tokenize_gpt4(text, output, 3),
-            ModelFamily::Qwen => self.tokenize_gpt4(text, output, 1),
+            ModelFamily::Qwen => self.tokenize_qwen(text, output),
         }
     }
 
@@ -771,6 +771,12 @@ impl Tokenizer {
         }
     }
 
+    fn tokenize_qwen(&self, text: &str, output: &mut Vec<i32>) {
+        for piece in qwen_pieces(text) {
+            self.emit_piece(piece.as_bytes(), output);
+        }
+    }
+
     fn tokenize_gpt4(&self, text: &str, output: &mut Vec<i32>, max_digits: usize) {
         let mut position = 0;
         while position < text.len() {
@@ -861,6 +867,39 @@ impl Tokenizer {
     }
 }
 
+/// The trained Qwen3 Split expression. Rust regex supports its Unicode
+/// categories but not the penultimate `\s+(?!\S)` alternative: before a
+/// non-whitespace character that branch backtracks by one whitespace scalar,
+/// leaving it for the next word/punctuation match. At EOF it keeps the run.
+fn qwen_pieces(text: &str) -> impl Iterator<Item = &str> {
+    static SPLIT: std::sync::LazyLock<regex::Regex> = std::sync::LazyLock::new(|| {
+        regex::Regex::new(
+            r"(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?[\p{L}\p{M}]+|\p{N}| ?[^\s\p{L}\p{M}\p{N}]+[\r\n]*|\s*[\r\n]+|(?P<whitespace>\s+)",
+        ).expect("valid Qwen3 pretokenizer expression")
+    });
+    let mut position = 0;
+    std::iter::from_fn(move || {
+        if position == text.len() {
+            return None;
+        }
+        let captures = SPLIT
+            .captures_at(text, position)
+            .expect("Qwen3 covers every Unicode scalar");
+        let matched = captures.get(0).unwrap();
+        debug_assert_eq!(matched.start(), position);
+        let mut end = matched.end();
+        if captures.name("whitespace").is_some() && end < text.len() {
+            let last = matched.as_str().char_indices().next_back().unwrap().0;
+            if last != 0 {
+                end = position + last;
+            }
+        }
+        let piece = &text[position..end];
+        position = end;
+        Some(piece)
+    })
+}
+
 fn byte_encode(bytes: &[u8]) -> String {
     bytes
         .iter()
@@ -1343,3 +1382,7 @@ mod tests {
         );
     }
 }
+
+#[cfg(test)]
+#[path = "tokenizer_issue111.rs"]
+mod issue111;

src/engine/tokenizer_issue111.rs

@@ -0,0 +1,154 @@
+use super::*;
+use sha2::{Digest, Sha256};
+
+fn oracle() -> std::path::PathBuf {
+    std::path::PathBuf::from(std::env::var_os("DS4_ISSUE111_ORACLE").unwrap())
+}
+
+#[test]
+#[ignore = "CPU-only; installed DS4 GGUFs and exact oracle IDs"]
+fn issue111_ds4_families() {
+    let expected: serde_json::Value =
+        serde_json::from_slice(&fs::read(oracle().join("ds4.json")).unwrap()).unwrap();
+    for family in expected["families"].as_array().unwrap() {
+        let choice = if family["family"] == "glm" {
+            crate::model::ModelChoice::Glm53Flash
+        } else {
+            crate::model::ModelChoice::DeepSeekV4Flash0731
+        };
+        let model =
+            super::super::Model::open_main(Path::new(family["model"].as_str().unwrap()), choice)
+                .unwrap();
+        for case in family["cases"].as_array().unwrap() {
+            let text = case["text"].as_str().unwrap();
+            for ids in [
+                model.tokenizer.tokenize(text),
+                model.tokenizer.tokenize_rendered(text),
+            ] {
+                assert_eq!(
+                    serde_json::json!(ids),
+                    case["ids"],
+                    "{} {text:?}",
+                    family["family"]
+                );
+            }
+        }
+        println!(
+            "{}",
+            serde_json::json!({"case":"ds4_control", "family":family["family"], "cases":family["cases"].as_array().unwrap().len(), "plain_rendered_exact":true})
+        );
+    }
+}
+
+#[test]
+fn qwen_split_boundaries() {
+    for (text, expected) in [
+        (" ① Ⅷ ² 𝟘", vec![" ", "①", " ", "Ⅷ", " ", "²", " ", "𝟘"]),
+        ("a\u{301}  \t b   ", vec!["a\u{301}", "  \t", " b", "   "]),
+        (
+            "I'm WE'RE\r\n  x",
+            vec!["I", "'m", " WE", "'RE", "\r\n", " ", " x"],
+        ),
+        ("́ १́b", vec!["́", " ", "१", "́b"]),
+    ] {
+        assert_eq!(qwen_pieces(text).collect::<Vec<_>>(), expected, "{text:?}");
+    }
+}
+
+#[test]
+#[ignore = "CPU-only; exhaustive pinned Qwen split receipt"]
+fn issue111_every_unicode_scalar_split() {
+    let expected: serde_json::Value =
+        serde_json::from_slice(&fs::read(oracle().join("scalar-split.json")).unwrap()).unwrap();
+    let mut text = String::new();
+    for c in (0..=0x10ffff).filter_map(char::from_u32) {
+        text.push('A');
+        text.push(c);
+        text.push_str("A ");
+    }
+    let mut hash = Sha256::new();
+    let mut count = 0;
+    for piece in qwen_pieces(&text) {
+        hash.update((piece.len() as u32).to_le_bytes());
+        hash.update(piece.as_bytes());
+        count += 1;
+    }
+    let hash = hash
+        .finalize()
+        .iter()
+        .map(|b| format!("{b:02x}"))
+        .collect::<String>();
+    assert_eq!(count, expected["pieces"].as_u64().unwrap());
+    assert_eq!(hash, expected["sha256"].as_str().unwrap());
+    println!(
+        "{}",
+        serde_json::json!({"case":"unicode_split","scalars":expected["scalars"],"pieces":count,"sha256":hash,"exact":true})
+    );
+}
+
+#[test]
+#[ignore = "CPU-only; installed Qwen tokenizer and pinned exact IDs"]
+fn issue111_qwen_tokens_and_requests() {
+    let root = std::path::PathBuf::from(std::env::var_os("DS4_QWEN38_ARTIFACTS").unwrap());
+    let tokenizer = Tokenizer::load_qwen(&root.join("tokenizer.json")).unwrap();
+    let expected: serde_json::Value =
+        serde_json::from_slice(&fs::read(oracle().join("qwen.json")).unwrap()).unwrap();
+    for case in expected["cases"].as_array().unwrap() {
+        let text = case["text"].as_str().unwrap();
+        assert_eq!(
+            serde_json::json!(qwen_pieces(text).collect::<Vec<_>>()),
+            case["pieces"],
+            "split {text:?}"
+        );
+        for ids in [tokenizer.tokenize(text), tokenizer.tokenize_rendered(text)] {
+            assert_eq!(serde_json::json!(ids), case["ids"], "{text:?}");
+            let decoded = ids
+                .iter()
+                .flat_map(|&id| tokenizer.token_bytes(id).unwrap())
+                .collect::<Vec<_>>();
+            assert_eq!(decoded, text.as_bytes());
+        }
+    }
+    let requests: serde_json::Value =
+        serde_json::from_slice(&fs::read(oracle().join("requests.json")).unwrap()).unwrap();
+    for case in requests.as_array().unwrap() {
+        let effort = match case["effort"].as_str().unwrap() {
+            "direct" => ReasoningMode::Direct,
+            "low" => ReasoningMode::Low,
+            "medium" => ReasoningMode::Medium,
+            "xhigh" => ReasoningMode::XHigh,
+            _ => unreachable!(),
+        };
+        let messages = case["messages"]
+            .as_array()
+            .unwrap()
+            .iter()
+            .map(|m| ChatTurn {
+                user: m["role"] == "user",
+                tool: false,
+                system: false,
+                skip_previous_eos: false,
+                reasoning: m["reasoning_content"].as_str().map(str::to_owned),
+                reasoning_complete: true,
+                content: m["content"].as_str().unwrap().to_owned(),
+            })
+            .collect::<Vec<_>>();
+        let ids = tokenizer.encode_conversation("", &messages, effort);
+        assert_eq!(serde_json::json!(ids), case["ids"], "{}", case["effort"]);
+        assert_eq!(
+            serde_json::json!(tokenizer.tokenize_rendered(case["rendered"].as_str().unwrap())),
+            case["ids"]
+        );
+        let mut resumed = tokenizer.encode_history("", &messages[..2], effort);
+        resumed.extend(tokenizer.encode_continuation(&messages[2].content, effort, false));
+        assert_eq!(resumed, ids);
+        println!(
+            "{}",
+            serde_json::json!({"case":"request","effort":case["effort"],"context_tokens":ids.len(),"exact":true})
+        );
+    }
+    println!(
+        "{}",
+        serde_json::json!({"case":"tokenizer","number_count":expected["number_count"],"cases":expected["cases"].as_array().unwrap().len(),"plain_rendered_roundtrip_exact":true})
+    );
+}

tools/issue111-reference.py

@@ -0,0 +1,59 @@
+"""Pinned Qwen3 split/token/request IDs and local DS4 family controls; no inference."""
+import argparse,hashlib,json,os,struct,subprocess,sys
+from pathlib import Path
+parser=argparse.ArgumentParser(description=__doc__)
+parser.add_argument('checkout',type=Path)
+parser.add_argument('root',type=Path)
+parser.add_argument('out',type=Path)
+parser.add_argument('--ds4',type=Path)
+parser.add_argument('--glm',type=Path)
+parser.add_argument('--deepseek',type=Path)
+args=parser.parse_args()
+checkout,root,out=args.checkout,args.root,args.out
+assert subprocess.check_output(['git','-C',str(checkout),'rev-parse','HEAD'],text=True).strip()=='9882703f3105363ddc37eca9f97aa09a1d387112'
+sys.path.insert(0,str(checkout))
+os.environ.update(HF_HUB_OFFLINE='1',TRANSFORMERS_OFFLINE='1')
+from tokenizers import Regex,Tokenizer as BackendTokenizer,pre_tokenizers
+from mtplx.runtime import _QWEN3_PRETOKENIZER_SPLIT,_load_tokenizer_resilient
+from mtplx.server import openai as server
+out.mkdir(parents=True,exist_ok=True)
+tokenizer=_load_tokenizer_resilient(root,json.loads((root/'config.json').read_text()))
+component=BackendTokenizer.from_str(tokenizer.backend_tokenizer.to_str())
+component.normalizer=None
+split=pre_tokenizers.Split(Regex(_QWEN3_PRETOKENIZER_SPLIT),behavior='isolated')
+scalars=[chr(i) for i in range(0x110000) if not 0xd800<=i<=0xdfff]
+text=''.join('A'+c+'A ' for c in scalars)
+h=hashlib.sha256();count=0
+for piece,_ in split.pre_tokenize_str(text):
+ b=piece.encode(); h.update(struct.pack('<I',len(b)));h.update(b);count+=1
+(out/'scalar-split.json').write_text(json.dumps(dict(scalars=len(scalars),pieces=count,sha256=h.hexdigest(),pattern=_QWEN3_PRETOKENIZER_SPLIT)))
+number_split=pre_tokenizers.Split(Regex(r'\p{N}'),behavior='isolated')
+numbers=[p for p,_ in number_split.pre_tokenize_str('A'.join(scalars)) if len(p)==1 and p!='A']
+texts=[' ①',' Ⅷ',' ²',' 𝟘','Item ①, Ⅷ, ², 𝟘 and 123.','भारत एक विशाल देश है जिसकी संस्कृति बहुत पुरानी है।','ประเทศไทยมีวัฒนธรรมที่เก่าแก่และสวยงามมาก','الْعَرَبِيَّةُ','The quick brown fox jumps over the lazy dog.','中文，标点。','def f(x):\n    return x + 123\n',"I'm WE'RE it's he'd I'll you've can't 'ſx",'\u0301a a\u0301 \u0301 १\u0301b','a\r\n  b\t\t','a   ','a\t \u2003b','a\u0085\u00a0b','x \n \r\n   y','😀 ①🙂Ⅷ ²+𝟘']
+texts += [prefix+c+suffix for c in numbers for prefix,suffix in [(' ',''),('a','b'),(' \u0301','\u0301x')]]
+records=[]
+for text in texts:
+ full_ids=tokenizer.encode(text,add_special_tokens=False)
+ ids=component.encode(text,add_special_tokens=False).ids
+ assert component.decode(ids,skip_special_tokens=False)==text,(text,ids)
+ records.append(dict(text=text,ids=ids,full_ids=full_ids,pieces=[p for p,_ in split.pre_tokenize_str(text)]))
+(out/'qwen.json').write_text(json.dumps(dict(number_count=len(numbers),cases=records),ensure_ascii=False))
+requests=[]
+for effort in ['direct','low','medium','xhigh']:
+ messages=[dict(role='user',content=' ① Ⅷ ² 𝟘 भारत \u0301a\n '),dict(role='assistant',content='One ①.',reasoning_content='' if effort=='direct' else 'Brief ² thought.'),dict(role='user',content='Continue Ⅷ and 𝟘.\n   x')]
+ ids=server._encode_messages_uncached(tokenizer,[server.ChatMessage(**m) for m in messages],enable_thinking=effort!='direct',reasoning_effort=None if effort=='direct' else effort,preserve_reasoning_history=True,tools=None)
+ requests.append(dict(effort=effort,messages=messages,ids=ids,rendered=tokenizer.decode(ids,skip_special_tokens=False,clean_up_tokenization_spaces=False)))
+(out/'requests.json').write_text(json.dumps(requests,ensure_ascii=False))
+print(json.dumps(dict(case='qwen_oracle',number_count=len(numbers),cases=len(records),requests=len(requests),scalar_count=len(scalars),split_count=count,split_sha256=h.hexdigest())),flush=True)
+
+# CPU vocabulary loading only; DS4 remains an external behavioral oracle.
+if args.ds4:
+ assert args.glm and args.deepseek
+ families=[]
+ for family,path in [('glm',args.glm),('deepseek',args.deepseek)]:
+  cases=[]
+  for text in texts[:19]:
+   raw=subprocess.check_output([str(args.ds4),'-m',str(path),'--dump-tokens','--raw','-p',text],text=True)
+   cases.append(dict(text=text,ids=json.loads(raw.splitlines()[0])))
+  families.append(dict(family=family,model=str(path),cases=cases))
+ (out/'ds4.json').write_text(json.dumps(dict(revision=subprocess.check_output(['git','-C',str(args.ds4.parent),'rev-parse','HEAD'],text=True).strip(),binary_sha256=hashlib.sha256(args.ds4.read_bytes()).hexdigest(),families=families),ensure_ascii=False))