Compare Qwen cache snapshots at matching stages

docs/issue109-verify-gate/README.md

@@ -84,3 +84,13 @@ MTPLX_QWEN4_VERIFY_SDPA_HEAD_CHUNK=0 \
   tools/issue109-model-reference.py /path/to/pinned/mtplx \
   "$DS4_QWEN38_ARTIFACTS" "$DS4_ISSUE109_DIR" /tmp/issue109-model-oracle
 ```
+
+## Snapshot comparison correction (#114)
+
+The historical zero-sign receipt above measured an extra deep oracle snapshot
+against a lazy native bank entry. The deep clone adds zero; this alone creates
+the reported bit change. The raw/live and matching lazy snapshots agree exactly.
+The corrected shared oracle now compares the server's actual snapshot stages,
+requires every raw cache bit to match, and has no signed-zero exception. See
+[the isolated trace and regression](../issue114-snapshot-zero/README.md). The
+original receipts remain unchanged as evidence of the measurement error.

docs/issue114-snapshot-zero/README.md

@@ -0,0 +1,94 @@
+# Matching cache snapshot stages (#114)
+
+The reported layer-35 zero-sign difference came from the oracle measurement.
+`tools/issue109-model-reference.py` called `snapshot_cache` on the reference
+trunk but compared it with the native server's lazy bank entry. Pinned
+`cache_state.py::_clone_tree` implements a deep clone as `value + zeros`, which
+turns negative zero into positive zero. `snapshot_cache_lazy_hybrid`, selected
+by the server's default `MTPLX_SESSION_LAZY_SNAPSHOT` policy, retains the QSA key
+bits. Native production already implements these two contracts correctly.
+
+The oracle tool now compares the actual bank stages: policy-selected lazy/deep
+trunk, always-deep committed MTP head, and the suffix owner's final hidden row.
+Every cache leaf must match its raw-bit hash. The old zero-sign exception and
+numerical/canonical-zero comparison are removed. The #113 entry point invokes
+this same strict implementation so the two tools cannot drift. Historical #109
+receipts are preserved and annotated with this correction.
+
+This resolves a verification defect. There is no production arithmetic, RoPE,
+cache-format, model or DS4/GLM change, and no platform mismatch to approve.
+Changing the native zero sign would violate the actual lazy-snapshot oracle.
+This is a correction to the Qwen MTPLX comparison, not a DS4Server-only feature.
+
+## First changed bit
+
+Pinned MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2,
+Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7` on M5 Max.
+Native starting commit: `d9026aa7bfbc8f74ec82c120178c34c5f08510d3`.
+
+The independent snapshot probe reads the **original** #109 native and oracle
+raw key files. Reconstructing the BF16 key from its F32 export is required to
+round-trip every bit before the probe proceeds. Installing it in a real pinned
+QSACache and reading raw state / lazy snapshot preserves the native hash.
+Taking the additional deep snapshot reproduces the original oracle file
+byte-for-byte. In all three states, the only changed coordinate is
+`[0, 1, 0, 2]`: F32 export `0x80000000` → `0x00000000`.
+
+| Original state | Key shape | Raw/lazy hash prefix | Extra deep snapshot / old oracle hash prefix |
+|---|---|---|---|
+| Cold prefill | 1×2×46×256 | d81b466dea1ff5b7 | 520dd44032eda478 |
+| Cold final | 1×2×58×256 | 6a1c56e7333fc86e | 44b87b49e9f9853e |
+| Warm final | 1×2×88×256 | 061f8fe052fe21c8 | 98186c7fec249807 |
+
+Full hashes, coordinates and independently generated BF16/F32 fixtures are in
+`tests/fixtures/mtplx-snapshot-zero.json`. This locates the first changed bit at
+the measurement clone, without attributing it to an earlier model operation.
+
+## Regression and reproduction
+
+The native regression exercises actual `snapshot` and `restore` paths for
+BF16 and F32, lazy/deep snapshots and both restore-cloning modes. It compares
+all bits to the independent fixture and verifies the source and banked arrays
+remain unchanged, together with the restored cache offset.
+
+Generate the fixture under isolated MLX 0.32.2 Python using:
+`tools/issue114-snapshot-reference.py PINNED_CHECKOUT ORIGINAL_NATIVE_DIR
+ORIGINAL_ORACLE_DIR tests/fixtures/mtplx-snapshot-zero.json`.
+The original directories contain `cold-prefill-key35.f32`, `cold-key35.f32`
+and `warm-key35.f32` from the #109 run, before the harness correction.
+
+Build the release test executable (`cargo test --release --all-features --bin
+ds4-server issue114 --no-run`), copy it outside `target`, and run
+`issue114_snapshot_stage_preserves_zero_bits --ignored --nocapture
+--test-threads=1` under `target/debug/test-supervisor 4096 60 120 --command`.
+
+For complete model verification, run the shared native
+`issue113_default_verify_and_checkpoint` test in separate default/off processes
+under the 114688 MiB supervisor, using the #113 documented environment and new
+output directories. Run `tools/issue109-model-reference.py` against the original
+explicit-off reproduction and `tools/issue113-model-reference.py` against the
+default run. Both now use identical strict raw-bit comparison. The native
+request is the original 46-token low-reasoning prompt, greedy seed 1, MTP depth
+3 and max 12, with `MTPLX_CONTEXT_COPY=0`.
+
+## Completion review
+
+Fresh serial default and explicit-off native/oracle runs pass: all eight verify
+forwards per configuration, every prefill/final logit and hidden byte, all
+trunk/MTP cache bits, emitted tokens, warm reuse and disk reopen. There are no
+remaining key coordinates with different bits. The original prefill key is
+negative zero in both implementations at the reported coordinate. All eight
+BF16/F32 snapshot/restore combinations match the independent reference.
+Installed DeepSeek session switching and GLM verifier/recurrent-state controls
+also pass; production code is unchanged.
+
+Formatting, Clippy with `-D warnings`, signed app bundle and the full suite
+pass (313 application + 8 tool tests; 237 ignored hardware/reference tests).
+The first full-suite attempt reproduced existing #115: an unquoted Git SHA
+parsed as a YAML float (`5568000000.0`). Its failure evidence is retained; the
+unchanged retry passed. #115 remains a separate next-priority commit.
+
+Review against the issue and pinned source is complete. The diagnostic cause
+is isolated, the incorrect oracle comparison is fixed, the raw-bit exception
+is removed, and focused plus full-model regressions verify the actual contract.
+No runtime mismatch is waived and no implementation is deferred.

docs/issue114-snapshot-zero/bundle.log

@@ -0,0 +1,8 @@
+cargo build --release --all-targets --all-features
+   Compiling ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
+    Finished `release` profile [optimized] target(s) in 20.17s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
+cargo packager --release --formats app
+codesign --force --deep --sign - --timestamp=none target/release/DS4Server.app
+target/release/DS4Server.app: replacing existing signature

docs/issue114-snapshot-zero/clippy.log

@@ -0,0 +1,4 @@
+    Checking ds4-server v0.1.0 (/Users/gb/Projects/DS4Server)
+    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.03s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`

docs/issue114-snapshot-zero/deepseek.log

@@ -0,0 +1,10 @@
+
+running 1 test
+test engine::metal::tests::resident_multi_session_switching_preserves_each_kv_frontier ... ds4: Metal device Apple M5 Max, 128.00 GiB RAM
+ds4: Metal 4 tensor API enabled for Tensor kernels
+ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off rope_exp2_log2=off math_safe=off tensor_matmul=on
+ds4: Metal model views created in 2.029 ms, residency requested in 9018.687 ms, warmup 3.972 ms (mapped 82697.67 MiB from offset 5.09 MiB)
+ds4: Metal mapped mmaped model as 2 overlapping shared buffers
+ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 549 filtered out; finished in 9.59s

docs/issue114-snapshot-zero/default-cold-arrays.json

@@ -0,0 +1 @@
+{"trunk": [[{"shape": [1, 3, 10240], "sha256": "1dadce7f74c7b48ad423170b8884bf5024c3d6dd91b2f9d800afca7d32679d7b"}, {"shape": [1, 48, 128, 128], "sha256": "d7436faf9d50e278f7fa095e9c217467020a6f534a48195775c212caa0ece52d"}], [{"shape": [1, 3, 10240], "sha256": "ca1b3cf93598b62575f01aea05ff0d0f087c00fdf1b63654626ef5ec3487715a"}, {"shape": [1, 48, 128, 128], "sha256": "f1364bb8165b9189279b7769e4081f213c562f4e0541acc5962cf2ae9f42a831"}, {"shape": [1, 9, 10240], "sha256": "c9110c820fc39f8c19c770594c15e0997693015090a3c7af4b6c8ec9a31ebc4e"}, {"shape": [1, 2], "sha256": "9febaa06152dac026dba5f2c1151c11ff6a1b2f00e92278cdfd2905c87cf2c2f"}], [{"shape": [1, 3, 10240], "sha256": "320fdcf9c7ab899133952e093d1f3de613ba5581b3aaad45c4ad7b1b4e43f1f3"}, {"shape": [1, 48, 128, 128], "sha256": "8cae4ba20fd0e54f8da130e5f8681ece7c54672b7a6404800cded8e64a977e08"}], [{"shape": [1, 2, 58, 256], "sha256": "404510256f371f0bc2e7b8af2d1f7eb2ae0322d42eb9a0ee5827a9a5c8d489c0"}, {"shape": [1, 2, 58, 256], "sha256": "f14afc7fe1c5e4c62c84121e804e14aa484db378861904d4e234ffeef10252ef"}, {"shape": [1, 58, 128], "sha256": "ca2d0b2f09e87fab8d0a6aede7073ec53bc37b584851f0342c76244f05ca3e12"}, {"shape": [1, 14, 128], "sha256": "a609a4f52c9b3dc0fb5d6554019786bebecdad220bf5ae6b40d4939821f12c8b"}], [{"shape": [1, 3, 10240], "sha256": "c14f586416a8e772fa53002514ff980ab579060981040071c3e4636fc9762070"}, {"shape": [1, 48, 128, 128], "sha256": "3dd403eccb148601e1ab505dee16956ad59128e2d51f1c0ad5fbe20509b6d4fc"}], [{"shape": [1, 3, 10240], "sha256": "d3506442712f1e9433327907d1390a1166912fac6ce36d7c339a50f9adb570fb"}, {"shape": [1, 48, 128, 128], "sha256": "1d5a82226d350ca721e1b3a12e5c28ae175d9116e36fccda4218efca9fe699f1"}], [{"shape": [1, 3, 10240], "sha256": "e1f5a0ab427dca8d091996febccbeebd5cbc987d7189ed1badbfe57dc14f8747"}, {"shape": [1, 48, 128, 128], "sha256": "17feff27d8744d9a3b9b115b3c723b650a581c22c32cef569d827bbc24de7206"}], [{"shape": [1, 2, 58, 256], "sha256": "364736a6d66b0aa29df064c406d3f7190f05fa385ffc10b031760541dc3cc8b5"}, {"shape": [1, 2, 58, 256], "sha256": "22369eed488919ec8c7aea527654f8c8e335ef0a995620b06e0a1ec391f91b30"}, {"shape": [1, 58, 128], "sha256": "20dd11e3fd1a655cf5d52fe260518fd0bc896ba7abf9709d590ab891d8e75120"}, {"shape": [1, 14, 128], "sha256": "378afa222e09564605d3ccd930cc86a2c4456ca99e122c8075def851bb453568"}], [{"shape": [1, 3, 10240], "sha256": "3ae083450733c47e30d8564cecca1056ab20d358acd48d3c81d63eae3577980d"}, {"shape": [1, 48, 128, 128], "sha256": "5044b96ba3ff509a76564d5a63a68de9744bdac90c2df5e191af4f12e5478f48"}], [{"shape": [1, 3, 10240], "sha256": "b827dfbf602825d83f60b7f079a9cd3a67123ae1ad08ab046bd3878b411fa6c3"}, {"shape": [1, 48, 128, 128], "sha256": "70af0a5e7ad8fa8b63c40b40cc09c225378faf36ab648ffffc5c3bd4746802c7"}], [{"shape": [1, 3, 10240], "sha256": "63f127bb20b6544b831d9cb3c476f4a093024fd425376a57538755be07e517df"}, {"shape": [1, 48, 128, 128], "sha256": "1d5d23064fbf2193615d4a1f83aa1897ed9c48e4c61a217abda7fd166302fea6"}], [{"shape": [1, 2, 58, 256], "sha256": "b13ce3a6861f643b03be4bb7ed27ed736001b445252123ba1b2c4b40b5dfe4a9"}, {"shape": [1, 2, 58, 256], "sha256": "114605ae9e08f2c065eee34f25b8f11a4805e14d8de8845ad905fa33453bc2e2"}, {"shape": [1, 58, 128], "sha256": "e94d105d822b6cd44c92e25fc08ef1e33af833f9175b714f3cc805bcdf25a6d2"}, {"shape": [1, 14, 128], "sha256": "3ebde0b0615cff228c577a6f5f886dcedcf4f60671c4a2cedcb1ee440f286824"}], [{"shape": [1, 3, 10240], "sha256": "14fe37048d44b26273932c62ffc6e992fe7fa1adb78be368fb5926c16759c51b"}, {"shape": [1, 48, 128, 128], "sha256": "ca98f06ce0c91c1c826aa7cd86491e46dabc370ac286a968f282b871357f81a1"}], [{"shape": [1, 3, 10240], "sha256": "57ce536e225a28e6386be44194455e9b225c5a5ac12762bb1c98026a3e556020"}, {"shape": [1, 48, 128, 128], "sha256": "870ae60d413be6a21d74ebea374d389fba081d608cf1a96b4b6d7f87f5a16f7a"}], [{"shape": [1, 3, 10240], "sha256": "e096b86584a0e8150c81139f61472e2b3df9cc968d1be3ac02efa94f45fd5836"}, {"shape": [1, 48, 128, 128], "sha256": "81b5d40eed643f0577e92d4638d1db83a5b9dd681bb4ff369881286d57b0f809"}], [{"shape": [1, 2, 58, 256], "sha256": "288f45328bac8901c5b91da6a791961eb681ab260bb2819b51316dd2348f6c4b"}, {"shape": [1, 2, 58, 256], "sha256": "9fe4421eb07afacd538e0bfc117656f6d2c2aeb9143a30b0f5caac9e15b5afb1"}, {"shape": [1, 58, 128], "sha256": "82b986471d2e3e61d4425a5459813c57e3c5d48228c584d88b18caef96620463"}, {"shape": [1, 14, 128], "sha256": "f679a8da4c496a4753ae1c55b99f4a33b4a81575cc24e5247275c36581002f1e"}], [{"shape": [1, 3, 10240], "sha256": "b64dc7d77ed93c7972187edf51a9d5098c86c0a1f79b7f40745efd0d062a8a50"}, {"shape": [1, 48, 128, 128], "sha256": "39659f369f3f579226f5209b37f7ae3034adb849b4369e17fae74fd9cdc540e5"}], [{"shape": [1, 3, 10240], "sha256": "33480e35ac640ecebfd73fa6f97843af70ea15d4a15db0a6502ab942a4691455"}, {"shape": [1, 48, 128, 128], "sha256": "3bffda871c549738052fbc2b8ecdd90b9ca5397aeac63054f2523c64a71899c7"}], [{"shape": [1, 3, 10240], "sha256": "7c0531d30334402ea9baa94147f3995a20463509b5baffa403c2d401d394af50"}, {"shapLine truncated

docs/issue114-snapshot-zero/default-warm-arrays.json

@@ -0,0 +1 @@
+{"trunk": [[{"shape": [1, 3, 10240], "sha256": "1df54fa91eb80b88e8b4c0d566828d36540f3508099fc1751dd700d9bd904a84"}, {"shape": [1, 48, 128, 128], "sha256": "1cc66986fdbef7d449d321ff6c558b4e862d2d9211d74038325c72c4f2f0cebf"}], [{"shape": [1, 3, 10240], "sha256": "1ba5aa461d83294641ab065813f2a27ce3a1aa280890046cdc6d839d49c86232"}, {"shape": [1, 48, 128, 128], "sha256": "b789c69d02395058a02c548725e7897e906176c9647e88b5c413a9c743a73aaf"}, {"shape": [1, 9, 10240], "sha256": "eaffcb3f7958776a888b3134ef5f9fa1285f21c242d76b0ebc19c64d75d26c08"}, {"shape": [1, 2], "sha256": "b5e055307f70d9b61bc8427b1da30e8f711b4da0c73d28a34879de8c3f253152"}], [{"shape": [1, 3, 10240], "sha256": "1ebb9fb3628cf830f4c16a815ecaa0c9ccc2049bff2b14d32cb4e3f08f1b91b2"}, {"shape": [1, 48, 128, 128], "sha256": "403226ed2eeae9becd4b498b0fa14de8152beac9138bff0c118698cd4c4fcc34"}], [{"shape": [1, 2, 88, 256], "sha256": "b4995a3d443eb57598940e4a80ece426f8e8442e07f30b8168b247baebfc422f"}, {"shape": [1, 2, 88, 256], "sha256": "eb25629ce30b96850b39bf034124fb8d627c82402baeb89d72e24869005ace24"}, {"shape": [1, 88, 128], "sha256": "1561cf66bb84117ff09036ee128d5c2e5aa21f69db459b82000dc467527df748"}, {"shape": [1, 22, 128], "sha256": "4366612cb46ecc942d06aabe6043fea79b494a34f08f797fec5e0a76ff067a1a"}], [{"shape": [1, 3, 10240], "sha256": "23af29f2397aace825e958a5faa77cb734a968ef2a7a7f058f5dc543b582c847"}, {"shape": [1, 48, 128, 128], "sha256": "d4c4e8b85d76cfb913593521150b144cb2bb6a6a3aed2bfe511fad2f716616ae"}], [{"shape": [1, 3, 10240], "sha256": "a342824984d9f603f852b6419dbd2be196dd3c2ad66b8d272ba1c723bfc58854"}, {"shape": [1, 48, 128, 128], "sha256": "a46384ccc56e527d39df7582277baeba2229d208cc0529ecaeafd3e53b8d6db2"}], [{"shape": [1, 3, 10240], "sha256": "21a78ff13a7b64234bd2db960ccfcae99cc575c3f42b185eb8831473666547e8"}, {"shape": [1, 48, 128, 128], "sha256": "69cea0bd54b8996f97bdc9f718cb4ddba75dd4c416a1f1590a1118a4503235ee"}], [{"shape": [1, 2, 88, 256], "sha256": "75ca9e3382094e7fb1d1fb79c1b492164178b9dc028a75a084d82cba8c719b37"}, {"shape": [1, 2, 88, 256], "sha256": "2ceb5beed581a848481bd02f8d09b07901c37fbda93fd92d94ba1344579c65e0"}, {"shape": [1, 88, 128], "sha256": "bdbd34eb86fe22271437831df743bf44efb01a7dce240a9e6b6321d0c6356cde"}, {"shape": [1, 22, 128], "sha256": "bd0a7f30f06069bd50a87a8981cfd1657a172fe5407b78f666899f49aa1a96fc"}], [{"shape": [1, 3, 10240], "sha256": "ef974274168dd5c569007e254179cf0ac745e7a7b28d2fbda9f06bc58c618158"}, {"shape": [1, 48, 128, 128], "sha256": "6706d6620ead67281abd40ad7239479a760992e42f0e2b8bfd2d3841c1bde52f"}], [{"shape": [1, 3, 10240], "sha256": "1c6b6c06774bcfb8190cd48a437e99c9796d88d9fa6619f461f3fc90155ac15f"}, {"shape": [1, 48, 128, 128], "sha256": "78d2709f54d5687657883f8bd9216eb981d83ea4554959108c29dc0494e85f9f"}], [{"shape": [1, 3, 10240], "sha256": "57977ed3fb0c1ed17d8b10bd2f525f28d63aad65eb434c513b490c9eb940352f"}, {"shape": [1, 48, 128, 128], "sha256": "c33c4474957879b91be8e0ca721ce66ba6a6eb533028aa3cf9509a23620203ff"}], [{"shape": [1, 2, 88, 256], "sha256": "e19a00773b642b8e4e9309d6cb70cf123211fcd86ba6999301a2809115edb78f"}, {"shape": [1, 2, 88, 256], "sha256": "aa68365ecfd52961b728f8192765e1aa834cbe56ecddff87cc007d3b98cc5dac"}, {"shape": [1, 88, 128], "sha256": "0ef422450f947767c661a851484b2c255722dee05525638e2faa342cc681e300"}, {"shape": [1, 22, 128], "sha256": "1ed1799c8e3d0f0656911bfdf233f591d3a2532856fa4701dc43032087ef1ab2"}], [{"shape": [1, 3, 10240], "sha256": "33a2b31ca65d7f70d132117054cd457e87466b4939920dd29263564ea20d5c0c"}, {"shape": [1, 48, 128, 128], "sha256": "8ba516367289ba29ea548682ca846fa0764bd3948ee856be8e5d19a06c86bea4"}], [{"shape": [1, 3, 10240], "sha256": "6e6b859184153d7b80f1e84a6e4224d521dae73c5bc9815076c037497fd9678a"}, {"shape": [1, 48, 128, 128], "sha256": "e57bcb1b6ccc4bb49d6c8c792fd854114696365377ca09e386aae9663fe62111"}], [{"shape": [1, 3, 10240], "sha256": "aff052554b4f8095bac69952dc9637b28b28ccc03bf27ac26bb98da8be635769"}, {"shape": [1, 48, 128, 128], "sha256": "2a2975148dfa647ca7af6ffaa3d6b82a6ee4e46f1b3d08e8d941c6073b8dc708"}], [{"shape": [1, 2, 88, 256], "sha256": "c36614a9b99f84ef13a7550abd83a81bc1fe4b2c619418f7eb78ba5f630af56b"}, {"shape": [1, 2, 88, 256], "sha256": "c00d7aa398b8e627be8c98584509c04e2d668cbe85cff3c7032e46cb0148903c"}, {"shape": [1, 88, 128], "sha256": "156d95137a26b9ebb84b2ccaeea64e145d45f5557950c86da0d19e776fc1bc5f"}, {"shape": [1, 22, 128], "sha256": "945c939ef933319f802fde62d6db22728c35738821d21647ac64c3ee30ed65b9"}], [{"shape": [1, 3, 10240], "sha256": "457d091ab375ff69cd61041c116a903d6a7ecac75a12b3f77d92411b05dd19cb"}, {"shape": [1, 48, 128, 128], "sha256": "6d43c9ebfc0ad35176429ffb7379a80f0d1d4e44830655a1a66408230abe90bf"}], [{"shape": [1, 3, 10240], "sha256": "8edba564162e6152c3cada686e749f37f4ba50a999b1c1cb93a41fc13132e191"}, {"shape": [1, 48, 128, 128], "sha256": "cc1e836d7be25016bdd272aea2674735026363b02a9e49b088ca54c9e419ecaf"}], [{"shape": [1, 3, 10240], "sha256": "8ca7077aef12bb5981f3dcbf42f45d8d961ad65248f74bca91f7c271a3c73b9c"}, {"shapLine truncated

docs/issue114-snapshot-zero/glm.log

@@ -0,0 +1,58 @@
+
+running 1 test
+test engine::metal::glm::tests::glm53_two_row_verifier_restores_recurrent_state ... ds4: Metal device Apple M5 Max, 128.00 GiB RAM
+ds4: Metal 4 tensor API enabled for Tensor kernels
+ds4: drift-patch flags hc_stable=on norm_unify=on kv_raw_f32=off rope_exp2_log2=off math_safe=off tensor_matmul=on
+ds4: Metal model views created in 2.052 ms, residency requested in 10681.125 ms, warmup 3.104 ms (mapped 92026.06 MiB from offset 9.07 MiB)
+ds4: Metal mapped mmaped model as 2 overlapping shared buffers
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+partial prefill HC workspace guards passed
+partial prefill HC workspace guards passed
+partial prefill HC workspace guards passed
+partial prefill HC workspace guards passed
+partial prefill HC workspace guards passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP verify2 at 12: 43.795 ms, draft 729, target 729
+ds4: GLM MTP draft 387 at position 13 in 2.4 ms
+verify2 accepted and rewind keep=0 passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP verify2 at 12: 38.788 ms, draft 729, target 729
+ds4: GLM MTP draft 387 at position 13 in 2.3 ms
+verify2 accepted and rewind keep=1 passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP verify2 at 12: 38.730 ms, draft 730, target 729
+ds4: GLM MTP draft 5733 at position 12 in 2.4 ms
+verify2 rejected and scalar replay state passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP draft 5733 at position 12 in 2.4 ms
+verify2 scalar boundary cap=1 cancelled=false passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP draft 5733 at position 12 in 2.4 ms
+verify2 scalar boundary cap=2 cancelled=true passed
+verify2 prefill 0
+verify2 prefill 4
+verify2 prefill 8
+verify2 prefill 12
+ds4: GLM MTP draft 5733 at position 12 in 2.4 ms
+engine-lifetime counters across new and restored chats passed
+ok
+
+test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 549 filtered out; finished in 12.69s

docs/issue114-snapshot-zero/known-issue115-gate-failure.log

@@ -0,0 +1,12 @@
+---- dev_brain::tests::source_versions_follow_that_file_instead_of_the_whole_worktree stdout ----
+
+thread 'dev_brain::tests::source_versions_follow_that_file_instead_of_the_whole_worktree' (3812331) panicked at src/dev_brain.rs:2054:51:
+called `Result::unwrap()` on an `Err` value: "Invalid frontmatter in concepts/answer.md: invalid type: floating point `5568000000.0`, expected a string"
+
+
+failures:
+    dev_brain::tests::source_versions_follow_that_file_instead_of_the_whole_worktree
+
+test result: FAILED. 312 passed; 1 failed; 237 ignored; 0 measured; 0 filtered out; finished in 5.62s
+
+error: test failed, to rerun pass `--bin ds4-server`

docs/issue114-snapshot-zero/off-cold-arrays.json

@@ -0,0 +1 @@
+{"trunk": [[{"shape": [1, 3, 10240], "sha256": "1dadce7f74c7b48ad423170b8884bf5024c3d6dd91b2f9d800afca7d32679d7b"}, {"shape": [1, 48, 128, 128], "sha256": "d7436faf9d50e278f7fa095e9c217467020a6f534a48195775c212caa0ece52d"}], [{"shape": [1, 3, 10240], "sha256": "ca1b3cf93598b62575f01aea05ff0d0f087c00fdf1b63654626ef5ec3487715a"}, {"shape": [1, 48, 128, 128], "sha256": "f1364bb8165b9189279b7769e4081f213c562f4e0541acc5962cf2ae9f42a831"}, {"shape": [1, 9, 10240], "sha256": "c9110c820fc39f8c19c770594c15e0997693015090a3c7af4b6c8ec9a31ebc4e"}, {"shape": [1, 2], "sha256": "9febaa06152dac026dba5f2c1151c11ff6a1b2f00e92278cdfd2905c87cf2c2f"}], [{"shape": [1, 3, 10240], "sha256": "320fdcf9c7ab899133952e093d1f3de613ba5581b3aaad45c4ad7b1b4e43f1f3"}, {"shape": [1, 48, 128, 128], "sha256": "8cae4ba20fd0e54f8da130e5f8681ece7c54672b7a6404800cded8e64a977e08"}], [{"shape": [1, 2, 58, 256], "sha256": "404510256f371f0bc2e7b8af2d1f7eb2ae0322d42eb9a0ee5827a9a5c8d489c0"}, {"shape": [1, 2, 58, 256], "sha256": "f14afc7fe1c5e4c62c84121e804e14aa484db378861904d4e234ffeef10252ef"}, {"shape": [1, 58, 128], "sha256": "ca2d0b2f09e87fab8d0a6aede7073ec53bc37b584851f0342c76244f05ca3e12"}, {"shape": [1, 14, 128], "sha256": "a609a4f52c9b3dc0fb5d6554019786bebecdad220bf5ae6b40d4939821f12c8b"}], [{"shape": [1, 3, 10240], "sha256": "0c5bdce8d9cbd2a24162139582f1889f0413632feafdba4e79cdd88353e6abc2"}, {"shape": [1, 48, 128, 128], "sha256": "d47c0c965fb6c41a7e1824144ef61546a7e40ead8d8606822173e21a17a4fc0f"}], [{"shape": [1, 3, 10240], "sha256": "66014c1bce9cbbd223ec6af2f5517de66f71ce443721088167dad0c0186289fb"}, {"shape": [1, 48, 128, 128], "sha256": "2593987c9ca9aeb30910e7eb69f8d6bd61c1e035861915cbe27b0838e26f3098"}], [{"shape": [1, 3, 10240], "sha256": "45a3dbbf3dc7a79b7db631fe4c2802d6e1f4433f4b03e961df072147e8c242a2"}, {"shape": [1, 48, 128, 128], "sha256": "e266e3456f01268de8868258a750ee18c5f8b1ff4391f647aec60a39d56f98a0"}], [{"shape": [1, 2, 58, 256], "sha256": "ba60b2860a5717f46a4a7ffef0257799f96acb37de1a00ba88ddfc149bf44ca2"}, {"shape": [1, 2, 58, 256], "sha256": "57d040a722a1427e9b0744ee652aecca9ec30cc7b617e3db9534e772d79f42e4"}, {"shape": [1, 58, 128], "sha256": "84d339a2f8558893f3d73e751d5ee3d81bb1f3b1184c0b35555bf5456fdc5a22"}, {"shape": [1, 14, 128], "sha256": "a5378133f448c4b6380dc4711420d3b069dd8491ffc797fd16e023294405b707"}], [{"shape": [1, 3, 10240], "sha256": "e6d4afab81c6a58f08c8d252991e91d25d2d60e497c3fc00614451edd0ea908c"}, {"shape": [1, 48, 128, 128], "sha256": "f94349b9898b0d1a5770f4aca791d4c114abad341641688012de2e2fcaa6237c"}], [{"shape": [1, 3, 10240], "sha256": "54d2e63c68f9fc1412178f3f21bcf5897ab228967e01f709ed9d5ba0e6d0c016"}, {"shape": [1, 48, 128, 128], "sha256": "868155f0fd6a62a46decb4f87c3e2f27310f9da38796fe0a8d926844248c7259"}], [{"shape": [1, 3, 10240], "sha256": "0073cc8754020915aedaad03803669a61281e18cb2dd389cb19ef7b5ed769b84"}, {"shape": [1, 48, 128, 128], "sha256": "abd539c013e58f489c23c48dfc168be7a9e2145deee859432cb83d0d8764e2df"}], [{"shape": [1, 2, 58, 256], "sha256": "d855a9efa61fd043cf03b444aefe90a1fcefa5af03770819333ec695af328dc5"}, {"shape": [1, 2, 58, 256], "sha256": "e2d0094873c5a47590a465e5c06fa7cab90e6ae99cb2eba7df88340755be2725"}, {"shape": [1, 58, 128], "sha256": "4ad0c060f771f328465a3dd5a6333d34fb9aa4cb9052c8c7811ab2dbd17d088e"}, {"shape": [1, 14, 128], "sha256": "2e108a4d5870102d8e2d4a9b69fe55507b0e75f12455792f2b05610fa305d638"}], [{"shape": [1, 3, 10240], "sha256": "8e0affe3773cbb974718f2a1dce2e75f60cc29c8290617c559b5f93122ae763d"}, {"shape": [1, 48, 128, 128], "sha256": "8293399e05778a26da5207d0dd00d1802ab2314ec846951bccdea731df48e8a7"}], [{"shape": [1, 3, 10240], "sha256": "7b9ed3fd1bf6ea7c486f62899e64152b9d24bc948a26fd02920f8ab823c077ea"}, {"shape": [1, 48, 128, 128], "sha256": "8ee1a98bfc2a5d85fef8c57b94d166755b3766c0fa223c635dbdc013e512c810"}], [{"shape": [1, 3, 10240], "sha256": "9daf0b317a2cd918d02808a7b7ffc304e4ca9217b12dc8921984c9e5dc4acb44"}, {"shape": [1, 48, 128, 128], "sha256": "bb3e8804e600bc41b69b14e4f89339e876f975fcd8e4caad4318ef88e424fd85"}], [{"shape": [1, 2, 58, 256], "sha256": "85ce674b27b84da8c85b6f50d944f7173c2fcfed8bb112cd518aa52fed92beea"}, {"shape": [1, 2, 58, 256], "sha256": "1a7b2de350076386b7499577e28d627164afe1b2e00f7a2fbbc1b1ea3dae35ab"}, {"shape": [1, 58, 128], "sha256": "fa5d6207583355c27b91d17b333b6f672d62a64dbab9ee3f787619a70a924c07"}, {"shape": [1, 14, 128], "sha256": "f1033fd4afbbe2909dada065e00234e58a3ad7a9d92eade7fd112c0adab4af24"}], [{"shape": [1, 3, 10240], "sha256": "95c15e85c7cbc93aade6e85ba021ba4a32383a7bd6e94cfa85d6376a2f1baff1"}, {"shape": [1, 48, 128, 128], "sha256": "8b5e6b8b2ca9f2d347a64581632db8fa98bf0ee2e0c5de6d8889964c6739b7d7"}], [{"shape": [1, 3, 10240], "sha256": "2509502b2ec7afecab230e5c532f1a034a8f0f42c2fafa57163c6da3ec308eed"}, {"shape": [1, 48, 128, 128], "sha256": "e73f022d9beaba67a5ebd2b54f0ac2ab982323756bc0876489c2256d5685e744"}], [{"shape": [1, 3, 10240], "sha256": "46817f7782dd86d4c6375cfd76bf18b6a0661ba43ff7a4360040729f076b8b81"}, {"shapLine truncated

docs/issue114-snapshot-zero/off-warm-arrays.json

@@ -0,0 +1 @@
+{"trunk": [[{"shape": [1, 3, 10240], "sha256": "1df54fa91eb80b88e8b4c0d566828d36540f3508099fc1751dd700d9bd904a84"}, {"shape": [1, 48, 128, 128], "sha256": "1cc66986fdbef7d449d321ff6c558b4e862d2d9211d74038325c72c4f2f0cebf"}], [{"shape": [1, 3, 10240], "sha256": "1ba5aa461d83294641ab065813f2a27ce3a1aa280890046cdc6d839d49c86232"}, {"shape": [1, 48, 128, 128], "sha256": "b789c69d02395058a02c548725e7897e906176c9647e88b5c413a9c743a73aaf"}, {"shape": [1, 9, 10240], "sha256": "eaffcb3f7958776a888b3134ef5f9fa1285f21c242d76b0ebc19c64d75d26c08"}, {"shape": [1, 2], "sha256": "b5e055307f70d9b61bc8427b1da30e8f711b4da0c73d28a34879de8c3f253152"}], [{"shape": [1, 3, 10240], "sha256": "1ebb9fb3628cf830f4c16a815ecaa0c9ccc2049bff2b14d32cb4e3f08f1b91b2"}, {"shape": [1, 48, 128, 128], "sha256": "403226ed2eeae9becd4b498b0fa14de8152beac9138bff0c118698cd4c4fcc34"}], [{"shape": [1, 2, 88, 256], "sha256": "b4995a3d443eb57598940e4a80ece426f8e8442e07f30b8168b247baebfc422f"}, {"shape": [1, 2, 88, 256], "sha256": "eb25629ce30b96850b39bf034124fb8d627c82402baeb89d72e24869005ace24"}, {"shape": [1, 88, 128], "sha256": "1561cf66bb84117ff09036ee128d5c2e5aa21f69db459b82000dc467527df748"}, {"shape": [1, 22, 128], "sha256": "4366612cb46ecc942d06aabe6043fea79b494a34f08f797fec5e0a76ff067a1a"}], [{"shape": [1, 3, 10240], "sha256": "431c351776e8973a28d2206fee9c9262c74df879b561ecb28fdf53f1c6941dda"}, {"shape": [1, 48, 128, 128], "sha256": "c8d64ecbf9dab273a4be2d0b7b00b4fccbb59d25d99ef47ba26a3488dc1ba28f"}], [{"shape": [1, 3, 10240], "sha256": "f4ce935821cd12c5981db92dc849ca6fb25d845a8b38f0ca2ac7f98afa99666a"}, {"shape": [1, 48, 128, 128], "sha256": "8093f937ad983da7c65154990a2a163c7ab09d7e776c7d2b84a8310469433d6f"}], [{"shape": [1, 3, 10240], "sha256": "3c1051a65bbd484bd5bc7ff79919bbb0f8b99f899215efc47ad415541c4ccc7d"}, {"shape": [1, 48, 128, 128], "sha256": "bcbf58854152187ff9556062117970701ac56898d9af860d2b9b3840eaf84b1f"}], [{"shape": [1, 2, 88, 256], "sha256": "9e6bc46ee9a8170bf6bfe9a0cd592543b15d24225ba39e71bddb4aeef947555b"}, {"shape": [1, 2, 88, 256], "sha256": "b84ee35f5c3e0ac8181df0ae66e2278a03db0783c0ea83e4bbabc21bdf47f7ba"}, {"shape": [1, 88, 128], "sha256": "491cccffe6e1c810e6ce8d32fe160005762d1964da055d58746febef7000f5ae"}, {"shape": [1, 22, 128], "sha256": "104bbc74bbb75851ad6a7466e2f2e1a1421a9c93fc0d62aaaf37aa60e2a1dcdb"}], [{"shape": [1, 3, 10240], "sha256": "e87be44991fddef0510651dc23006f2a056a56cf0a61a5f8b38daa34be0c8b57"}, {"shape": [1, 48, 128, 128], "sha256": "0645fe9e0d09d4d55d3919d729ea1340b4ade128dec572010ebb07a5de306f43"}], [{"shape": [1, 3, 10240], "sha256": "f2904788ae98c816e237db5ecc398c269205f38acb680a17e7d00b53289400d0"}, {"shape": [1, 48, 128, 128], "sha256": "75f9e87f1c559b714f5d07911976152045b4b283c50d49794c68cdc66a3cea52"}], [{"shape": [1, 3, 10240], "sha256": "c23b7e524522714aaf620c9e0bbb6b1d3db3585971f0cffafd7fe7e2018176dc"}, {"shape": [1, 48, 128, 128], "sha256": "4ce5d0905ec5e5ae834215c16b25d810a5ef6e54a6d7d96f95e477209775ddaa"}], [{"shape": [1, 2, 88, 256], "sha256": "7803bca47639bdcb533d5985f37930fb74cd112882f8cfa26513bc297e1245fe"}, {"shape": [1, 2, 88, 256], "sha256": "40bb50faa5d147165f88cd5ba2379dc395f4f296c66246499b9efd03c9eef7ac"}, {"shape": [1, 88, 128], "sha256": "6e20787bdd02bd9615279eac42cf9f2664430aeb8cdd8c93c6a9881556722493"}, {"shape": [1, 22, 128], "sha256": "c17459f225244530703a28076f3006f7e53d50e728ecb79ae4f5b968b594d96d"}], [{"shape": [1, 3, 10240], "sha256": "8e1e94ade5f7453693e4d99f003d190122a3cd69b0bddca1a1fc2409d9a32b7d"}, {"shape": [1, 48, 128, 128], "sha256": "976efc317438cb07b1e6d80ce6ae12d8e0095a51fa7b490b8a84a8e0d6b79399"}], [{"shape": [1, 3, 10240], "sha256": "6153601aa51b613aa617cb823b2d5e582847afba59eac2c190a96ea465ad77e1"}, {"shape": [1, 48, 128, 128], "sha256": "077106a1cc1cf3e3d24ba44378f788cab30d80fd38aef87fcf957a99b662379d"}], [{"shape": [1, 3, 10240], "sha256": "d020e70e7cab233eb7d188fa42e975f0a20cdce167f52db6043933e583f7c280"}, {"shape": [1, 48, 128, 128], "sha256": "c05cdc62e13b7c238ad02e2a4a81286dc5d0fd8cd9c91af6c8ea6380d203f080"}], [{"shape": [1, 2, 88, 256], "sha256": "be1eed7cb68714c49599bea49ffdc831b5caac8444781088d21c87892445d2d7"}, {"shape": [1, 2, 88, 256], "sha256": "abe8a391f5a932dee7718ea3abe5419e015750b85a522856f1a59f435d5688dc"}, {"shape": [1, 88, 128], "sha256": "c1cd1549e2032d2bc28d2d0c9f1bfeb608f81fe8abe2b18b77a1658a4c0ed9c4"}, {"shape": [1, 22, 128], "sha256": "072bc306a1b4a56332dc90d889c42e6b2037a820330da0b897815a3750c00043"}], [{"shape": [1, 3, 10240], "sha256": "7f4500e05eb6e52b25bd9405ccaae398fcc426835af0550599175d0785694154"}, {"shape": [1, 48, 128, 128], "sha256": "e33affece02b985e41ea0d4ce467679226e9f9b5f55c4001cdbf8ebcb087db40"}], [{"shape": [1, 3, 10240], "sha256": "d133a043fa3abeaf2498e4336142e34a6d3222e4b2eec943931e97eb060f5657"}, {"shape": [1, 48, 128, 128], "sha256": "465ca838c8b5abbaac2b32a5f2f204ebe674d20ff50ed9b3532e81ede40c1164"}], [{"shape": [1, 3, 10240], "sha256": "379bfa2a27e16bc7d2d1213c0d145307954a9d4d36d8c5f5a6cb9b4f172b400c"}, {"shapLine truncated

docs/issue114-snapshot-zero/provenance.json

@@ -0,0 +1,12 @@
+{
+  "baseline": "d9026aa7bfbc8f74ec82c120178c34c5f08510d3",
+  "oracle": "9882703f3105363ddc37eca9f97aa09a1d387112",
+  "mlx": "0.32.2",
+  "sha256": {
+    "tests/fixtures/mtplx-snapshot-zero.json": "af9a81d1fb78b6d2aa7e32a584020e4ba40ffc2a11dd68673dcff9da2294dc15",
+    "/private/tmp/ds4-issue114-test": "8ce50cce2bbb6243371f49c85c2d0a56765f356d485c8a890359cf4c216d0253",
+    "tools/issue109-model-reference.py": "19d8afab17e8aea7757c0df0ab3870f92cdab7784ce7547d5444ac9ad223e2e8",
+    "tools/issue113-model-reference.py": "df13f2d81a81707d4453b31e443e65054666f97b55d8a4e24aab128640a8f167",
+    "tools/issue114-snapshot-reference.py": "b52326794e4f69c911cd0ed240a33fd974778cb3739fda74141988e300566bfa"
+  }
+}

docs/issue114-snapshot-zero/resources.json

@@ -0,0 +1,106 @@
+{
+  "snapshot-oracle": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 871,
+    "peak": {
+      "cpu_time_ns": 508869208,
+      "resident_bytes": 83951616,
+      "physical_bytes": 63521416,
+      "peak_physical_bytes": 63521416,
+      "disk_read_bytes": 77074432,
+      "disk_write_bytes": 0
+    },
+    "error": null
+  },
+  "native-zero": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 1172,
+    "peak": {
+      "cpu_time_ns": 55972750,
+      "resident_bytes": 61161472,
+      "physical_bytes": 32096952,
+      "peak_physical_bytes": 32096952,
+      "disk_read_bytes": 11984896,
+      "disk_write_bytes": 0
+    },
+    "error": null
+  },
+  "off-product": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 10788,
+    "peak": {
+      "cpu_time_ns": 13820742750,
+      "resident_bytes": 25308233728,
+      "physical_bytes": 77012314160,
+      "peak_physical_bytes": 77012314160,
+      "disk_read_bytes": 73724788736,
+      "disk_write_bytes": 286756864
+    },
+    "error": null
+  },
+  "off-oracle": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 14634,
+    "peak": {
+      "cpu_time_ns": 16676218875,
+      "resident_bytes": 25308561408,
+      "physical_bytes": 77094332464,
+      "peak_physical_bytes": 77095757920,
+      "disk_read_bytes": 104186204160,
+      "disk_write_bytes": 32649216
+    },
+    "error": null
+  },
+  "default-product": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 9968,
+    "peak": {
+      "cpu_time_ns": 12783430375,
+      "resident_bytes": 26438025216,
+      "physical_bytes": 77014706176,
+      "peak_physical_bytes": 77014706176,
+      "disk_read_bytes": 73467215872,
+      "disk_write_bytes": 286756864
+    },
+    "error": null
+  },
+  "default-oracle": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 14323,
+    "peak": {
+      "cpu_time_ns": 16661781333,
+      "resident_bytes": 24527880192,
+      "physical_bytes": 76986902600,
+      "peak_physical_bytes": 76987639712,
+      "disk_read_bytes": 101799014400,
+      "disk_write_bytes": 32649216
+    },
+    "error": null
+  },
+  "deepseek": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 9628,
+    "peak": {
+      "cpu_time_ns": 2803177125,
+      "resident_bytes": 130236416,
+      "physical_bytes": 314032992,
+      "peak_physical_bytes": 319603552,
+      "disk_read_bytes": 86729781248,
+      "disk_write_bytes": 0
+    },
+    "error": null
+  },
+  "glm": {
+    "event": "test_resource_summary",
+    "elapsed_ms": 12757,
+    "peak": {
+      "cpu_time_ns": 3682264416,
+      "resident_bytes": 905068544,
+      "physical_bytes": 1335248384,
+      "peak_physical_bytes": 1488520824,
+      "disk_read_bytes": 96510414848,
+      "disk_write_bytes": 0
+    },
+    "error": null
+  }
+}

docs/issue114-snapshot-zero/tests.log

@@ -0,0 +1,592 @@
+    Finished `test` profile [unoptimized + debuginfo] target(s) in 0.19s
+warning: the following packages contain code that will be rejected by a future version of Rust: block v0.1.6
+note: to see what the problems were, use the option `--future-incompat-report`, or run `cargo report future-incompatibilities --id 1`
+     Running unittests src/main.rs (target/debug/deps/ds4_server-5679c54d8848b708)
+
+running 550 tests
+test a2ui::tests::basic_formatters_follow_catalog_examples ... ok
+test agent::tests::ai_risk_response_requires_structured_risk_and_reason ... ok
+test a2ui::tests::streaming_parser_only_accepts_complete_a2ui_jsonl_lines ... ok
+test agent::tests::anchored_edits_require_unique_head_and_tail ... ok
+test agent::tests::ai_shell_risk_assessment_receives_the_dev_brain_root ... ok
+test a2ui::tests::latest_created_surface_is_the_active_surface ... ok
+test agent::tests::dev_brain_paths_are_trusted_by_shell_risk_assessment ... ok
+test a2ui::tests::dismissal_boundaries_replay_history_and_a_fresh_active_epoch ... ok
+test agent::tests::model_adapters_preserve_primitives_and_report_recoverable_syntax_errors ... ok
+test a2ui::tests::embedded_catalog_and_validator_cover_the_same_components ... ok
+test a2ui::tests::catalog_validation_rejects_invalid_variants_and_function_arguments ... ok
+test a2ui::export::tests::static_svg_renderer_covers_the_complete_catalog ... ok
+test a2ui::tests::catalog_accepts_pie_donut_and_heatmap_charts ... ok
+test a2ui::tests::catalog_validation_and_actions_use_current_local_data ... ok
+test agent::tests::present_svg_parsers_recover_exact_source_for_both_model_transports ... ok
+test a2ui::tests::all_basics_compose_into_one_surface ... ok
+test a2ui::tests::lifecycle_updates_incrementally_and_json_pointer_upserts ... ok
+test a2ui::tests::v1_action_and_function_round_trips_follow_current_schema ... ok
+test agent::tests::ralph_loop_stops_on_cap_blocker_failure_and_cancellation ... ok
+test agent::tests::risky_shell_commands_require_one_time_approval ... ok
+test agent::tests::scripted_ralph_backend_gets_fresh_prompts_and_only_validated_handoffs ... ok
+test agent::tests::generated_schemas_match_the_executable_tool_contracts ... ok
+test agent::tests::ralph_report_semantics_and_handoff_bound_are_enforced ... ok
+test agent::tests::present_svg_executes_without_side_effects_and_maps_only_completed_cards ... ok
+test agent::tests::repeated_call_advisories_use_canonical_arguments_and_reset_per_user_turn ... ok
+test agent::tests::stored_tool_cards_restore_ai_approval_reasons_by_call ... ok
+test agent::tests::ralph_report_schemas_are_child_only_and_parse_for_both_models ... ok
+test agent::tests::present_svg_validation_enforces_the_static_vector_boundary ... ok
+test agent::tests::project_boundary_rejects_parent_paths ... ok
+test agent::tests::tool_output_keeps_terminal_lines_but_parameter_summaries_do_not ... ok
+test agent::web::tests::headless_browser_extracts_a_local_page ... ignored, requires an installed Chrome browser
+test agent::web::tests::browser_helpers_preserve_ds4_wire_values ... ok
+test agent::tests::glob_confinement_skips_symlinks_and_honors_cancellation ... ok
+test agent::tests::prompts_and_parsers_expose_the_reference_tool_set ... ok
+test app::generation::tests::attached_image_marker_is_model_only ... ok
+test agent::tests::validation_is_precise_and_prevents_side_effects_until_corrected ... ok
+test agent::tests::successful_structured_file_tools_report_project_paths ... ok
+test agent::tests::standard_agent_skills_are_discovered_prompted_and_read_only ... ok
+test agent::tests::shell_commands_skip_user_startup_files ... ok
+test agent::tests::glob_finds_all_regular_file_names_deterministically_without_reading_them ... ok
+test app::generation::tests::plain_chat_uses_the_same_unmodified_prompt_in_ui_and_headless ... ok
+test app::generation::tests::queued_guidance_is_one_reference_style_user_turn ... ok
+test app::generation::tests::title_context_starts_at_the_first_user_and_ignores_system_rows ... ok
+test app::generation::tests::turn_summary_accumulates_model_continuations ... ok
+test agent::tests::ralph_round_contexts_are_unique_and_child_jobs_are_cleaned_up ... ok
+test agent::tests::standard_file_tools_can_maintain_only_managed_dev_brain_pages ... ok
+test agent::tests::shell_environment_is_loaded_once_from_login_and_interactive_startup_files ... ok
+test app::git::tests::unified_diff_parser_aligns_replacements_for_split_view ... ok
+test app::git::tests::configured_git_diff_options_change_the_rendered_patch ... ok
+test app::git::tests::unstage_works_before_the_first_commit ... ok
+test app::preferences::tests::runtime_config_updates_after_lock_poisoning ... ok
+test app::git::tests::libgit2_status_merges_staged_and_worktree_changes ... ok
+test app::projects::tests::discarding_a_session_removes_every_checkpoint_stage ... ok
+test app::projects::tests::draft_titles_follow_the_stored_session_count ... ok
+test app::projects::tests::git_state_lists_and_switches_local_branches ... ok
+test app::projects::tests::session_states_are_exclusive_and_ordered ... ok
+test app::projects::tests::sweeping_removes_checkpoints_of_sessions_that_no_longer_exist ... ok
+test agent::tests::denial_and_stop_cancel_commands_awaiting_approval ... ok
+test app::tests::background_activity_drives_polling_indicators_and_quit_confirmation ... ok
+test app::git::tests::origin_fetch_pull_and_push_use_the_current_branch ... ok
+test app::generation::tests::model_history_uses_persisted_content_without_changing_the_transcript ... ok
+test app::tests::chat_html_export_embeds_loaded_a2ui_images_as_data_urls ... ok
+test app::tests::chat_export_preserves_completed_svg_and_keeps_failures_generic ... ok
+test app::tests::chat_only_follows_updates_while_near_the_tail ... ok
+test app::tests::ds4_gib_and_streaming_cache_inputs_are_typed ... ok
+test app::tests::one_shot_titles_are_reduced_to_a_single_sidebar_line ... ok
+test app::tests::owned_runtime_paths_stay_in_application_support ... ok
+test app::tests::preferences_shortcut_and_dspark_support_are_explicit ... ok
+test agent::tests::shell_environment_probe_stops_blocked_startup_files ... ok
+test app::tests::stored_session_always_selects_its_model ... ok
+test app::tests::tab_scrolls_a_field_back_into_the_preferences_viewport ... ok
+test app::view::a2ui::tests::renderer_input_helpers_cover_dates_numbers_choices_and_weights ... ok
+test app::view::chat::tests::chat_divider_groups_token_counts_by_thousands ... ok
+test app::view::chat::tests::composer_uses_shift_enter_for_line_breaks ... ok
+test app::view::git::tests::split_diff_width_covers_the_longest_side_and_tabs ... ok
+test app::tests::assistant_stream_keeps_reasoning_separate_from_the_answer ... ok
+test app::view::tests::action_buttons_share_geometry ... ok
+test app::view::tests::context_pie_wedge_follows_the_fill_grade ... ok
+test app::view::tests::download_measurements_are_readable ... ok
+test app::view::tests::raised_surfaces_stay_dark ... ok
+test compaction::tests::a_new_summary_replaces_the_previous_rebuild_summary ... ok
+test compaction::tests::hard_trigger_and_retry_reserve_match_the_reference ... ok
+test compaction::tests::soft_trigger_matches_reference_thresholds ... ok
+test compaction::tests::summary_budget_keeps_answer_room_and_rejects_exhausted_contexts ... ok
+test compaction::tests::summary_control_markup_is_private ... ok
+test compaction::tests::tail_prefers_the_first_user_boundary_after_target ... ok
+test config::tests::defaults_write_an_empty_file_and_reload_unchanged ... ok
+test config::tests::deprecated_flash_config_migrates_and_persists_only_0731 ... ok
+test app::preferences::tests::multiline_system_prompt_round_trips_through_the_editor ... ok
+test config::tests::invalid_files_are_reported_instead_of_reset ... ok
+test config::tests::explicit_0731_dspark_opt_out_survives_the_default ... ok
+test config::tests::model_profiles_expose_only_their_upstream_thinking_modes ... ok
+test config::tests::legacy_flat_model_settings_are_dropped_but_the_prompt_is_kept ... ok
+test app::generation::tests::assistant_markdown_keeps_llm_tables ... ok
+test app::tests::chat_export_formats_visible_turns_reasoning_and_bash_tools ... ok
+test config::tests::only_changed_values_are_written_and_read_back ... ok
+test config::tests::qwen_selection_and_native_defaults_survive_restart ... ok
+test app::generation::tests::last_compaction_selects_its_tail_without_hiding_history ... ok
+test app::preferences::tests::selectors_rehydrate_their_generation_and_model_profiles ... ok
+test app::generation::tests::legacy_generation_rows_collapse_into_their_user_turn ... ok
+test app::generation::tests::complete_tool_call_in_reasoning_gets_one_correction ... ok
+test app::view::stats::tests::current_session_summary_uses_persisted_chat_facts ... ok
+test app::tests::chat_html_export_is_safe_standalone_and_renders_embedded_surfaces ... ok
+test app::generation::tests::a2ui_auto_switch_waits_for_a_renderable_surface ... ok
+test app::generation::tests::extension_context_deduplicates_session_start_and_rearms_after_compaction ... ok
+test app::generation::tests::forced_compaction_requires_new_visible_chat ... ok
+test agent::tests::oversized_shell_output_is_bounded_and_retained_until_cleanup ... ok
+test dev_brain::tests::skill_index_migration_does_not_overwrite_an_unrelated_root_note ... ok
+test dev_brain::tests::path_confinement_and_unrelated_notes_are_preserved ... ok
+test app::tests::source_does_not_depend_on_a_sibling_ds4_checkout ... ok
+test dsml::tests::parses_reference_and_legacy_dsml_syntaxes ... ok
+test dsml::tests::recovers_complete_invokes_before_an_incomplete_one ... ok
+test dsml::tests::recovers_complete_invokes_without_the_outer_closing_tag ... ok
+test engine::ds4_sampling::tests::heap_fallback_preserves_rng_and_signed_zero_ties ... ok
+test engine::gguf::tests::accepts_exact_mxfp4_block_layout ... ok
+test engine::gguf::tests::keeps_unknown_tensor_kinds_rejected ... ok
+test engine::gguf::tests::maps_metadata_and_tensor_payload_without_copying_weights ... ok
+test engine::gguf::tests::mxfp4_scalar_dot_matches_independent_values ... ok
+test engine::gguf::tests::rejects_mxfp4_dimension_overflow ... ok
+test engine::gguf::tests::rejects_mxfp4_rows_that_are_not_block_aligned ... ok
+test engine::gguf::tests::rejects_truncated_mxfp4_payload ... ok
+test engine::kvstore::tests::budget_evicts_the_least_valuable_unprotected_entry ... ok
+test engine::kvstore::tests::longest_compatible_prefix_wins_and_hits_survive_reopen ... ok
+test engine::kvstore::tests::opening_reclaims_checkpoints_the_index_cannot_reach ... ok
+test engine::metal::glm::tests::dsa_indexer_schedule_matches_the_reference ... ok
+test engine::metal::glm::tests::glm53_m5_text_performance_gate ... ignored, requires the installed GLM 5.3 Flash Q2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm53_m5_vision_performance_and_oracle_gate ... ignored, requires GLM 5.3 Q2, its vision sidecar, a fixed image, and Apple Metal
+test engine::metal::glm::tests::glm53_reference_logits_replay_separates_sampling_from_execution ... ignored, requires the installed GLM 5.3 Flash Q2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm53_reference_prompt_tokens_match_shared_runtime ... ignored, installed GLM and DS4SERVER_GLM_REFERENCE_LOG from the standalone reference
+test engine::metal::glm::tests::glm53_two_row_verifier_restores_recurrent_state ... ignored, installed GLM 5.3 and Metal; run under test-supervisor
+test engine::metal::glm::tests::glm_byte_budget_reserves_prefill_before_dynamic_experts ... ok
+test engine::metal::glm::tests::glm_mtp_preserves_target_tokens_and_drafts ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::glm::tests::glm_prefill_schedule_matches_ds4_boundaries ... ok
+test engine::metal::glm::tests::glm_resident_decode_flushes_periodically_without_a_final_or_ssd_flush ... ok
+test engine::metal::glm::tests::repeated_glm_prompt_rewinds_one_token_for_logits ... ok
+test engine::metal::glm::tests::resident_and_streamed_glm_match_ds4_decode_oracles ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::glm::tests::streamed_glm_uses_ds4_indexed_prefill_for_long_prompts ... ignored, requires the 197 GiB GLM 5.2 checkpoint and Apple Metal
+test engine::metal::gpu::qwen_mtplx_allocation_tests::backend_allocation_policy_is_explicit_inside_and_outside_qwen_scope ... ignored, requires Apple Metal; allocation only, no GPU dispatch or model
+test engine::metal::gpu::qwen_mtplx_allocation_tests::dispatch_retains_allocation_without_changing_data_donation ... ignored, requires Apple Metal, no model weights
+test engine::metal::gpu::qwen_mtplx_allocation_tests::empty_storage_preserves_data_identity_views_and_zero_allocator_usage ... ignored, requires Apple Metal context; empty descriptors, no GPU allocation
+test engine::metal::gpu::qwen_mtplx_allocation_tests::mtplx_tensor_cpu_access_releases_native_storage ... ignored, requires Apple Metal; CPU buffer access only, no GPU dispatch or model
+test engine::metal::markov::tests::installed_markov_worker_dispatch ... ignored, requires installed 0731 DSpark support; CPU dispatch diagnostic, not parity acceptance
+test database::tests::context_work_reactivates_an_archived_session ... ok
+test engine::metal::power_throttle_keeps_ds4_ewma_and_full_power_bypass ... ok
+test database::tests::inline_svg_tool_call_survives_session_switch_compaction_and_reopen ... ok
+test engine::metal::qwen::tests::affine_split_k_matches_reference_shapes_and_scratch_bound ... ok
+test engine::metal::qwen::tests::dense_attention_crossover_is_phase_specific ... ok
+test database::tests::sessions_are_ordered_by_recent_use_within_sidebar_groups ... ok
+test engine::metal::qwen::tests::qwen_affine_qmv_and_qmm_match_mlx_reference ... ignored, requires the pinned Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_affine_qmv_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_core_boundary_and_checkpoint_are_stable ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_fused_gdn_input_rows_match_separate_projections ... ignored, requires the pinned Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_gdn_conv_norm_rows_matches_mtplx ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_gdn_delta_rows_matches_mtplx ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_layer_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_lazy_target_probabilities_match_batched_sampling ... ignored, requires the pinned Qwen metadata and Apple Metal; no model inference
+test engine::metal::qwen::tests::qwen_m5_performance_gate ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_metal_primitives_match_reference_vectors ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_mtp_matches_target_and_restores_its_cache ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_ple_gather_copies_only_the_requested_row ... ok
+test engine::metal::markov::tests::markov_workers_reuse_threads_and_buffers_with_ordered_ties ... ok
+test engine::metal::qwen::tests::qwen_qsa_dense_mask_accepts_unsorted_blocks ... ignored, requires Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_full_context_preallocation_succeeds ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_logical_depth_oracles_cover_prefill_decode_and_state ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_qsa_real_layer_executes_the_first_sparse_token ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qsa_selection_and_capacity_cover_the_full_native_window ... ok
+test engine::metal::qwen::tests::qwen_qsa_selection_and_attention_remain_bounded_at_native_depths ... ignored, requires Apple Metal and allocates the full QSA score window
+test engine::metal::qwen::tests::qwen_reference_layer_trace ... ignored, diagnostic trace against the pinned MLX-LM reference
+test engine::metal::qwen::tests::qwen_reference_sdpa_trace ... ignored, diagnostic comparison with captured MTPLX SDPA inputs
+test engine::metal::qwen::tests::qwen_reference_stateful_trace ... ignored, diagnostic trace against the pinned MLX-LM reference
+test engine::metal::qwen::tests::qwen_reference_verify_trace ... ignored, diagnostic trace against the pinned MTPLX verifier
+test engine::metal::qwen::tests::qwen_sorted_routed_prefill_matches_grouped_output ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_stage_counters_preserve_async_batches ... ignored, requires pinned Qwen embeddings and Apple Metal; stage profiler opt-in
+test engine::metal::qwen::tests::qwen_target_submission_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen::tests::qwen_verify_layer_profile ... ignored, requires the pinned 105 GB Qwen artifact set and Apple Metal
+test engine::metal::qwen_mtplx::allocator::mtplx_allocations_preserve_logical_bounds_with_reference_rounding ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::allocator::mtplx_native_allocator_heap_limits_and_gpu_lifetime ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx::allocator::mtplx_native_allocator_replays_reference_allocations ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::ar::mtplx_ar_driver_matches_reference ... ignored, requires Apple Metal; unchanged MTPLX generate_ar orchestration oracle
+test engine::metal::qwen::tests::qwen_ple_hash_contract_matches_golden_boundaries ... ok
+test engine::metal::qwen_mtplx::array::mtplx_array_detach_clears_every_siblings_inputs_and_primitive ... ok
+test engine::metal::qwen_mtplx::array::mtplx_array_materialization_keeps_geometry_and_rejects_short_storage ... ignored, requires Apple Metal; typed materialized descriptors without model weights
+test engine::metal::qwen_mtplx::array::mtplx_array_layout_preserves_dtype_shape_stride_and_materialization_flags ... ok
+test engine::metal::qwen_mtplx::array::mtplx_dtype_promotion_matches_all_reference_pairs ... ok
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_dense_mask_graph_matches_reference ... ignored, requires Apple Metal; original MTPLX block mask including sentinel and tails
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_eager_array_graph_matches_reference ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_prefill_gather_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX tiled prefill gather
+test engine::metal::qwen_mtplx::attention::mtplx_qsa_rows_gather_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX QSA rows-gather attention
+test engine::metal::qwen_mtplx::attention::mtplx_sdpa_fallback_graph_matches_reference ... ignored, requires Apple M5 Max; actual Qwen SDPA fallback reference
+test engine::metal::qwen_mtplx::binary::mtplx_graph_binary_preserves_reference_branches ... ignored, requires Apple Metal; binary graph broadcast/layout and donation branches
+test engine::metal::qwen_mtplx::binding::mtplx_installed_module_binding_matches_reference ... ignored, requires installed DS4_QWEN38_ARTIFACTS and SSD policy; only small norms and four table rows evaluated
+test engine::metal::qwen_mtplx::cache::mtplx_kv_cache_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX KVCache growth, trim and state receipts
+test engine::metal::qwen_mtplx::cache::mtplx_qsa_cache_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX positional cache, mirror and held state aliases
+test engine::metal::qwen_mtplx::compile_graph::mtplx_compile_scalar_bits_and_custom_kernel_identity ... ignored, requires Metal buffers and external supervisor; scalar ownership only, no GPU dispatch
+test engine::metal::qwen_mtplx::allocator::mtplx_allocator_cache_matches_live_reference_trace ... ok
+test engine::metal::qwen_mtplx::compiled::mtplx_replay_replaces_descriptors_without_rebuilding_primitives ... ok
+test engine::metal::qwen_mtplx::context_copy::prompt_copy_match_and_backoff_preserve_reference_policy ... ok
+test engine::metal::qwen_mtplx::decoder::mtplx_decoder_connected_attention_and_moe_advance_caches ... ignored, requires Apple Metal and external test-supervisor; complete decoder/cache smoke
+test engine::metal::qwen_mtplx::decoder::mtplx_decoder_residual_preserves_graph_order ... ok
+test engine::metal::qwen_mtplx::dense::mtplx_dense_attention_batches_match_reference ... ignored, requires Apple M5 Max; actual BF16 batched matmul reference
+test engine::metal::qwen_mtplx::dense::mtplx_dense_float_graph_matches_reference ... ignored, requires Apple M5 Max; floating Matmul factory and original dispatch
+test engine::metal::qwen_mtplx::dense::mtplx_qsa_score_array_graph_matches_reference ... ignored, requires Apple M5 Max; canonical QSA F32 graph with the original TF32 setting
+test database::tests::a2ui_dismissal_persists_a_fresh_surface_boundary ... ok
+test engine::metal::qwen_mtplx::encoder::mtplx_empty_bindings_preserve_nil_dependencies_without_dummy_storage ... ignored, requires Apple Metal; empty original-kernel grids and nil bindings
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_commits_after_primitives_and_balances_completion ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_barrier_epochs_and_commit_thresholds ... ok
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_scope_routes_operators_and_allocator ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_shared_events_and_error_completion_order ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::encoder::mtplx_encoder_untracked_dependencies_across_encoders_and_commands ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_drains_failure_preserves_cause_and_retries_only_failed_nodes ... ignored, requires Apple Metal; supervised evaluator failure and retry
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_releases_intermediates_and_preserves_descriptor_donation ... ignored, requires Apple Metal; graph ownership and last-consumer release
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_retains_tracer_topology_without_input_donation ... ignored, requires Apple Metal; evaluator tracer donation and retained topology
+test engine::metal::qwen_mtplx::eval::execution_tests::mtplx_eval_runs_cpu_gpu_siblings_and_async_continuations ... ignored, requires Apple Metal; tiny supervised CPU/GPU evaluator graph
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_preserves_scheduled_detach_and_tracer_errors ... ok
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_preserves_bfs_width_and_sibling_choice ... ok
+test engine::metal::qwen_mtplx::eval::mtplx_eval_tape_tracks_stream_fences_per_leaf_and_cross_device_union ... ok
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_ar_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_AR_REFERENCE required
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_mtp_prefill_matches_reference ... ignored, full installed MTP model; external guard and DS4_QWEN38_MTP_PREFILL_REFERENCE required
+test engine::metal::qwen_mtplx::execution::mtplx_full_installed_ping_forward_matches_reference ... ignored, full installed model; requires external memory guard and DS4_QWEN38_FORWARD_REFERENCE
+test engine::metal::qwen_mtplx::execution::qwen_power_throttle_drains_before_sleep_and_bypasses_at_full_power ... ignored, requires Apple Metal; run under test-supervisor
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_constant_spelling ... ok
+test database::tests::session_model_migration_preserves_data_and_accepts_catalog ... ok
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_fusion_gpu_matches_reference ... ignored, requires Metal and external supervisor; generated fusion vs pinned SiLU plus multi-output/scalar dispatch
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_forward_matches_model_reference ... ignored, requires Apple Metal; actual MTPLX GDN forward and cache receipts
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_prefill_and_verify_match_model_reference ... ignored, requires Apple Metal; complete GDN Conv1d prefill and fused verify graphs
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_staged_primitives_match_reference ... ignored, requires Apple Metal; 80 actual delta-update receipts and six conv-normalization kernels
+test engine::metal::qwen_mtplx::gdn::mtplx_graph_gdn_step_preserves_views_and_siblings ... ignored, requires Apple Metal; GDN custom primitive strided copies and sibling roots
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_event_snapshots_preserve_values_streams_and_shared_errors ... ignored, requires Apple Metal events, no model weights
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_fence_coherence_preserves_empty_and_partial_word_spans ... ignored, requires Apple Metal3, model-free empty coherent dispatch
+test engine::metal::qwen_mtplx::graph_sync::mtplx_graph_fences_transfer_cpu_gpu_gpu_cpu_in_both_modes ... ignored, requires Apple Metal3, tiny model-free coherent fence kernels
+test engine::metal::qwen_mtplx::hyper::mtplx_graph_hyper_forward_matches_reference ... ignored, requires Apple M5 Max; actual MTPLX GatedResidual reference receipts
+test engine::metal::qwen_mtplx::hyper::mtplx_graph_hyper_pack_and_reads_match_reference ... ignored, requires Apple Metal; original hyper primitives and runtime pack
+test engine::metal::qwen_mtplx::indexing::mtplx_graph_indexing_preserves_layouts_and_offsets ... ignored, requires Apple Metal; Select layout branches, GatherAxis views and empty arguments
+test engine::metal::qwen_mtplx::indexing::mtplx_scatter_axis_graph_matches_reference ... ignored, requires Apple Metal; pinned Bool ScatterAxis layout and broadcast paths
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_numeric_receipts ... ignored, #103 diagnostic; Metal required, eager versus compiled numeric operands
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_sampling_receipts ... ignored, #103 diagnostic; Metal required, pins currently broken sampling law and invalid rows
+test engine::metal::qwen_mtplx::fused::mtplx_compile_fusion_preserves_parent_and_output_boundaries ... ok
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_sorted_gather_receipts ... ignored, #103 diagnostic; M5 and pinned runtime shader required; initialized sentinels
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_tokenizer_receipts ... ignored, #103 diagnostic; installed tokenizer and latest oracle JSONL; no GPU
+test engine::metal::qwen_mtplx::issue103_analysis::issue103_vision_position_receipts ... ignored, #103 diagnostic; small active attention module on Metal; mixed position boundary
+test engine::metal::qwen_mtplx::fused::mtplx_compiled_emitter_matches_all_pinned_silu_bodies ... ok
+test engine::metal::qwen_mtplx::issue104::issue104_installed_ar_stream_and_state ... ignored, full pinned Bare-Speed model; external memory guard required
+test engine::metal::qwen_mtplx::issue105::issue105_batched_greedy_rejects_each_invalid_row ... ignored, Metal required; run serially under test-supervisor
+test engine::metal::qwen_mtplx::issue106::issue106_complete_state_and_checkpoint_recovery ... ignored, installed model and actual pre-fix checkpoint; external memory guard required
+test engine::metal::qwen_mtplx::issue106::issue106_upstream_exit_matrix ... ignored, installed model, scripted logits on real caches; external memory guard required
+test engine::metal::qwen_mtplx::issue108::issue108_attention_chunk_positions ... ignored, Metal and pinned MTPLX chunk fixtures; external supervisor required
+test engine::metal::qwen_mtplx::issue109::issue109_gate_phase_and_dtype_laws ... ignored, Metal and exhaustive pinned gate fixtures; external supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_actual_projection_rows_match_oracle ... ignored, installed layer weights and pinned oracle receipts; external memory supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_guard_allocation_refusal_releases_graph_and_retries ... ignored, Metal allocation refusal and retry; external supervisor required
+test engine::metal::qwen_mtplx::issue110::issue110_large_cold_and_warm_prefill ... ignored, full installed model and large partial chunks; serial memory supervisor required
+test engine::metal::qwen_mtplx::issue112_bench::issue112_hidden_prefill ... ignored, full installed model hidden-state prefill; serial supervisor required
+test engine::metal::qwen_mtplx::issue112_bench::issue112_paired_prefill ... ignored, full installed model; serial memory supervisor and explicit output directory
+test engine::metal::qwen_mtplx::linear::mtplx_graph_affine_dequantize_matches_reference ... ignored, requires Apple Metal; original table dequantize dtype/group/bit/layout receipts
+test engine::metal::qwen_mtplx::linear::mtplx_graph_affine_quantize_matches_reference ... ignored, requires Apple Metal; original affine quantize dtype/group/bit/layout receipts
+test engine::metal::qwen_mtplx::linear::mtplx_graph_gather_qmm_matches_reference ... ignored, requires Apple Metal; original GatherQMM array layouts and expert routing
+test engine::metal::qwen_mtplx::linear::mtplx_graph_quantized_linear_batched_identity ... ignored, requires Apple Metal and external supervisor; strided/broadcast batched affine dispatch
+test engine::metal::qwen_mtplx::linear::mtplx_graph_quantized_linear_matches_reference ... ignored, requires Apple M5 Max Metal; 78 pinned affine fixtures through lazy graph
+test engine::metal::qwen_mtplx::load::mtplx_safetensors_load_laziness_boundaries_and_lifetime ... ignored, requires Apple Metal; bounded CPU Load and dependent GPU graph
+test engine::metal::qwen_mtplx::mlp::mtplx_graph_shared_mlp_matches_reference ... ignored, requires Apple M5 Max; actual MTPLX shared MLP reference receipts
+test engine::metal::qwen_mtplx::mlp::mtplx_graph_split_and_compiled_silu_layouts ... ignored, requires Apple Metal; Split sibling ownership and compiled SiLU layouts
+test engine::metal::qwen_mtplx::moe::mtplx_graph_sparse_moe_matches_reference ... ignored, requires Apple M5 Max; full MoE Array graph, no model weights; use test-supervisor
+test engine::metal::qwen_mtplx::moe_sorted::issue112_all_kernel_variants ... ignored, Metal synthetic all supported row-map variants; external supervisor
+test engine::metal::qwen_mtplx::moe_sorted::issue112_canary_failures ... ignored, Metal canary rejection and sticky fallback; external supervisor
+test engine::metal::qwen_mtplx::linear::sorted_gather_guard_matches_pinned_call_predicates ... ok
+test engine::metal::qwen_mtplx::moe_sorted::issue112_full_model_state ... ignored, full installed model; validates all cold/warm state receipts against the pre-port oracle
+test engine::metal::qwen_mtplx::model::mtplx_model_compile_policy_and_run_boundaries ... ok
+test engine::metal::qwen_mtplx::moe_sorted::issue112_installed_gate_up ... ignored, installed projection weights; external supervisor required
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_embedding_matches_reference ... ignored, requires Apple Metal; full resident NGram graph and lazy decode history
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_gpu_rows_and_history_match_reference ... ignored, requires Apple Metal; original lazy NGram GPU row graph, no table
+test engine::metal::qwen_mtplx::moe_sorted::row_map_applicability ... ok
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_resident_matches_reference ... ignored, requires Apple Metal; original NGramTable resident gather/dequantize receipts
+test database::tests::migrations_and_crud_work ... ok
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_table_priority ... ignored, requires Apple Metal; NGram table routing and missing-sidecar behavior
+test engine::metal::qwen_mtplx::ngram_sidecar::mtplx_ngram_sidecar_matches_reference ... ignored, requires Apple Metal; actual file-backed MTPLX sidecar/LRU/dequantization receipts
+test engine::metal::qwen_mtplx::ngram_stage::mtplx_ngram_stage_allclose_branches ... ignored, requires Apple Metal; stage verification special values and original reduction branches
+test engine::metal::qwen_mtplx::ngram_stage::mtplx_ngram_stage_matches_reference ... ignored, requires Apple Metal; original complete stage lifecycle and file-backed tables
+test engine::metal::qwen_mtplx::normalization::mtplx_graph_norm_modules_match_reference ... ignored, requires Apple Metal; connected Qwen normalization graph golden fixtures
+test engine::metal::qwen_mtplx::normalization::mtplx_graph_rms_norm_preserves_reference_layout_and_values ... ignored, requires Apple Metal; graph RMSNorm donation, copy, broadcast and looped kernels
+test engine::metal::qwen_mtplx::normalization::mtplx_logsumexp_graph_matches_reference ... ignored, requires Apple Metal; original sampler LogSumExp branches
+test engine::metal::qwen_mtplx::normalization::mtplx_softmax_graph_storage_matches_reference ... ignored, requires Apple Metal; original Softmax type and storage branches
+test engine::metal::qwen_mtplx::ops::mtplx_concatenate_preserves_general_general_copy_semantics ... ignored, requires Apple Metal; Concatenate promotion, offsets, strided writers and validation
+test engine::metal::qwen_mtplx::ops::mtplx_owned_operations_build_and_execute_qsa_layout_graph ... ignored, requires Apple Metal; self-contained lazy QSA-layout recipe, no callback map
+test engine::metal::qwen_mtplx::ops::mtplx_slice_and_contiguous_preserve_cache_window_semantics ... ignored, requires Apple Metal; normalized Slice and Contiguous cache-window graph
+test engine::metal::qwen_mtplx::owner::issue105_product::issue105_product_ar_cleanup_and_http_retry ... ignored, installed Qwen model; serial test-supervisor with memory guard required
+test engine::metal::qwen_mtplx::owner::issue105_product::issue105_product_mtp_cleanup_and_http_retry ... ignored, installed Qwen MTP model; serial test-supervisor with memory guard required
+test engine::metal::qwen_mtplx::owner::issue106_product::issue106_product_discards_old_checkpoint_and_recovers_cold ... ignored, installed model and actual pre-fix checkpoint; external memory guard required
+test engine::metal::qwen_mtplx::owner::issue107_product::issue107_vision_product ... ignored, installed model plus vision tower; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::issue109_product::issue109_verify_rows_and_warm_checkpoint ... ignored, installed model; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::issue109_product::issue113_default_verify_and_checkpoint ... ignored, installed model and actual pre-head-chunk checkpoint; serial external memory supervisor
+test engine::metal::qwen_mtplx::owner::issue111_product::issue111_token_prefix_cache ... ignored, installed model; serial external memory supervisor required
+test engine::metal::qwen_mtplx::owner::qwen_product_generator_chat_matches_reference ... ignored, full installed model through product Generator; external supervisor and chat reference required
+test engine::metal::qwen_mtplx::owner::qwen_vision_chat_checkpoint_preserves_image_identity ... ignored, requires installed Qwen, Metal, and DS4_QWEN_VISION_IMAGE; run under test-supervisor
+test engine::metal::qwen_mtplx::owner::qwen_mtp_progress_publishes_without_turn_completion_or_double_counting ... ok
+test engine::metal::qwen_mtplx::ple::mtplx_graph_ple_connected_matches_reference ... ignored, requires Apple M5 Max; complete original PLELayer with real lookup
+test engine::metal::qwen_mtplx::ple::mtplx_graph_ple_projection_and_cache_match_reference ... ignored, requires Apple M5 Max; actual MTPLX PLELayer after embedding lookup
+test engine::metal::qwen_mtplx::policy::qwen_server_policy_matches_original_functions ... ignored, requires DS4_QWEN38_POLICY_REFERENCE from pinned MTPLX; CPU-only
+test engine::metal::qwen_mtplx::prefill::mtplx_mtp_prefill_driver_matches_reference ... ignored, requires Apple Metal; original streaming MTP history prefill
+test engine::metal::qwen_mtplx::prefill::mtplx_prefill_driver_matches_reference ... ignored, requires Apple Metal; original _prefill control flow and raw cache roots
+test engine::metal::qwen_mtplx::policy::sorted_moe_switch_and_family_rehearsal ... ok
+test engine::metal::qwen_mtplx::qsa_attention::mtplx_qsa_attention_module_matches_reference ... ignored, requires Metal and unchanged whole-Attention receipts; no model download
+test engine::metal::qwen_mtplx::qsa_compile::mtplx_compiled_indexer_array_replay_matches_reference ... ignored, requires Metal and original compiled indexer receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_core::mtplx_qsa_core_host_matches_reference ... ignored, requires Metal and original host-core receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_cache_donation_requires_descriptor_data_and_allocation_contract ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_completion_owns_buffers_until_gpu_finishes ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_graph::qsa_evaluator_retains_inputs_and_siblings_not_primary_data ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::qsa_indexer::issue107::issue107_real_geometry_query_and_block_start_positions ... ignored, Metal and independently verified pinned indexer fixtures; external supervisor required
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_canonical_indexer_routes_match_reference ... ignored, requires Metal and original ongoing indexer routing receipts; no weights
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_indexer_compiled_commit_matches_reference ... ignored, requires Metal and original indexer lifecycle receipts; no model weights
+test engine::metal::qwen_mtplx::qsa_indexer::mtplx_qsa_indexer_prefix_matches_reference ... ignored, requires Apple Metal, original stateful QSAIndexer receipts, no model weights
+test engine::metal::qwen_mtplx::qsa_prefill::mtplx_qsa_prefill_graph_matches_reference ... ignored, requires Metal and original two-stage prefill receipts, no weights
+test engine::metal::qwen_mtplx::qsa_prepare::mtplx_qsa_prepare_graph_matches_reference ... ignored, requires Apple Metal; parameterized original QSA preparation and traced positions
+test engine::metal::qwen_mtplx::qsa_select::mtplx_qsa_fused_graph_matches_reference ... ignored, requires Apple Metal and original QSA selector receipts; no model weights
+test engine::metal::qwen_mtplx::random::mtplx_sampler_prefix_matches_reference ... ignored, requires Apple Metal; original target-prefix sampler and default RNG
+test engine::metal::qwen_mtplx::random::mtplx_sampler_resident_matches_reference ... ignored, requires Apple Metal; original MTPLX resident sampling and keyed RNG
+test engine::metal::qwen_mtplx::reduce::mtplx_sampler_argmax_matches_reference ... ignored, requires Apple Metal; original MTPLX greedy ArgReduce and storage
+test engine::metal::qwen_mtplx::reduce::mtplx_sum_graph_matches_reference ... ignored, requires Apple Metal; original shared sum dispatch and QSA mean
+test engine::metal::qwen_mtplx::policy::retired_ar_pipeline_flag_cannot_change_server_or_turn_policy ... ok
+test engine::metal::qwen_mtplx::qsa_graph::qsa_schedule_preserves_the_selected_sibling ... ok
+test engine::metal::qwen_mtplx::request::qwen_request_progress_uses_ui_frontiers_and_decode_timer ... ok
+test engine::metal::qwen_mtplx::request::qwen_request_settings_override_backend_policy ... ok
+test engine::metal::qwen_mtplx::residency::mtplx_residency_matches_reference_lifecycle_cases ... ignored, requires Apple Metal 3 / macOS 15, no model weights
+test engine::metal::qwen_mtplx::rope::mtplx_rope_graph_matches_reference ... ignored, requires Apple Metal; actual MTPLX RoPE, static YaRN and multi-axis receipts
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_batched_matches_reference ... ignored, requires Apple Metal; original MTPLX bound and serial batch distributions
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_entry_matches_reference ... ignored, requires Apple Metal; actual MTPLX generator sampling entry
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_fallback_matches_reference ... ignored, requires Apple Metal; original sparse fallback and host dense sampler
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_penalties_match_reference ... ignored, requires Apple Metal; original GPU and CPU raw-logit penalties
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_selection_matches_reference ... ignored, requires Apple Metal; original MTPLX sampler selection, no model weights
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_serial_support_matches_reference ... ignored, requires Apple Metal; actual MTPLX serial candidate route, no weights
+test engine::metal::qwen_mtplx::scheduler::mtplx_cpu_dispatch_counts_tenth_and_completes_after_failed_work ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_cpu_primitive_holds_data_and_temporaries_in_fifo_order ... ignored, requires Apple Metal shared storage, no GPU dispatch or model weights
+test engine::metal::qwen_mtplx::compile_graph::mtplx_compile_simplify_preserves_reference_parent_rules ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_cpu_events_propagate_errors_without_consuming_source ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_streams_preserve_fifo_errors_and_shutdown ... ok
+test engine::metal::qwen_mtplx::sdpa::mtplx_sdpa_vector_matches_reference ... ignored, requires Apple M5 Max; actual pinned fused SDPA kernels
+test engine::metal::qwen_mtplx::session_cache::issue114_snapshot_stage_preserves_zero_bits ... ignored, Metal; exact pinned lazy/deep snapshot zero-sign law
+test engine::metal::qwen_mtplx::session_cache::mtplx_full_installed_chat_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_CHAT_REFERENCE required
+test engine::metal::qwen_mtplx::session_cache::mtplx_session_boundary_policy ... ok
+test engine::metal::qwen_mtplx::session_cache::mtplx_session_final_state_requires_capture_and_safe_cache ... ok
+test engine::metal::qwen_mtplx::session_checkpoint::qwen_checkpoint_roundtrip_and_failed_restore_preserve_live_state ... ignored, model-free Metal checkpoint; external supervisor and DS4_QWEN38_CHECKPOINT_DIR required
+test engine::metal::qwen_mtplx::session_codec::mtplx_session_codec_host_import_matches_reference ... ignored, model-free Metal imports; external supervisor and DS4_HOST_IMPORT_REFERENCE required
+test engine::metal::qwen_mtplx::session_codec::mtplx_session_codec_matches_original_blobs_and_restore ... ignored, tiny GPU arrays; external supervisor and DS4_SESSION_CODEC_REFERENCE required
+test engine::metal::qwen_mtplx::sort::mtplx_sort_graph_matches_reference ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::sparse_attention::mtplx_sparse_attention_graph_matches_reference ... ignored, requires Apple M5; actual MTPLX sparse Decode and TensorOps Prefill consumers
+test engine::metal::qwen_mtplx::speculative::mtplx_full_installed_mtp_generation_matches_reference ... ignored, full installed model; external guard and DS4_QWEN38_MTP_GENERATION_REFERENCE required
+test engine::metal::profile::tests::profile_reports_locality_and_lru_hits ... ok
+test engine::metal::qwen_mtplx::stream::mtplx_stream_routes_queues_events_and_finalizes_all_under_pressure ... ignored, requires Apple Metal, tiny model-free two-queue copy
+test engine::metal::qwen_mtplx::stream::mtplx_stream_registry_preserves_thread_defaults_templates_and_clear ... ok
+test engine::metal::qwen_mtplx::submission::mtplx_submission_attaches_at_flush_eval_finish_and_drop ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx::unary::mtplx_graph_unary_preserves_reference_layout_and_values ... ignored, requires Apple Metal; graph unary layout, donation and kernel boundaries
+test engine::metal::qwen_mtplx::verify_sdpa::issue113_head_chunk_outputs_and_dispatch ... ignored, Metal, pinned head-chunk numerical and dispatch fixtures; serial supervisor
+test database::tests::chat_and_compaction_history_survive_reopen_and_session_deletion ... ok
+test engine::metal::qwen_mtplx::views::mtplx_dtype_conversion_and_views_preserve_values_layout_and_donation ... ignored, requires Apple Metal; original conversion/view kernels and donation in lazy eval
+test engine::metal::qwen_mtplx::views::mtplx_dynamic_slice_graph_matches_reference ... ignored, requires Metal and original dynamic slice receipts; no model weights
+test engine::metal::qwen_mtplx::views::mtplx_reshape_shares_or_copies_in_the_connected_evaluator ... ignored, requires Apple Metal; original Reshape copy kernels and shared backing
+test engine::metal::qwen_mtplx::views::mtplx_views_preserve_aliases_offsets_strides_and_gpu_output ... ignored, requires Apple Metal; connected zero-copy views and offset-aware original copy kernel
+test engine::metal::qwen_mtplx::vision::qwen_vision_tower_matches_mtplx_image ... ignored, requires Metal and pinned Qwen vision artifacts/reference values
+test engine::metal::qwen_mtplx::weights::mtplx_checkpoint_projection_fusions_match_reference ... ignored, requires Apple Metal; bounded lazy checkpoint Load and fusion
+test engine::metal::qwen_mtplx::weights::mtplx_installed_checkpoint_fusion_metadata_matches_reference ... ignored, requires installed DS4_QWEN38_ARTIFACTS; headers only, never evaluate weights
+test engine::metal::qwen_mtplx_tests::mtplx_affine_selection_preserves_reference_boundaries ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_canonical_hyper_and_gdn_kernels_match_reference ... ignored, requires Apple Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_canonical_source_bodies_preserve_pinned_hashes ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_connected_delta_update_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gather_sort_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_decode_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_forward_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_gdn_staged_matches_model_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_norm_modules_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_router_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_sparse_moe_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_connected_switch_glu_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_gate_up_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_cache_lifecycle_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_backings_and_dynamic_copies_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_cache_routing_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_cache_window_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_capacity_matches_reference ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_indexer_modes_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx::verify_sdpa::issue113_plan_and_policy_match_pinned_oracle ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_connected_cache_selection_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_eager_topk_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_f32_scores_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_indexer_preparation_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_indexer_routes_match_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_producer_dispatch_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_compiled_scalar_sources_match_reference ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_selector_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_affine_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_argsort_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_gather_matches_reference ... ignored, requires Apple M5 Max Metal, no model weights
+test dev_brain::tests::parser_and_index_rebuild_ignore_links_in_code ... ok
+test engine::metal::tests::batch_raw_span_includes_the_chunk_and_previous_window ... ok
+test engine::metal::tests::compression_schedule_tracks_the_deepseek_model_shape ... ok
+test engine::metal::tests::decode_feature_policy_keeps_device_generations_and_rollbacks_separate ... ok
+test engine::metal::tests::decode_raw_span_tracks_the_logical_sliding_window ... ok
+test engine::metal::tests::directional_steering_matches_the_ds4_token_oracle ... ignored, requires the installed Flash GGUF, DS4 steering fixture, and a Metal device
+test engine::metal::tests::dspark_confidence_quantizes_q8_input_like_ds4 ... ok
+test engine::metal::tests::dspark_markov_activation_matches_q8_rounding ... ok
+test engine::metal::tests::dspark_matches_original_summary_cycles ... ignored, requires installed 0731/DSpark and DS4SERVER_DSPARK_REFERENCE cycle recording
+test engine::metal::tests::dspark_runs_a_target_owned_greedy_cycle ... ignored, requires the installed 0731 Flash and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::dspark_scheduler_matches_the_ds4_default_window ... ok
+test engine::metal::tests::dspark_verifier_selects_only_the_committed_row ... ok
+test engine::metal::tests::flash_0731_long_context_crosses_indexed_prefill_boundary ... ignored, requires a 0731 Flash GGUF and an Apple M5 device
+test engine::metal::tests::flash_0731_m5_decode_performance_gate ... ignored, requires a 0731 Flash GGUF and an Apple M5 device
+test engine::metal::tests::flash_0731_resident_and_ssd_streaming_choose_the_same_tokens ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::flash_0731_runs_exact_sampled_dspark ... ignored, requires the installed 0731 target and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::flash_0731_ssd_streaming_maps_and_seeds_batched_prefill_layers ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::generated_hotlists_match_each_model_shape ... ok
+test engine::metal::tests::half_conversion_handles_normal_and_subnormal_values ... ok
+test engine::metal::tests::pro_q4_model_spans_remain_isolated ... ok
+test engine::metal::tests::pro_resident_admission_includes_weights_context_and_scratch ... ok
+test engine::metal::tests::pro_ssd_streaming_full_layer_and_selected_batch_agree ... ignored, requires the installed DeepSeek V4 Pro GGUF and Apple Metal
+test engine::metal::tests::resident_multi_session_switching_preserves_each_kv_frontier ... ignored, requires the installed 81 GiB Flash GGUF fixture and a Metal device
+test engine::metal::tests::ssd_streaming_supports_dspark ... ignored, requires the installed 0731 Flash and checkpoint-specific DSpark GGUF fixtures
+test engine::metal::tests::streaming_memory_plan_matches_ds4_graph_formulas ... ok
+test engine::metal::vision::tests::glm53_resize_matches_reference_token_grids ... ok
+test engine::qwen::tests::frozen_inventory_and_memory_categories_are_exact ... ok
+test engine::qwen::tests::metadata_and_safetensors_boundaries_fail_closed ... ok
+test engine::qwen::tests::pinned_artifact_set_loads_and_renders_goldens ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test engine::qwen::tests::pinned_qwen_metadata_matches_product_without_executor_state ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test engine::qwen_vision::qwen_image_cache_keys_distinguish_pixels_and_preserve_prior_text ... ok
+test engine::metal::qwen_mtplx::submission::mtplx_all_native_command_commits_pass_the_submission_hook ... ok
+test engine::qwen_vision::qwen_vision_positions_and_resize_contract ... ok
+test engine::sampling_tests::checkpoint_headers_identify_the_exact_model ... ok
+test engine::sampling_tests::checkpoint_rebuilds_explain_compatibility_and_history_misses ... ok
+test engine::sampling_tests::checkpoint_tag_accepts_an_unchanged_prefix_before_reminders ... ok
+test engine::sampling_tests::checkpoint_tag_covers_the_canonical_chat_state ... ok
+test engine::metal::qwen_mtplx::scheduler::mtplx_scheduler_activity_preserves_reference_pressure_and_wait_rules ... ok
+test engine::sampling_tests::ds4_sampler_real_logits_cpu_benchmark ... ignored, CPU-only; requires DS4_SAMPLER_RECORDING with the recorded GLM logits
+test engine::sampling_tests::exact_delta_sampling_accepts_or_corrects_the_draft ... ok
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_distribution_and_rng_match_reference ... ok
+test engine::sampling_tests::generated_text_drains_incomplete_bytes_before_phase_changes_and_finish ... ok
+test engine::sampling_tests::metal_executes_real_flash_token ... ignored, requires the 80 GiB Flash checkpoint and Apple Metal
+test engine::sampling_tests::generated_text_preserves_thinking_utf8_and_split_stops ... ok
+test engine::sampling_tests::qwen_rng_matches_numpy_pcg64 ... ok
+test engine::sampling_tests::qwen_sampling_uses_mtplx_global_nucleus_mass_before_top_k ... ok
+test engine::sampling_tests::resumed_prefill_crossover_matches_each_ds4_graph ... ok
+test engine::sampling_tests::sampling_probabilities_match_ds4_filter_order ... ok
+test engine::sampling_tests::split_utf8_token_bytes_are_joined_before_decoding ... ok
+test engine::sampling_tests::top_k_and_stream_stops_are_applied_before_output ... ok
+test engine::sampling_tests::workspace_instruction_identity_participates_in_append_only_kv_keys ... ok
+test engine::sampling_tests::zero_temperature_is_greedy ... ok
+test engine::tokenizer::issue111::issue111_ds4_families ... ignored, CPU-only; installed DS4 GGUFs and exact oracle IDs
+test engine::tokenizer::issue111::issue111_every_unicode_scalar_split ... ignored, CPU-only; exhaustive pinned Qwen split receipt
+test engine::tokenizer::issue111::issue111_qwen_tokens_and_requests ... ignored, CPU-only; installed Qwen tokenizer and pinned exact IDs
+test engine::tokenizer::issue111::qwen_split_boundaries ... ok
+test engine::tokenizer::tests::character_helpers_are_total_at_every_byte_offset ... ok
+test engine::tokenizer::tests::ds4_chat_matches_original_session_tokens ... ignored, CPU-only; requires DS4SERVER_CHAT_REFERENCE and its installed GGUF
+test engine::tokenizer::tests::qwen_checkpoint_continuation_matches_full_conversation ... ignored, requires the installed Qwen tokenizer, but no GPU or model weights
+test engine::tokenizer::tests::qwen_plain_chat_matches_mtplx_server_tokens ... ignored, requires local tokenizer and tools/qwen-chat-reference.py output; no inference
+test engine::tokenizer::tests::reasoning_prefixes_match_each_model_family_contract ... ok
+test engine::validation::tests::configured_glm53_main_fixture_passes_the_exact_layout ... ok
+test engine::validation::tests::configured_glm53_vision_fixture_passes_the_exact_layout ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_qsa_prefill_workspace_matches_reference ... ok
+test engine::validation::tests::installed_dspark_fixture_passes_the_target_layout ... ok
+test engine::validation::tests::mxfp4_is_limited_to_routed_experts ... ok
+test engine::metal::qwen_mtplx::ngram::mtplx_ngram_cpu_rows_and_raw_history_match_reference ... ok
+test extensions::tests::duplicate_skill_names_and_update_rollback_are_deterministic ... ok
+test dev_brain::tests::skill_validation_and_default_guide_restore_are_strict ... ok
+test extensions::tests::hook_output_is_bounded_typed_and_event_matched ... ok
+test engine::sampling_tests::probability_ratio_speculation_preserves_the_target_distribution ... ok
+test dev_brain::tests::source_hash_drift_excludes_page_from_authoritative_queries ... ok
+test extensions::tests::pinned_ponytail_runs_all_lifecycle_modes ... ignored, downloads the pinned Ponytail reference integration
+test extensions::tests::manifest_and_paths_are_validated_without_following_escapes ... ok
+test extensions::tests::symlink_and_variable_escapes_are_rejected ... ok
+test extensions::tests::registry_persists_enable_disable_errors_and_uninstall ... ok
+test instructions::tests::changes_removals_resume_and_compaction_are_append_only ... ok
+test instructions::tests::empty_and_repaired_baselines_do_not_repeat_prompt_content ... ok
+test instructions::tests::discovery_is_global_to_local_deduplicated_and_confined ... ok
+test instructions::tests::newly_crossed_scope_is_added_once_and_unavailable_content_is_retained ... ok
+test metrics::tests::cache_report_splits_buckets_and_folds_index_files_into_their_checkpoint ... ok
+test metrics::tests::decode_timer_excludes_later_work_and_resets_between_requests ... ok
+test metrics::tests::prompt_timing_keeps_unmeasured_components_distinct_from_zero ... ok
+test metrics::tests::removed_preview_metric_code_stays_decodable ... ok
+test metrics::tests::snapshots_track_runtime_and_server_counters ... ok
+test model::qwen::qwen_vision_is_an_independent_managed_artifact_set ... ok
+test model::qwen::tests::embedded_manifest_is_the_frozen_complete_set ... ok
+test model::qwen::tests::verification_rejects_hash_mismatch_and_honors_cancellation ... ok
+test model::transfer::tests::cancellation_keeps_the_partial_file_for_the_next_run ... ok
+test engine::metal::qwen_mtplx::array::mtplx_array_releases_sibling_cycles_and_deep_input_graphs ... ok
+test model::transfer::tests::catalog_and_checksum_verification_are_explicit ... ok
+test model::transfer::tests::qwen_set_verifies_reloads_and_removes_as_one_managed_artifact ... ignored, requires DS4_QWEN38_ARTIFACTS to point at the pinned 105 GB source
+test model::transfer::tests::managed_artifact_inventory_and_delete_include_partial_files ... ok
+test instructions::tests::frames_escape_control_tags_and_enforce_utf8_byte_limits ... ok
+test model::transfer::tests::verification_reports_bytes_read ... ok
+test model_eval::tests::acceleration_receipt_reports_effective_sampling_contract ... ok
+test model_eval::tests::acceleration_switch_uses_each_models_native_path ... ok
+test model::transfer::tests::restart_resumes_at_the_existing_partial_byte ... ok
+test model_eval::tests::canary_cross_phase_samples_keep_their_start_phase ... ok
+test model_eval::tests::evaluations_exclude_overlapping_workers_and_release_on_exit ... ok
+test model_eval::tests::external_canary_observes_phases_and_finishes_on_eof ... ignored, Metal probe only; DS4SERVER_CANARY_BINARY must point to the current release binary
+test model_eval::tests::external_canary_protocol_accepts_only_known_phases ... ok
+test model_eval::tests::parses_safe_defaults_and_overrides ... ok
+test model_eval::tests::rejects_unbounded_or_invalid_limits ... ok
+test model_eval::tests::repeated_prompts_form_one_ordered_conversation ... ok
+test model_eval::tests::warmup_timing_separates_loading_without_inventing_missing_durations ... ok
+test model_eval::tests::worker_progress_events_advance_the_watchdog ... ok
+test native_edit::tests::edit_history_supports_undo_and_redo ... ok
+test native_edit::tests::paste_replays_a_complete_command_shortcut ... ok
+test native_media::tests::media_player_only_accepts_remote_http_urls ... ok
+test native_menu::tests::edit_items_are_custom_menu_events ... ok
+test dev_brain::tests::validation_names_every_drifted_source_and_requires_semantic_refresh ... ok
+test native_menu::tests::user_guide_uses_unmodified_f1 ... ok
+test native_menu::tests::view_items_keep_the_selected_pane_checked ... ok
+test runtime::tests::dropping_active_generation_cancels_it ... ok
+test runtime::tests::operation_failures_use_the_matching_event ... ok
+test runtime::tests::operation_tracking_is_consistent ... ok
+test runtime::tests::runtime_panic_recovery_accepts_the_next_operation ... ok
+test server::request::tests::reasoning_requests_preserve_each_models_exact_modes ... ok
+test server::tests::canonical_tool_replay_preserves_argument_types ... ok
+test server::tests::connection_slots_enforce_the_limit_and_release_on_drop ... ok
+test server::tests::content_arrays_match_ds4_text_projection ... ok
+test server::tests::cors_headers_are_opt_in_and_cover_preflight ... ok
+test server::tests::dsml_tool_stream_projects_incremental_json_arguments ... ok
+test server::tests::context_limit_errors_use_protocol_standard_fields ... ok
+test server::tests::generated_dsml_becomes_openai_tool_calls ... ok
+test server::tests::model_metadata_uses_the_requested_alias_and_default_token_limit ... ok
+test server::tests::generated_tool_parser_accepts_ds4_recovery_syntaxes ... ok
+test server::tests::protocol_tool_results_require_live_or_replayed_call_ids ... ok
+test server::tests::responses_parser_rejects_dropped_state_and_preserves_hosted_calls ... ok
+test server::tests::sampled_tool_replay_survives_server_restart ... ok
+test server::tests::streaming_errors_match_ds4_server_framing ... ok
+test dev_brain::tests::validation_keeps_broken_links_editable_and_reports_provenance_errors ... ok
+test server::tests::tool_prompt_matches_ds4_server_text ... ok
+test server::tests::tool_stream_projects_ds4_recovery_syntaxes ... ok
+test settings::tests::captured_ds4_cli_options_are_uniquely_classified ... ok
+test settings::tests::effective_settings_build_one_engine_and_turn_configuration ... ok
+test settings::tests::effective_settings_preserve_unset_model_defaults ... ok
+test settings::tests::execution_settings_keep_ds4_automatic_values_and_glm_rules ... ok
+test settings::tests::kv_cache_gates_follow_the_ds4_store_rules ... ok
+test settings::tests::reasoning_selection_is_exact_and_model_specific ... ok
+test settings::tests::runtime_settings_preserve_automatic_values_and_cross_field_rules ... ok
+test settings::tests::speculative_settings_match_acceleration_defaults_and_dependencies ... ok
+test dev_brain::tests::verified_skills_are_indexed_and_injected_on_demand ... ok
+test server::tests::streaming_prefill_keeps_the_client_connection_alive ... ok
+test dev_brain::tests::source_versions_follow_that_file_instead_of_the_whole_worktree ... ok
+test model_eval::tests::cpu_accounting_matches_getrusage_units ... ok
+test engine::validation::tests::installed_ds4_fixture_opens_and_renders_a_prompt ... ok
+test extensions::tests::direct_hook_runner_isolated_environment_and_handles_failures ... ok
+test model::transfer::tests::stalled_download_times_out_and_keeps_the_partial_file ... ok
+test server::tests::request_deadline_stops_slow_clients ... ok
+test engine::sampling_tests::ds4_sampler_matches_original_reference_tokens_and_rng ... ok
+test extensions::tests::lifecycle_matchers_preserve_declared_event_order ... ok
+test engine::qwen_vision::qwen_image_preprocessing_matches_pillow_mtplx ... ok
+test agent::tests::execution_failures_and_timeouts_have_distinct_codes ... ok
+kill: 97686: No such process
+test agent::tests::dropping_session_tools_stops_live_jobs_and_removes_output ... ok
+test agent::tests::local_file_and_bash_tools_execute_in_the_project ... ok
+test engine::metal::qwen_mtplx_tests::mtplx_runtime_shaders_are_pinned ... ok
+test extensions::tests::hook_failures_are_bounded_actionable_and_non_shell ... ok
+test engine::metal::qwen_mtplx::sampling::mtplx_sampler_mixed_distributions_match_reference ... ok
+
+test result: ok. 313 passed; 0 failed; 237 ignored; 0 measured; 0 filtered out; finished in 5.40s
+
+     Running unittests tools/qwen38-artifacts.rs (target/debug/deps/qwen38_artifacts-1288d410762a1a77)
+
+running 4 tests
+test tests::artifact_names_cannot_escape_the_output_directory ... ok
+test tests::affine_sample_digest_uses_packed_values_scales_and_biases ... ok
+test tests::safetensors_validation_rejects_duplicate_names_and_invalid_ranges ... ok
+test tests::artifact_download_resumes_an_authenticated_partial ... ok
+
+test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
+
+     Running unittests tools/test-supervisor.rs (target/debug/deps/test_supervisor-df80f419c8443a12)
+
+running 4 tests
+test supervisor::tests::direct_worker_arguments_are_not_rust_test_arguments ... ok
+test supervisor::tests::split_records_cannot_be_interleaved_with_watchdog_samples ... ok
+test supervisor::tests::split_probe_records_do_not_reset_progress ... ok
+ready{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+ready
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+{"event":"gpu_canary_sample","ok":true}
+progressprogressprogressprogressprogressprogressprogressprogresstest supervisor::tests::watchdog_stops_memory_and_silence_but_not_total_runtime ... ok
+
+test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.35s

src/engine/metal/qwen_mtplx/session_cache.rs

@@ -1352,3 +1352,93 @@ struct SessionPrompt {
     history_seconds: f64,
     restore_seconds: f64,
 }
+
+#[test]
+#[ignore = "Metal; exact pinned lazy/deep snapshot zero-sign law"]
+fn issue114_snapshot_stage_preserves_zero_bits() {
+    use super::super::{gpu::Context, qwen_mtplx_tests::hex_digest};
+    use super::array::Dtype;
+    use super::cache::QsaCache;
+    super::configure_sources().unwrap();
+    let _context = Context::open_qwen(0).unwrap();
+    let streams = super::stream::Streams::new(Some(super::allocator::Allocator::new().unwrap()));
+    let gpu = streams.default_stream(super::stream::Device::Gpu).unwrap();
+    let fixture: serde_json::Value = serde_json::from_str(include_str!(
+        "../../../../tests/fixtures/mtplx-snapshot-zero.json"
+    ))
+    .unwrap();
+    let read = |a: &Array| {
+        ops::evaluate(&streams, std::slice::from_ref(a), gpu, false).unwrap();
+        let mut bytes = vec![0; a.layout().nbytes()];
+        a.buffer().read(a.offset(), &mut bytes).unwrap();
+        bytes
+    };
+    for case in fixture["cases"].as_array().unwrap() {
+        let decode = |key| {
+            case[key]
+                .as_str()
+                .unwrap()
+                .as_bytes()
+                .chunks_exact(2)
+                .map(|pair| u8::from_str_radix(std::str::from_utf8(pair).unwrap(), 16).unwrap())
+                .collect::<Vec<_>>()
+        };
+        let dtype = if case["dtype"].as_str().unwrap().contains("bfloat16") {
+            Dtype::BF16
+        } else {
+            Dtype::F32
+        };
+        let raw = decode("raw");
+        let a = Array::new(&[1, 2, 1, 4], dtype, super::scalar_buffer(&raw).unwrap()).unwrap();
+        let make = || {
+            let mut qsa = QsaCache::new(4);
+            qsa.set_state(&[Some(object(a.clone())), Some(object(a.clone())), None, None])
+                .unwrap();
+            vec![Some(Cache {
+                attention: AttentionCache::Qsa(qsa),
+                ple: Default::default(),
+                capture_ple: None,
+            })]
+        };
+        for lazy in [false, true] {
+            let saved = snapshot(&make(), |_| false, lazy, false, gpu).unwrap();
+            let Some(State::Qsa(leaves)) = &saved[0] else {
+                panic!("QSA state missing")
+            };
+            let expected = decode(if lazy { "lazy" } else { "deep" });
+            assert_eq!(read(leaves[0].as_ref().unwrap()), expected);
+            assert_eq!(
+                read(&a),
+                raw,
+                "taking either snapshot must not mutate its source"
+            );
+            for clone in [false, true] {
+                let mut restored = make();
+                restore(&mut restored, &saved, clone, gpu).unwrap();
+                let AttentionCache::Qsa(qsa) = &restored[0].as_ref().unwrap().attention else {
+                    unreachable!()
+                };
+                assert_eq!(qsa.kv.offset, 1);
+                let values = qsa.state(gpu).unwrap();
+                let got = read(&values[0].as_ref().unwrap().borrow());
+                assert_eq!(
+                    got,
+                    if clone {
+                        decode("deep")
+                    } else {
+                        expected.clone()
+                    }
+                );
+                assert_eq!(
+                    read(leaves[0].as_ref().unwrap()),
+                    expected,
+                    "restore must not mutate banked state"
+                );
+                println!(
+                    "{}",
+                    serde_json::json!({"case":"snapshot_zero","dtype":case["dtype"],"lazy":lazy,"clone_on_restore":clone,"sha256":hex_digest(&got),"exact":true})
+                );
+            }
+        }
+    }
+}

tests/fixtures/mtplx-snapshot-zero.json

@@ -0,0 +1,95 @@
+{
+  "revision": "9882703f3105363ddc37eca9f97aa09a1d387112",
+  "mlx": "0.32.2",
+  "cases": [
+    {
+      "dtype": "mlx.core.bfloat16",
+      "shape": [
+        1,
+        2,
+        1,
+        4
+      ],
+      "raw": "0080000080bf803f00000080004000c0",
+      "lazy": "0080000080bf803f00000080004000c0",
+      "deep": "0000000080bf803f00000000004000c0"
+    },
+    {
+      "dtype": "mlx.core.float32",
+      "shape": [
+        1,
+        2,
+        1,
+        4
+      ],
+      "raw": "0000008000000000000080bf0000803f000000000000008000000040000000c0",
+      "lazy": "0000008000000000000080bf0000803f000000000000008000000040000000c0",
+      "deep": "0000000000000000000080bf0000803f000000000000000000000040000000c0"
+    }
+  ],
+  "real_key_traces": [
+    {
+      "phase": "cold-prefill",
+      "shape": [
+        1,
+        2,
+        46,
+        256
+      ],
+      "coordinates": [
+        [
+          0,
+          1,
+          0,
+          2
+        ]
+      ],
+      "raw_sha256": "d81b466dea1ff5b774e86c0af47069f555bbb59abb0f3f61165f5bc51aa28ac6",
+      "lazy_sha256": "d81b466dea1ff5b774e86c0af47069f555bbb59abb0f3f61165f5bc51aa28ac6",
+      "deep_sha256": "520dd44032eda4789b123296b5499afb88f0d88d6cdccffa04ebe06c2bc3a853",
+      "original_oracle_sha256": "520dd44032eda4789b123296b5499afb88f0d88d6cdccffa04ebe06c2bc3a853"
+    },
+    {
+      "phase": "cold",
+      "shape": [
+        1,
+        2,
+        58,
+        256
+      ],
+      "coordinates": [
+        [
+          0,
+          1,
+          0,
+          2
+        ]
+      ],
+      "raw_sha256": "6a1c56e7333fc86e4e11bb265e99274ff727408f9ab32508f374198d4e8553c7",
+      "lazy_sha256": "6a1c56e7333fc86e4e11bb265e99274ff727408f9ab32508f374198d4e8553c7",
+      "deep_sha256": "44b87b49e9f9853e6a3cb5efcefe1f25ff4f06d56e247847d0054de3aab3fab4",
+      "original_oracle_sha256": "44b87b49e9f9853e6a3cb5efcefe1f25ff4f06d56e247847d0054de3aab3fab4"
+    },
+    {
+      "phase": "warm",
+      "shape": [
+        1,
+        2,
+        88,
+        256
+      ],
+      "coordinates": [
+        [
+          0,
+          1,
+          0,
+          2
+        ]
+      ],
+      "raw_sha256": "061f8fe052fe21c8505782c220e03d09ab31deb93e7987709c55cd5dc3b778bd",
+      "lazy_sha256": "061f8fe052fe21c8505782c220e03d09ab31deb93e7987709c55cd5dc3b778bd",
+      "deep_sha256": "98186c7fec2498072f93b621a7f5b14456d4a67b0548a006353d87b4e4c2e89f",
+      "original_oracle_sha256": "98186c7fec2498072f93b621a7f5b14456d4a67b0548a006353d87b4e4c2e89f"
+    }
+  ]
+}

tools/issue109-model-reference.py

@@ -1,4 +1,8 @@
-"""Compare actual MTP verify/repair/commit rows and warm continuation to native."""
+"""Strict real MTP oracle: compare the same snapshot stage as the native bank.
+
+#114: snapshot_cache on a lazy trunk adds zero, creating a false signed-zero
+mismatch. Preserve lazy trunk bits and deep-clone only the committed MTP head.
+"""
 import argparse,hashlib,json,os,subprocess,sys
 from pathlib import Path
 checkout,root,native,out=map(Path,sys.argv[1:5])
@@ -17,8 +21,8 @@ from mtplx.runtime import _load_tokenizer_resilient,MTPLXRuntime
 from mtplx.mtp_patch import MTPContract
 from mtplx.generation import generate_mtpk
 from mtplx.sampling import SamplerConfig
-from mtplx.session_bank import SessionBank
-from mtplx.cache_state import snapshot_cache
+from mtplx.session_bank import SessionBank,_lazy_snapshot_enabled
+from mtplx.cache_state import snapshot_cache,snapshot_cache_lazy_hybrid
 from mtplx.attention_context import current_attention_phase
 out.mkdir(parents=True,exist_ok=True)
 print(json.dumps(dict(case='environment',overrides={k:v for k,v in os.environ.items() if k.startswith('MTPLX_')})),flush=True)
@@ -39,9 +43,11 @@ def wrap(fn):
         if outer and current_attention_phase() == 'prefill' and isinstance(result,tuple) and result[0] is not None and result[1] is not None:
             cache=kwargs.get('cache',args[0] if args else None)
             if cache is not None:
-                key=np.asarray(snapshot_cache(cache).states[35][0].astype(mx.float32))
+                key=np.asarray((snapshot_cache_lazy_hybrid(cache) if _lazy_snapshot_enabled() else snapshot_cache(cache)).states[35][0].astype(mx.float32))
                 (out/f'{phase}-prefill-key35.f32').write_bytes(key.tobytes())
             for part,a in zip(['logits','hidden'],result[:2]):
+                # The suffix owner retains only the final hidden row (generation.py:4689).
+                if part=='hidden':a=a[:,-1:,:]
                 data=np.asarray(a.astype(mx.float32)).tobytes()
                 (out/f'{phase}-prefill-{part}.f32').write_bytes(data)
                 path=native/f'{phase}-prefill-{part}.f32'
@@ -85,18 +91,16 @@ for phase in ['cold','warm']:
     for part,a in [('logits',final.final_logits),('hidden',final.final_hidden)]:
         data=np.asarray(a.astype(mx.float32)).tobytes()
         assert data==(native/f'{phase}-{part}.f32').read_bytes(),(phase,part)
-    def states(cache):
+    def states(cache,lazy=False):
         def array(a):
             if a is None: return None
             values=np.asarray(a.astype(mx.float32)).copy()
             bits=hashlib.sha256(values.tobytes()).hexdigest()
-            values[values==0]=0
-            return dict(shape=list(a.shape),sha256=bits,value_sha256=hashlib.sha256(values.tobytes()).hexdigest())
-        return [[array(a) for a in state] if state is not None else None for state in snapshot_cache(cache).states]
-    arrays=dict(trunk=states(final.final_trunk_cache),mtp=states(final.final_committed_mtp_cache))
+            return dict(shape=list(a.shape),sha256=bits)
+        return [[array(a) for a in state] if state is not None else None for state in (snapshot_cache_lazy_hybrid(cache) if lazy else snapshot_cache(cache)).states]
+    arrays=dict(trunk=states(final.final_trunk_cache,_lazy_snapshot_enabled()),mtp=states(final.final_committed_mtp_cache))
     (out/f'{phase}-arrays.json').write_text(json.dumps(arrays))
     native_arrays=json.loads((native/f'{phase}-arrays.json').read_text())
-    zero_sign_differences=[]
     for family in arrays:
         assert len(arrays[family])==len(native_arrays[family])
         for layer,(actual,want) in enumerate(zip(arrays[family],native_arrays[family])):
@@ -108,18 +112,17 @@ for phase in ['cold','warm']:
                 if a is None or b is None:
                     assert a==b
                     continue
-                assert a['shape']==b['shape'] and a['value_sha256']==b['value_sha256'],(phase,family,layer,slot,a,b)
-                if a['sha256']!=b['sha256']: zero_sign_differences.append([family,layer,slot])
-    assert zero_sign_differences in ([], [['trunk',35,0]]), zero_sign_differences
-    # Retain raw evidence of the prefill signed-zero mismatch; never hide it in
-    # the bit-exact verify/logit checks or substitute a numerical tolerance.
-    key=np.asarray(snapshot_cache(final.final_trunk_cache).states[35][0].astype(mx.float32))
+                assert a['shape']==b['shape'] and a['sha256']==b['sha256'],(phase,family,layer,slot,a,b)
+    # Compare the same lazy trunk snapshot used by the server/native bank.
+    # An extra deep clone would add zero and change negative-zero bits.
+    key=np.asarray((snapshot_cache_lazy_hybrid(final.final_trunk_cache) if _lazy_snapshot_enabled() else snapshot_cache(final.final_trunk_cache)).states[35][0].astype(mx.float32))
     (out/f'{phase}-key35.f32').write_bytes(key.tobytes())
     want=np.fromfile(native/f'{phase}-key35.f32',np.float32).reshape(key.shape)
     different=key.view(np.uint32)!=want.view(np.uint32)
-    assert np.all(key[different]==0) and np.all(want[different]==0)
-    assert np.argwhere(different).tolist() in ([], [[0,1,0,2]])
-    print(json.dumps(dict(case='cache_arrays',phase=phase,values_exact=True,zero_sign_differences=zero_sign_differences,key35_bit_coordinates=np.argwhere(different).tolist())),flush=True)
+    assert not np.any(different)
+    for part in ["key35","logits","hidden"]:
+        assert (out/f"{phase}-prefill-{part}.f32").read_bytes()==(native/f"{phase}-prefill-{part}.f32").read_bytes(),(phase,part,"prefill differs")
+    print(json.dumps(dict(case='cache_arrays',phase=phase,raw_bits_exact=True,key35_bit_coordinates=np.argwhere(different).tolist())),flush=True)
     head=snapshot_cache(final.final_committed_mtp_cache)
     key=ids+result.tokens
     assert bank.put(runtime=rt,token_ids=key,cache=final.final_trunk_cache,logits=final.final_logits,hidden=final.final_hidden,

tools/issue113-model-reference.py

@@ -1,132 +1,5 @@
-"""Strict default/off head-chunk oracle for real verify rows and matching bank snapshots."""
-import argparse,hashlib,json,os,subprocess,sys
+"""Use the shared strict real-request oracle for the head-chunk on/off controls."""
 from pathlib import Path
-checkout,root,native,out=map(Path,sys.argv[1:5])
-assert subprocess.check_output(['git','-C',str(checkout),'rev-parse','HEAD'],text=True).strip()=='9882703f3105363ddc37eca9f97aa09a1d387112'
-sys.path.insert(0,str(checkout))
-from mtplx.server.openai import _server_runtime_env_overrides
-os.environ.update(_server_runtime_env_overrides(argparse.Namespace(model=str(root),generation_mode='mtp'),None))
-os.environ['MTPLX_SUSTAINED_PREFILL']='1'
-os.environ['MTPLX_PREFILL_CHUNK_SIZE']='2048'
-os.environ['MTPLX_CONTEXT_COPY']='0'
-import mlx.core as mx
-import numpy as np
-from mlx_lm.utils import load_model
-from mtplx.models import qwen4_exp as qwen
-from mtplx.runtime import _load_tokenizer_resilient,MTPLXRuntime
-from mtplx.mtp_patch import MTPContract
-from mtplx.generation import generate_mtpk
-from mtplx.sampling import SamplerConfig
-from mtplx.session_bank import SessionBank,_lazy_snapshot_enabled
-from mtplx.cache_state import snapshot_cache,snapshot_cache_lazy_hybrid
-from mtplx.attention_context import current_attention_phase
-out.mkdir(parents=True,exist_ok=True)
-print(json.dumps(dict(case='environment',overrides={k:v for k,v in os.environ.items() if k.startswith('MTPLX_')})),flush=True)
-tokenizer=_load_tokenizer_resilient(root,json.loads((root/'config.json').read_text()))
-model,_=load_model(root,lazy=False,strict=True,get_model_classes=lambda **_:(qwen.Model,qwen.ModelArgs))
-model.post_weight_load(root)
-assert model.attach_mtp(root)
-rt=MTPLXRuntime(model,tokenizer,root,True,MTPContract())
-phase='cold'; index=0; nested=0
+import runpy
 
-def wrap(fn):
-    def call(input_ids,*args,**kwargs):
-        global index,nested
-        outer=nested==0
-        nested+=1
-        try: result=fn(input_ids,*args,**kwargs)
-        finally: nested-=1
-        if outer and current_attention_phase() == 'prefill' and isinstance(result,tuple) and result[0] is not None and result[1] is not None:
-            cache=kwargs.get('cache',args[0] if args else None)
-            if cache is not None:
-                key=np.asarray((snapshot_cache_lazy_hybrid(cache) if _lazy_snapshot_enabled() else snapshot_cache(cache)).states[35][0].astype(mx.float32))
-                (out/f'{phase}-prefill-key35.f32').write_bytes(key.tobytes())
-            for part,a in zip(['logits','hidden'],result[:2]):
-                # The suffix owner retains only the final hidden row (generation.py:4689).
-                if part=='hidden':a=a[:,-1:,:]
-                data=np.asarray(a.astype(mx.float32)).tobytes()
-                (out/f'{phase}-prefill-{part}.f32').write_bytes(data)
-                path=native/f'{phase}-prefill-{part}.f32'
-                print(json.dumps(dict(case='prefill_compare',phase=phase,part=part,shape=list(a.shape),exact=path.exists() and path.read_bytes()==data)),flush=True)
-        if outer and current_attention_phase()=='decode_verify':
-            expected=json.loads((native/phase/f'{index}.json').read_text())
-            ids=np.asarray(input_ids).reshape(-1).tolist()
-            assert expected['verify'] and ids==expected['ids'], (phase,index,ids,expected['ids'])
-            receipt=dict(case='verify_forward',phase=phase,index=index,ids=ids)
-            for part,a in zip(['logits','hidden'],result[:2]):
-                data=np.asarray(a.astype(mx.float32)).tobytes()
-                (out/f'{phase}-{index}-{part}.f32').write_bytes(data)
-                ref=(native/phase/f'{index}-{part}.f32').read_bytes()
-                receipt[part+'_exact']=data==ref
-                receipt[part+'_sha256']=hashlib.sha256(data).hexdigest()
-                if data!=ref:
-                    x=np.frombuffer(data,np.float32); y=np.frombuffer(ref,np.float32)
-                    receipt[part+'_different']=int(np.count_nonzero(x!=y))
-                    receipt[part+'_max_abs']=float(np.max(np.abs(x-y)))
-            print(json.dumps(receipt),flush=True)
-            assert receipt['logits_exact'] and receipt['hidden_exact'],receipt
-            index+=1
-        return result
-    return call
-rt.forward_ar=wrap(rt.forward_ar)
-rt.forward_ar_capture=wrap(rt.forward_ar_capture)
-bank=SessionBank()
-for phase in ['cold','warm']:
-    index=0
-    expected=json.loads((native/f'{phase}.json').read_text())
-    ids=expected['prompt']
-    result=generate_mtpk(rt,ids,max_tokens=12,sampler=SamplerConfig(temperature=0.,top_p=.95,top_k=20),speculative_depth=3,
-        seed=1,stop_token_ids=set(tokenizer.eos_token_ids),mtp_history_policy='committed',verify_strategy='batched',capture_final_state=True,
-        session_bank=bank,session_id='issue113',store_prefix_snapshot=False,
-        token_callback=lambda ids:print(json.dumps(dict(case='tokens',phase=phase,ids=ids)),flush=True))
-    assert result.tokens==expected['tokens'],(result.tokens,expected['tokens'])
-    assert result.stats.cached_tokens==expected['cached'],(result.stats.cached_tokens,expected['cached'])
-    assert index==len(list((native/phase).glob('*.json')))
-    final=result.final_state
-    assert final.safe_to_commit
-    for part,a in [('logits',final.final_logits),('hidden',final.final_hidden)]:
-        data=np.asarray(a.astype(mx.float32)).tobytes()
-        assert data==(native/f'{phase}-{part}.f32').read_bytes(),(phase,part)
-    def states(cache,lazy=False):
-        def array(a):
-            if a is None: return None
-            values=np.asarray(a.astype(mx.float32)).copy()
-            bits=hashlib.sha256(values.tobytes()).hexdigest()
-            values[values==0]=0
-            return dict(shape=list(a.shape),sha256=bits,value_sha256=hashlib.sha256(values.tobytes()).hexdigest())
-        return [[array(a) for a in state] if state is not None else None for state in (snapshot_cache_lazy_hybrid(cache) if lazy else snapshot_cache(cache)).states]
-    arrays=dict(trunk=states(final.final_trunk_cache,_lazy_snapshot_enabled()),mtp=states(final.final_committed_mtp_cache))
-    (out/f'{phase}-arrays.json').write_text(json.dumps(arrays))
-    native_arrays=json.loads((native/f'{phase}-arrays.json').read_text())
-    zero_sign_differences=[]
-    for family in arrays:
-        assert len(arrays[family])==len(native_arrays[family])
-        for layer,(actual,want) in enumerate(zip(arrays[family],native_arrays[family])):
-            if actual is None or want is None:
-                assert actual==want
-                continue
-            assert len(actual)==len(want)
-            for slot,(a,b) in enumerate(zip(actual,want)):
-                if a is None or b is None:
-                    assert a==b
-                    continue
-                assert a['shape']==b['shape'] and a['value_sha256']==b['value_sha256'],(phase,family,layer,slot,a,b)
-                if a['sha256']!=b['sha256']: zero_sign_differences.append([family,layer,slot])
-    assert zero_sign_differences==[], zero_sign_differences
-    # Compare the same lazy trunk snapshot used by the server/native bank.
-    # An extra deep clone would add zero and change negative-zero bits.
-    key=np.asarray((snapshot_cache_lazy_hybrid(final.final_trunk_cache) if _lazy_snapshot_enabled() else snapshot_cache(final.final_trunk_cache)).states[35][0].astype(mx.float32))
-    (out/f'{phase}-key35.f32').write_bytes(key.tobytes())
-    want=np.fromfile(native/f'{phase}-key35.f32',np.float32).reshape(key.shape)
-    different=key.view(np.uint32)!=want.view(np.uint32)
-    assert np.all(key[different]==0) and np.all(want[different]==0)
-    assert not np.any(different)
-    for part in ["key35","logits","hidden"]:
-        assert (out/f"{phase}-prefill-{part}.f32").read_bytes()==(native/f"{phase}-prefill-{part}.f32").read_bytes(),(phase,part,"prefill differs")
-    print(json.dumps(dict(case='cache_arrays',phase=phase,values_exact=True,zero_sign_differences=zero_sign_differences,key35_bit_coordinates=np.argwhere(different).tolist())),flush=True)
-    head=snapshot_cache(final.final_committed_mtp_cache)
-    key=ids+result.tokens
-    assert bank.put(runtime=rt,token_ids=key,cache=final.final_trunk_cache,logits=final.final_logits,hidden=final.final_hidden,
-        hidden_variant='post_norm',session_id='issue113',mtp_history_policy='committed',mtp_history_snapshot=head,
-        snapshot_epoch=len(key),mtp_snapshot_epoch=len(key)) is not None
-    print(json.dumps(dict(case='turn',phase=phase,tokens=result.tokens,cached=result.stats.cached_tokens,verify_calls=index,final_exact=True)),flush=True)
+runpy.run_path(str(Path(__file__).with_name("issue109-model-reference.py")), run_name="__main__")

tools/issue114-snapshot-reference.py

@@ -0,0 +1,38 @@
+"""Isolate the extra oracle snapshot that changed the reported key's zero sign."""
+import hashlib,json,subprocess,sys
+from pathlib import Path
+checkout,native,old_oracle,out=map(Path,sys.argv[1:5])
+assert subprocess.check_output(['git','-C',str(checkout),'rev-parse','HEAD'],text=True).strip()=='9882703f3105363ddc37eca9f97aa09a1d387112'
+sys.path.insert(0,str(checkout))
+import mlx.core as mx
+import numpy as np
+from mtplx.models.qwen4_exp import QSACache
+from mtplx.cache_state import snapshot_cache,snapshot_cache_lazy_hybrid
+assert mx.__version__=='0.32.2'
+def data(a):return np.asarray(a.view(mx.uint16 if a.dtype==mx.bfloat16 else mx.uint32)).tobytes()
+def cache(a):
+ c=QSACache();c.state=(a,a,None,None);return c
+cases=[]
+for dtype in [mx.bfloat16,mx.float32]:
+ bits=np.array([0x80000000,0,0xbf800000,0x3f800000,0,0x80000000,0x40000000,0xc0000000],dtype=np.uint32)
+ a=mx.array(bits.view(np.float32).reshape(1,2,1,4)).astype(dtype)
+ c=cache(a);raw=c.state[0];lazy=snapshot_cache_lazy_hybrid([c]);deep=snapshot_cache([c])
+ row=dict(dtype=str(dtype),shape=list(a.shape),raw=data(raw).hex(),lazy=data(lazy.states[0][0]).hex(),deep=data(deep.states[0][0]).hex())
+ assert row['raw']==row['lazy'] and row['raw']!=row['deep']
+ cases.append(row)
+traces=[]
+for phase in ['cold-prefill','cold','warm']:
+ f=native/(phase+'-key35.f32');before=f.read_bytes();shape=(1,2,len(before)//(2*256*4),256)
+ a=mx.array(np.frombuffer(before,dtype='<f4').copy().reshape(shape)).astype(mx.bfloat16)
+ def f32(x):return np.asarray(x.astype(mx.float32)).astype('<f4').tobytes()
+ assert f32(a)==before,'input reconstruction must retain every bit'
+ c=cache(a);raw=f32(c.state[0]);lazy=f32(snapshot_cache_lazy_hybrid([c]).states[0][0]);deep=f32(snapshot_cache([c]).states[0][0])
+ original=(old_oracle/(phase+'-key35.f32')).read_bytes()
+ assert raw==lazy==before
+ assert deep==original,'extra clone must reproduce the original mismatch exactly'
+ x=np.frombuffer(raw,np.uint32).reshape(shape);y=np.frombuffer(deep,np.uint32).reshape(shape)
+ coords=np.argwhere(x!=y).tolist()
+ for at in coords:assert x[tuple(at)]==0x80000000 and y[tuple(at)]==0
+ traces.append(dict(phase=phase,shape=shape,coordinates=coords,raw_sha256=hashlib.sha256(raw).hexdigest(),lazy_sha256=hashlib.sha256(lazy).hexdigest(),deep_sha256=hashlib.sha256(deep).hexdigest(),original_oracle_sha256=hashlib.sha256(original).hexdigest()))
+out.write_text(json.dumps(dict(revision='9882703f3105363ddc37eca9f97aa09a1d387112',mlx='0.32.2',cases=cases,real_key_traces=traces),indent=2)+'\n')
+print(json.dumps(dict(cases=len(cases),real_key_traces=traces)),flush=True)