Compare Qwen cache snapshots at matching stages

docs/issue114-snapshot-zero/README.md

@@ -0,0 +1,94 @@
+# Matching cache snapshot stages (#114)
+
+The reported layer-35 zero-sign difference came from the oracle measurement.
+`tools/issue109-model-reference.py` called `snapshot_cache` on the reference
+trunk but compared it with the native server's lazy bank entry. Pinned
+`cache_state.py::_clone_tree` implements a deep clone as `value + zeros`, which
+turns negative zero into positive zero. `snapshot_cache_lazy_hybrid`, selected
+by the server's default `MTPLX_SESSION_LAZY_SNAPSHOT` policy, retains the QSA key
+bits. Native production already implements these two contracts correctly.
+
+The oracle tool now compares the actual bank stages: policy-selected lazy/deep
+trunk, always-deep committed MTP head, and the suffix owner's final hidden row.
+Every cache leaf must match its raw-bit hash. The old zero-sign exception and
+numerical/canonical-zero comparison are removed. The #113 entry point invokes
+this same strict implementation so the two tools cannot drift. Historical #109
+receipts are preserved and annotated with this correction.
+
+This resolves a verification defect. There is no production arithmetic, RoPE,
+cache-format, model or DS4/GLM change, and no platform mismatch to approve.
+Changing the native zero sign would violate the actual lazy-snapshot oracle.
+This is a correction to the Qwen MTPLX comparison, not a DS4Server-only feature.
+
+## First changed bit
+
+Pinned MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2,
+Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7` on M5 Max.
+Native starting commit: `d9026aa7bfbc8f74ec82c120178c34c5f08510d3`.
+
+The independent snapshot probe reads the **original** #109 native and oracle
+raw key files. Reconstructing the BF16 key from its F32 export is required to
+round-trip every bit before the probe proceeds. Installing it in a real pinned
+QSACache and reading raw state / lazy snapshot preserves the native hash.
+Taking the additional deep snapshot reproduces the original oracle file
+byte-for-byte. In all three states, the only changed coordinate is
+`[0, 1, 0, 2]`: F32 export `0x80000000` → `0x00000000`.
+
+| Original state | Key shape | Raw/lazy hash prefix | Extra deep snapshot / old oracle hash prefix |
+|---|---|---|---|
+| Cold prefill | 1×2×46×256 | d81b466dea1ff5b7 | 520dd44032eda478 |
+| Cold final | 1×2×58×256 | 6a1c56e7333fc86e | 44b87b49e9f9853e |
+| Warm final | 1×2×88×256 | 061f8fe052fe21c8 | 98186c7fec249807 |
+
+Full hashes, coordinates and independently generated BF16/F32 fixtures are in
+`tests/fixtures/mtplx-snapshot-zero.json`. This locates the first changed bit at
+the measurement clone, without attributing it to an earlier model operation.
+
+## Regression and reproduction
+
+The native regression exercises actual `snapshot` and `restore` paths for
+BF16 and F32, lazy/deep snapshots and both restore-cloning modes. It compares
+all bits to the independent fixture and verifies the source and banked arrays
+remain unchanged, together with the restored cache offset.
+
+Generate the fixture under isolated MLX 0.32.2 Python using:
+`tools/issue114-snapshot-reference.py PINNED_CHECKOUT ORIGINAL_NATIVE_DIR
+ORIGINAL_ORACLE_DIR tests/fixtures/mtplx-snapshot-zero.json`.
+The original directories contain `cold-prefill-key35.f32`, `cold-key35.f32`
+and `warm-key35.f32` from the #109 run, before the harness correction.
+
+Build the release test executable (`cargo test --release --all-features --bin
+ds4-server issue114 --no-run`), copy it outside `target`, and run
+`issue114_snapshot_stage_preserves_zero_bits --ignored --nocapture
+--test-threads=1` under `target/debug/test-supervisor 4096 60 120 --command`.
+
+For complete model verification, run the shared native
+`issue113_default_verify_and_checkpoint` test in separate default/off processes
+under the 114688 MiB supervisor, using the #113 documented environment and new
+output directories. Run `tools/issue109-model-reference.py` against the original
+explicit-off reproduction and `tools/issue113-model-reference.py` against the
+default run. Both now use identical strict raw-bit comparison. The native
+request is the original 46-token low-reasoning prompt, greedy seed 1, MTP depth
+3 and max 12, with `MTPLX_CONTEXT_COPY=0`.
+
+## Completion review
+
+Fresh serial default and explicit-off native/oracle runs pass: all eight verify
+forwards per configuration, every prefill/final logit and hidden byte, all
+trunk/MTP cache bits, emitted tokens, warm reuse and disk reopen. There are no
+remaining key coordinates with different bits. The original prefill key is
+negative zero in both implementations at the reported coordinate. All eight
+BF16/F32 snapshot/restore combinations match the independent reference.
+Installed DeepSeek session switching and GLM verifier/recurrent-state controls
+also pass; production code is unchanged.
+
+Formatting, Clippy with `-D warnings`, signed app bundle and the full suite
+pass (313 application + 8 tool tests; 237 ignored hardware/reference tests).
+The first full-suite attempt reproduced existing #115: an unquoted Git SHA
+parsed as a YAML float (`5568000000.0`). Its failure evidence is retained; the
+unchanged retry passed. #115 remains a separate next-priority commit.
+
+Review against the issue and pinned source is complete. The diagnostic cause
+is isolated, the incorrect oracle comparison is fixed, the raw-bit exception
+is removed, and focused plus full-model regressions verify the actual contract.
+No runtime mismatch is waived and no implementation is deferred.