# Matching cache snapshot stages (#114)

The reported layer-35 zero-sign difference came from the oracle measurement.
`tools/issue109-model-reference.py` called `snapshot_cache` on the reference
trunk but compared it with the native server's lazy bank entry. Pinned
`cache_state.py::_clone_tree` implements a deep clone as `value + zeros`, which
turns negative zero into positive zero. `snapshot_cache_lazy_hybrid`, selected
by the server's default `MTPLX_SESSION_LAZY_SNAPSHOT` policy, retains the QSA key
bits. Native production already implements these two contracts correctly.

The oracle tool now compares the actual bank stages: policy-selected lazy/deep
trunk, always-deep committed MTP head, and the suffix owner's final hidden row.
Every cache leaf must match its raw-bit hash. The old zero-sign exception and
numerical/canonical-zero comparison are removed. The #113 entry point invokes
this same strict implementation so the two tools cannot drift. Historical #109
receipts are preserved and annotated with this correction.

This resolves a verification defect. There is no production arithmetic, RoPE,
cache-format, model or DS4/GLM change, and no platform mismatch to approve.
Changing the native zero sign would violate the actual lazy-snapshot oracle.
This is a correction to the Qwen MTPLX comparison, not a DS4Server-only feature.

## First changed bit

Pinned MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2,
Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7` on M5 Max.
Native starting commit: `d9026aa7bfbc8f74ec82c120178c34c5f08510d3`.

The independent snapshot probe reads the **original** #109 native and oracle
raw key files. Reconstructing the BF16 key from its F32 export is required to
round-trip every bit before the probe proceeds. Installing it in a real pinned
QSACache and reading raw state / lazy snapshot preserves the native hash.
Taking the additional deep snapshot reproduces the original oracle file
byte-for-byte. In all three states, the only changed coordinate is
`[0, 1, 0, 2]`: F32 export `0x80000000` → `0x00000000`.

| Original state | Key shape | Raw/lazy hash prefix | Extra deep snapshot / old oracle hash prefix |
|---|---|---|---|
| Cold prefill | 1×2×46×256 | d81b466dea1ff5b7 | 520dd44032eda478 |
| Cold final | 1×2×58×256 | 6a1c56e7333fc86e | 44b87b49e9f9853e |
| Warm final | 1×2×88×256 | 061f8fe052fe21c8 | 98186c7fec249807 |

Full hashes, coordinates and independently generated BF16/F32 fixtures are in
`tests/fixtures/mtplx-snapshot-zero.json`. This locates the first changed bit at
the measurement clone, without attributing it to an earlier model operation.

## Regression and reproduction

The native regression exercises actual `snapshot` and `restore` paths for
BF16 and F32, lazy/deep snapshots and both restore-cloning modes. It compares
all bits to the independent fixture and verifies the source and banked arrays
remain unchanged, together with the restored cache offset.

Generate the fixture under isolated MLX 0.32.2 Python using:
`tools/issue114-snapshot-reference.py PINNED_CHECKOUT ORIGINAL_NATIVE_DIR
ORIGINAL_ORACLE_DIR tests/fixtures/mtplx-snapshot-zero.json`.
The original directories contain `cold-prefill-key35.f32`, `cold-key35.f32`
and `warm-key35.f32` from the #109 run, before the harness correction.

Build the release test executable (`cargo test --release --all-features --bin
ds4-server issue114 --no-run`), copy it outside `target`, and run
`issue114_snapshot_stage_preserves_zero_bits --ignored --nocapture
--test-threads=1` under `target/debug/test-supervisor 4096 60 120 --command`.

For complete model verification, run the shared native
`issue113_default_verify_and_checkpoint` test in separate default/off processes
under the 114688 MiB supervisor, using the #113 documented environment and new
output directories. Run `tools/issue109-model-reference.py` against the original
explicit-off reproduction and `tools/issue113-model-reference.py` against the
default run. Both now use identical strict raw-bit comparison. The native
request is the original 46-token low-reasoning prompt, greedy seed 1, MTP depth
3 and max 12, with `MTPLX_CONTEXT_COPY=0`.

## Completion review

Fresh serial default and explicit-off native/oracle runs pass: all eight verify
forwards per configuration, every prefill/final logit and hidden byte, all
trunk/MTP cache bits, emitted tokens, warm reuse and disk reopen. There are no
remaining key coordinates with different bits. The original prefill key is
negative zero in both implementations at the reported coordinate. All eight
BF16/F32 snapshot/restore combinations match the independent reference.
Installed DeepSeek session switching and GLM verifier/recurrent-state controls
also pass; production code is unchanged.

Formatting, Clippy with `-D warnings`, signed app bundle and the full suite
pass (313 application + 8 tool tests; 237 ignored hardware/reference tests).
The first full-suite attempt reproduced existing #115: an unquoted Git SHA
parsed as a YAML float (`5568000000.0`). Its failure evidence is retained; the
unchanged retry passed. #115 remains a separate next-priority commit.

Review against the issue and pinned source is complete. The diagnostic cause
is isolated, the incorrect oracle comparison is fixed, the raw-bit exception
is removed, and focused plus full-model regressions verify the actual contract.
No runtime mismatch is waived and no implementation is deferred.