Matching cache snapshot stages (#114)
The reported layer-35 zero-sign difference came from the oracle measurement.
tools/issue109-model-reference.py called snapshot_cache on the reference
trunk but compared it with the native server's lazy bank entry. Pinned
cache_state.py::_clone_tree implements a deep clone as value + zeros, which
turns negative zero into positive zero. snapshot_cache_lazy_hybrid, selected
by the server's default MTPLX_SESSION_LAZY_SNAPSHOT policy, retains the QSA key
bits. Native production already implements these two contracts correctly.
The oracle tool now compares the actual bank stages: policy-selected lazy/deep trunk, always-deep committed MTP head, and the suffix owner's final hidden row. Every cache leaf must match its raw-bit hash. The old zero-sign exception and numerical/canonical-zero comparison are removed. The #113 entry point invokes this same strict implementation so the two tools cannot drift. Historical #109 receipts are preserved and annotated with this correction.
This resolves a verification defect. There is no production arithmetic, RoPE, cache-format, model or DS4/GLM change, and no platform mismatch to approve. Changing the native zero sign would violate the actual lazy-snapshot oracle. This is a correction to the Qwen MTPLX comparison, not a DS4Server-only feature.
First changed bit
Pinned MTPLX 9882703f3105363ddc37eca9f97aa09a1d387112, MLX 0.32.2,
Bare-Speed artifact 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7 on M5 Max.
Native starting commit: d9026aa7bfbc8f74ec82c120178c34c5f08510d3.
The independent snapshot probe reads the original #109 native and oracle
raw key files. Reconstructing the BF16 key from its F32 export is required to
round-trip every bit before the probe proceeds. Installing it in a real pinned
QSACache and reading raw state / lazy snapshot preserves the native hash.
Taking the additional deep snapshot reproduces the original oracle file
byte-for-byte. In all three states, the only changed coordinate is
[0, 1, 0, 2]: F32 export 0x80000000 → 0x00000000.
| Original state | Key shape | Raw/lazy hash prefix | Extra deep snapshot / old oracle hash prefix |
|---|---|---|---|
| Cold prefill | 1×2×46×256 | d81b466dea1ff5b7 | 520dd44032eda478 |
| Cold final | 1×2×58×256 | 6a1c56e7333fc86e | 44b87b49e9f9853e |
| Warm final | 1×2×88×256 | 061f8fe052fe21c8 | 98186c7fec249807 |
Full hashes, coordinates and independently generated BF16/F32 fixtures are in
tests/fixtures/mtplx-snapshot-zero.json. This locates the first changed bit at
the measurement clone, without attributing it to an earlier model operation.
Regression and reproduction
The native regression exercises actual snapshot and restore paths for
BF16 and F32, lazy/deep snapshots and both restore-cloning modes. It compares
all bits to the independent fixture and verifies the source and banked arrays
remain unchanged, together with the restored cache offset.
Generate the fixture under isolated MLX 0.32.2 Python using:
tools/issue114-snapshot-reference.py PINNED_CHECKOUT ORIGINAL_NATIVE_DIR ORIGINAL_ORACLE_DIR tests/fixtures/mtplx-snapshot-zero.json.
The original directories contain cold-prefill-key35.f32, cold-key35.f32
and warm-key35.f32 from the #109 run, before the harness correction.
Build the release test executable (cargo test --release --all-features --bin ds4-server issue114 --no-run), copy it outside target, and run
issue114_snapshot_stage_preserves_zero_bits --ignored --nocapture --test-threads=1 under target/debug/test-supervisor 4096 60 120 --command.
For complete model verification, run the shared native
issue113_default_verify_and_checkpoint test in separate default/off processes
under the 114688 MiB supervisor, using the #113 documented environment and new
output directories. Run tools/issue109-model-reference.py against the original
explicit-off reproduction and tools/issue113-model-reference.py against the
default run. Both now use identical strict raw-bit comparison. The native
request is the original 46-token low-reasoning prompt, greedy seed 1, MTP depth
3 and max 12, with MTPLX_CONTEXT_COPY=0.
Completion review
Fresh serial default and explicit-off native/oracle runs pass: all eight verify forwards per configuration, every prefill/final logit and hidden byte, all trunk/MTP cache bits, emitted tokens, warm reuse and disk reopen. There are no remaining key coordinates with different bits. The original prefill key is negative zero in both implementations at the reported coordinate. All eight BF16/F32 snapshot/restore combinations match the independent reference. Installed DeepSeek session switching and GLM verifier/recurrent-state controls also pass; production code is unchanged.
Formatting, Clippy with -D warnings, signed app bundle and the full suite
pass (313 application + 8 tool tests; 237 ignored hardware/reference tests).
The first full-suite attempt reproduced existing #115: an unquoted Git SHA
parsed as a YAML float (5568000000.0). Its failure evidence is retained; the
unchanged retry passed. #115 remains a separate next-priority commit.
Review against the issue and pinned source is complete. The diagnostic cause is isolated, the incorrect oracle comparison is fixed, the raw-bit exception is removed, and focused plus full-model regressions verify the actual contract. No runtime mismatch is waived and no implementation is deferred.