95 lines
5.3 KiB
Markdown
95 lines
5.3 KiB
Markdown
# Matching cache snapshot stages (#114)
|
||
|
||
The reported layer-35 zero-sign difference came from the oracle measurement.
|
||
`tools/issue109-model-reference.py` called `snapshot_cache` on the reference
|
||
trunk but compared it with the native server's lazy bank entry. Pinned
|
||
`cache_state.py::_clone_tree` implements a deep clone as `value + zeros`, which
|
||
turns negative zero into positive zero. `snapshot_cache_lazy_hybrid`, selected
|
||
by the server's default `MTPLX_SESSION_LAZY_SNAPSHOT` policy, retains the QSA key
|
||
bits. Native production already implements these two contracts correctly.
|
||
|
||
The oracle tool now compares the actual bank stages: policy-selected lazy/deep
|
||
trunk, always-deep committed MTP head, and the suffix owner's final hidden row.
|
||
Every cache leaf must match its raw-bit hash. The old zero-sign exception and
|
||
numerical/canonical-zero comparison are removed. The #113 entry point invokes
|
||
this same strict implementation so the two tools cannot drift. Historical #109
|
||
receipts are preserved and annotated with this correction.
|
||
|
||
This resolves a verification defect. There is no production arithmetic, RoPE,
|
||
cache-format, model or DS4/GLM change, and no platform mismatch to approve.
|
||
Changing the native zero sign would violate the actual lazy-snapshot oracle.
|
||
This is a correction to the Qwen MTPLX comparison, not a DS4Server-only feature.
|
||
|
||
## First changed bit
|
||
|
||
Pinned MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2,
|
||
Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7` on M5 Max.
|
||
Native starting commit: `d9026aa7bfbc8f74ec82c120178c34c5f08510d3`.
|
||
|
||
The independent snapshot probe reads the **original** #109 native and oracle
|
||
raw key files. Reconstructing the BF16 key from its F32 export is required to
|
||
round-trip every bit before the probe proceeds. Installing it in a real pinned
|
||
QSACache and reading raw state / lazy snapshot preserves the native hash.
|
||
Taking the additional deep snapshot reproduces the original oracle file
|
||
byte-for-byte. In all three states, the only changed coordinate is
|
||
`[0, 1, 0, 2]`: F32 export `0x80000000` → `0x00000000`.
|
||
|
||
| Original state | Key shape | Raw/lazy hash prefix | Extra deep snapshot / old oracle hash prefix |
|
||
|---|---|---|---|
|
||
| Cold prefill | 1×2×46×256 | d81b466dea1ff5b7 | 520dd44032eda478 |
|
||
| Cold final | 1×2×58×256 | 6a1c56e7333fc86e | 44b87b49e9f9853e |
|
||
| Warm final | 1×2×88×256 | 061f8fe052fe21c8 | 98186c7fec249807 |
|
||
|
||
Full hashes, coordinates and independently generated BF16/F32 fixtures are in
|
||
`tests/fixtures/mtplx-snapshot-zero.json`. This locates the first changed bit at
|
||
the measurement clone, without attributing it to an earlier model operation.
|
||
|
||
## Regression and reproduction
|
||
|
||
The native regression exercises actual `snapshot` and `restore` paths for
|
||
BF16 and F32, lazy/deep snapshots and both restore-cloning modes. It compares
|
||
all bits to the independent fixture and verifies the source and banked arrays
|
||
remain unchanged, together with the restored cache offset.
|
||
|
||
Generate the fixture under isolated MLX 0.32.2 Python using:
|
||
`tools/issue114-snapshot-reference.py PINNED_CHECKOUT ORIGINAL_NATIVE_DIR
|
||
ORIGINAL_ORACLE_DIR tests/fixtures/mtplx-snapshot-zero.json`.
|
||
The original directories contain `cold-prefill-key35.f32`, `cold-key35.f32`
|
||
and `warm-key35.f32` from the #109 run, before the harness correction.
|
||
|
||
Build the release test executable (`cargo test --release --all-features --bin
|
||
ds4-server issue114 --no-run`), copy it outside `target`, and run
|
||
`issue114_snapshot_stage_preserves_zero_bits --ignored --nocapture
|
||
--test-threads=1` under `target/debug/test-supervisor 4096 60 120 --command`.
|
||
|
||
For complete model verification, run the shared native
|
||
`issue113_default_verify_and_checkpoint` test in separate default/off processes
|
||
under the 114688 MiB supervisor, using the #113 documented environment and new
|
||
output directories. Run `tools/issue109-model-reference.py` against the original
|
||
explicit-off reproduction and `tools/issue113-model-reference.py` against the
|
||
default run. Both now use identical strict raw-bit comparison. The native
|
||
request is the original 46-token low-reasoning prompt, greedy seed 1, MTP depth
|
||
3 and max 12, with `MTPLX_CONTEXT_COPY=0`.
|
||
|
||
## Completion review
|
||
|
||
Fresh serial default and explicit-off native/oracle runs pass: all eight verify
|
||
forwards per configuration, every prefill/final logit and hidden byte, all
|
||
trunk/MTP cache bits, emitted tokens, warm reuse and disk reopen. There are no
|
||
remaining key coordinates with different bits. The original prefill key is
|
||
negative zero in both implementations at the reported coordinate. All eight
|
||
BF16/F32 snapshot/restore combinations match the independent reference.
|
||
Installed DeepSeek session switching and GLM verifier/recurrent-state controls
|
||
also pass; production code is unchanged.
|
||
|
||
Formatting, Clippy with `-D warnings`, signed app bundle and the full suite
|
||
pass (313 application + 8 tool tests; 237 ignored hardware/reference tests).
|
||
The first full-suite attempt reproduced existing #115: an unquoted Git SHA
|
||
parsed as a YAML float (`5568000000.0`). Its failure evidence is retained; the
|
||
unchanged retry passed. #115 remains a separate next-priority commit.
|
||
|
||
Review against the issue and pinned source is complete. The diagnostic cause
|
||
is isolated, the incorrect oracle comparison is fixed, the raw-bit exception
|
||
is removed, and focused plus full-model regressions verify the actual contract.
|
||
No runtime mismatch is waived and no implementation is deferred.
|