Found during #109's full-cache oracle comparison, separately from verify-phase gates and #113's head-chunked SDPA route.
Pinned MTPLX 9882703f3105363ddc37eca9f97aa09a1d387112 / MLX 0.32.2, installed Bare-Speed artifact 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7, M5 Max: the 46-token low-reasoning prompt for Count from one to ten. produces one different bit pattern in QSA layer 35's cached key. At [batch=0, kv_head=1, position=0, dimension=2], native stores -0.0 (0x80000000 when expanded to F32); the oracle stores +0.0 (0x00000000). This is present immediately after PREFILL, before any verification forward. The rest of that key tensor matches byte-for-byte. Cold final state retains it; all cold cache arrays are numerically exact, and all verify/final logits/hidden are byte exact with the common unchunked SDPA consumer.
No nonzero numerical, token, behavioral or performance consequence is demonstrated. This is a narrow raw-state parity finding; do not describe it as the #109 gate defect or silently normalize production tensors globally.
Reproduce with issue109_verify_rows_and_warm_checkpoint and tools/issue109-model-reference.py, with native MTPLX_CONTEXT_COPY=0 and reference MTPLX_QWEN4_VERIFY_SDPA_HEAD_CHUNK=0 to isolate unrelated routes. The native and oracle tools emit cold-prefill-key35.f32, cold-key35.f32, full-state raw/value hashes and exact differing coordinates. Evidence is retained in docs/issue109-verify-gate/ with that implementation.
Trace the first source of the signed zero against pinned MTPLX and correct the Qwen route if it differs from the reference. Do not assume the cause is RoPE merely from the final tensor. Add a focused primitive/graph regression for the confirmed source, retain unchanged-family controls, and recheck real prefill, cold/warm complete cache bits and logits. If it is an unavoidable platform requirement, document the concrete evidence and request the user's approval before accepting a difference, per AGENTS.md. Production remains native Rust/Metal; no global math-flag or DS4/GLM semantic change.
Run all four commit gates and deliver one isolated commit, push, implementation/test comment and closure. Numerical equality alone does not close this raw-bit parity issue.
Found during #109's full-cache oracle comparison, separately from verify-phase gates and #113's head-chunked SDPA route.
Pinned MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112` / MLX 0.32.2, installed Bare-Speed artifact `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, M5 Max: the 46-token low-reasoning prompt for `Count from one to ten.` produces one different bit pattern in QSA layer 35's cached key. At `[batch=0, kv_head=1, position=0, dimension=2]`, native stores `-0.0` (`0x80000000` when expanded to F32); the oracle stores `+0.0` (`0x00000000`). This is present immediately after PREFILL, before any verification forward. The rest of that key tensor matches byte-for-byte. Cold final state retains it; all cold cache arrays are numerically exact, and all verify/final logits/hidden are byte exact with the common unchunked SDPA consumer.
No nonzero numerical, token, behavioral or performance consequence is demonstrated. This is a narrow raw-state parity finding; do not describe it as the #109 gate defect or silently normalize production tensors globally.
Reproduce with `issue109_verify_rows_and_warm_checkpoint` and `tools/issue109-model-reference.py`, with native `MTPLX_CONTEXT_COPY=0` and reference `MTPLX_QWEN4_VERIFY_SDPA_HEAD_CHUNK=0` to isolate unrelated routes. The native and oracle tools emit `cold-prefill-key35.f32`, `cold-key35.f32`, full-state raw/value hashes and exact differing coordinates. Evidence is retained in `docs/issue109-verify-gate/` with that implementation.
Trace the first source of the signed zero against pinned MTPLX and correct the Qwen route if it differs from the reference. Do not assume the cause is RoPE merely from the final tensor. Add a focused primitive/graph regression for the confirmed source, retain unchanged-family controls, and recheck real prefill, cold/warm complete cache bits and logits. If it is an unavoidable platform requirement, document the concrete evidence and request the user's approval before accepting a difference, per AGENTS.md. Production remains native Rust/Metal; no global math-flag or DS4/GLM semantic change.
Run all four commit gates and deliver one isolated commit, push, implementation/test comment and closure. Numerical equality alone does not close this raw-bit parity issue.
hugo
added the bug label 2026-10-04 12:46:37 +00:00
The first changed bit is in the reference measurement: snapshot_cache deep-clones arrays using value + zeros. The original tool compared this extra deep snapshot against the native server's lazy trunk bank. Pinned MTPLX's actual lazy snapshot preserves -0.0, exactly as native production does.
An independent probe reconstructed the original BF16 layer-35 key with a required lossless F32-export round trip, installed it in pinned QSACache, and compared raw/lazy/deep stages. For the original cold prefill (46 positions), cold final (58) and warm final (88), raw and lazy snapshots exactly reproduce the native bytes. The extra deep clone exactly reproduces the original oracle bytes. In all three, its sole change is [0,1,0,2], 0x80000000 → 0x00000000. This confirms the measurement defect without assuming an earlier RoPE/math defect.
The #109 oracle now uses the matching server snapshot policy for trunk state, retains the required deep snapshot for the committed MTP head, and requires every raw cache hash to match. The signed-zero allowance and canonical-zero/numerical comparison are removed. #113 now invokes the same strict tool. Historical receipts are preserved with a correction note.
Verification in isolation:
Native BF16/F32 QSA snapshot/restore regression: all eight lazy/deep and restore-cloning combinations match independent oracle bits; source/banked arrays and offsets remain intact.
Fresh original explicit-off and default head-chunk native/oracle runs: exact prefill key/logits/hidden, all verify rows, emitted tokens, final logits/hidden, complete trunk/MTP cache bits, warm reuse and disk reopen. Zero differing key coordinates; no tolerance or waived runtime difference.
Installed DeepSeek session switching and GLM verifier/recurrent-state controls pass. Production/model/DS4/GLM code is unchanged.
All four gates pass: fmt, Clippy -D warnings, make bundle, all-features tests (313 application + 8 tool tests). The first suite attempt reproduced separately tracked #115's unquoted numeric-looking Git SHA; its evidence is retained and the unchanged retry passed.
Resolved and pushed in [5a8782b9b6b5a4c95efdf60c01414a5ff5d6e5c7](https://git.rfc1437.de/hugo/DS4Server/commit/5a8782b9b6b5a4c95efdf60c01414a5ff5d6e5c7).
The first changed bit is in the reference measurement: `snapshot_cache` deep-clones arrays using `value + zeros`. The original tool compared this extra deep snapshot against the native server's lazy trunk bank. Pinned MTPLX's actual lazy snapshot preserves `-0.0`, exactly as native production does.
An independent probe reconstructed the original BF16 layer-35 key with a required lossless F32-export round trip, installed it in pinned QSACache, and compared raw/lazy/deep stages. For the original cold prefill (46 positions), cold final (58) and warm final (88), raw and lazy snapshots exactly reproduce the native bytes. The extra deep clone exactly reproduces the original oracle bytes. In all three, its sole change is `[0,1,0,2]`, `0x80000000` → `0x00000000`. This confirms the measurement defect without assuming an earlier RoPE/math defect.
The #109 oracle now uses the matching server snapshot policy for trunk state, retains the required deep snapshot for the committed MTP head, and requires every raw cache hash to match. The signed-zero allowance and canonical-zero/numerical comparison are removed. #113 now invokes the same strict tool. Historical receipts are preserved with a correction note.
Verification in isolation:
- Native BF16/F32 QSA snapshot/restore regression: all eight lazy/deep and restore-cloning combinations match independent oracle bits; source/banked arrays and offsets remain intact.
- Fresh original explicit-off and default head-chunk native/oracle runs: exact prefill key/logits/hidden, all verify rows, emitted tokens, final logits/hidden, complete trunk/MTP cache bits, warm reuse and disk reopen. Zero differing key coordinates; no tolerance or waived runtime difference.
- Installed DeepSeek session switching and GLM verifier/recurrent-state controls pass. Production/model/DS4/GLM code is unchanged.
- All four gates pass: fmt, Clippy -D warnings, make bundle, all-features tests (313 application + 8 tool tests). The first suite attempt reproduced separately tracked #115's unquoted numeric-looking Git SHA; its evidence is retained and the unchanged retry passed.
[Root-cause trace, exact hashes, regressions, fresh model receipts and gate evidence](https://git.rfc1437.de/hugo/DS4Server/src/commit/5a8782b9b6b5a4c95efdf60c01414a5ff5d6e5c7/docs/issue114-snapshot-zero). This is a correction to the MTPLX verification harness. Native lazy cache behavior already follows the oracle; no production zero-sign change or platform exception is needed. The issue is fully resolved.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Found during #109's full-cache oracle comparison, separately from verify-phase gates and #113's head-chunked SDPA route.
Pinned MTPLX
9882703f3105363ddc37eca9f97aa09a1d387112/ MLX 0.32.2, installed Bare-Speed artifact74559cdf34fbfc0b593de72d17e93f37fd4f9ea7, M5 Max: the 46-token low-reasoning prompt forCount from one to ten.produces one different bit pattern in QSA layer 35's cached key. At[batch=0, kv_head=1, position=0, dimension=2], native stores-0.0(0x80000000when expanded to F32); the oracle stores+0.0(0x00000000). This is present immediately after PREFILL, before any verification forward. The rest of that key tensor matches byte-for-byte. Cold final state retains it; all cold cache arrays are numerically exact, and all verify/final logits/hidden are byte exact with the common unchunked SDPA consumer.No nonzero numerical, token, behavioral or performance consequence is demonstrated. This is a narrow raw-state parity finding; do not describe it as the #109 gate defect or silently normalize production tensors globally.
Reproduce with
issue109_verify_rows_and_warm_checkpointandtools/issue109-model-reference.py, with nativeMTPLX_CONTEXT_COPY=0and referenceMTPLX_QWEN4_VERIFY_SDPA_HEAD_CHUNK=0to isolate unrelated routes. The native and oracle tools emitcold-prefill-key35.f32,cold-key35.f32, full-state raw/value hashes and exact differing coordinates. Evidence is retained indocs/issue109-verify-gate/with that implementation.Trace the first source of the signed zero against pinned MTPLX and correct the Qwen route if it differs from the reference. Do not assume the cause is RoPE merely from the final tensor. Add a focused primitive/graph regression for the confirmed source, retain unchanged-family controls, and recheck real prefill, cold/warm complete cache bits and logits. If it is an unavoidable platform requirement, document the concrete evidence and request the user's approval before accepting a difference, per AGENTS.md. Production remains native Rust/Metal; no global math-flag or DS4/GLM semantic change.
Run all four commit gates and deliver one isolated commit, push, implementation/test comment and closure. Numerical equality alone does not close this raw-bit parity issue.
Resolved and pushed in 5a8782b9b6b5a4c95efdf60c01414a5ff5d6e5c7.
The first changed bit is in the reference measurement:
snapshot_cachedeep-clones arrays usingvalue + zeros. The original tool compared this extra deep snapshot against the native server's lazy trunk bank. Pinned MTPLX's actual lazy snapshot preserves-0.0, exactly as native production does.An independent probe reconstructed the original BF16 layer-35 key with a required lossless F32-export round trip, installed it in pinned QSACache, and compared raw/lazy/deep stages. For the original cold prefill (46 positions), cold final (58) and warm final (88), raw and lazy snapshots exactly reproduce the native bytes. The extra deep clone exactly reproduces the original oracle bytes. In all three, its sole change is
[0,1,0,2],0x80000000→0x00000000. This confirms the measurement defect without assuming an earlier RoPE/math defect.The #109 oracle now uses the matching server snapshot policy for trunk state, retains the required deep snapshot for the committed MTP head, and requires every raw cache hash to match. The signed-zero allowance and canonical-zero/numerical comparison are removed. #113 now invokes the same strict tool. Historical receipts are preserved with a correction note.
Verification in isolation:
Root-cause trace, exact hashes, regressions, fresh model receipts and gate evidence. This is a correction to the MTPLX verification harness. Native lazy cache behavior already follows the oracle; no production zero-sign change or platform exception is needed. The issue is fully resolved.