Publication status: analysis commit dcc4c516f0e030566c67472f8f1be0baa21ab629 is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under /Users/gb/Projects/DS4Server/docs/issue103-analysis/ and the committed test/tool paths, not yet at the remote file URLs.
Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the independent review and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
Qwen 3.8 Flash Next: behavioral/numerical parity with the pinned MTPLX 2.12.2 revision 9882703f3105363ddc37eca9f97aa09a1d387112, including token processing, RNG/sampling, context accounting, inference and cache/session state.
DSv4 Flash and GLM 5.3 Flash: preserve DS4 behavior. A Qwen fix must not change their token IDs, sampler/RNG sequences, numerical inference, context accounting or cache behavior.
Keep changes in the Qwen family/phase/dispatch where applicable. Before changing any shared helper, trace every caller; preserve DS4-backed callers exactly and provide DS4 regression controls. A helper shared by Qwen and GLM must not silently impose MTPLX semantics on GLM. Do not globally change the DS4 sampler, shared RNG, Unicode tables, sigmoid/fast-math rules, or non-Qwen checkpoint formats for this ticket.
This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
Broken behavior and evidence
speculative::decode_mtp takes the pending primary, emits the next primary, then exits at round head for a stop or token limit without re-arming it for the final commit. The final pending-token commit consequently does nothing. TurnDecoded::final_arrays/capture_session regard pending=None, final_state_error=None as safe; restore_session trusts the banked token-key length. Active caller: Generator → QwenExecutor → LiveSession → generate_turn → decode_mtp → capture_session → LiveSession.current/persistence → next restore_session.
Real unchanged Bare-Speed model, raw prompt IDs [9707,11,1075], chunk 2048, SSD PLE, seed 0, depth 1/2/3, temperature 0/.6: both max_tokens=1 and first-primary stop (max_tokens=8, stop ID 11 greedy or 314 sampled) return a 4-token bank key, reported restore length 4, twelve trunk QSA KV offsets all 3, MTP offset 2, pending=None and no final-state error. Emitted primary is 11 or 314. Thus the saved recurrent/PLE/QSA state remains the prompt state. Next suffix [1075] differs from properly teacher-forced [314,1075] at 247,643 logits. Bad/good F32-byte SHA256: 9c22c019eeb07fead0760dc06ef796e4fbf3f5b4946b02ad808f430ecc6cee5c / 9a556dd9d3063864b70c3338cb0cdf76cc578a01de1c7771b1da0abf3e776feb. Peak physical memory 73,909,577,088 bytes; watchdog error null. See native-model.jsonl and native-final.log.
Native implementation scope
Start with the minimal round-head fix matching upstream: re-arm the emitted primary (result.pending = Some(cycle.primary) or equivalent) before the stop/max-token break, so the existing final-pending commit advances trunk state and committed MTP history exactly once. Preserve acceptance/RNG and other exit behavior. Do not shorten the bank key to disguise missing state. Keep final_arrays/capture_session/request/owner redesign outside scope unless an owned failing test proves an additional change necessary.
Handle already-persisted bad Qwen snapshots separately: current QWMTKV01 files may carry a key longer than the saved state. Before reuse after the fix, reject/invalidate pre-fix affected snapshots through a Qwen checkpoint format/policy version or a validated state/key consistency check that detects this defect. Also prevent stale in-memory reuse during any supported live upgrade. A repaired new snapshot alone does not repair an old persisted one. No change to DS4-family checkpoint formats or unrelated session owners.
Acceptance
Convert the owned first-primary stop/max-token diagnostic to complete-state assertions and teacher-forced next-turn comparison at depth 1/2/3, greedy/native .6. Assert trunk KV, GDN/conv, PLE/QSA and committed MTP history match the banked IDs; final commit must occur exactly once. Retain the upstream exit matrix for fresh primary, deferred greedy correction, bonus, accepted-draft stop and sampled correction as currently unmeasured coverage. Cancellation/final-capture failure must not publish a falsely complete state. Produce a pre-fix bad QWMTKV01 checkpoint with this reproduction, then prove post-fix load rejects/invalidate it and cold recovery is correct; persist/reopen a new correct snapshot and compare the next turn. DSv4/GLM checkpoint/state behavior stays unchanged. No broad snapshot redesign without reproduced need.
Severity
High: real-model warm-state corruption and next-turn logit divergence are reproduced.
Shared evidence and reproduction
Analysis parent: #103. Affected DS4Server revision: cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983. Compared MTPLX old e652d55e2652137a4abcf1312357abbf3eb9d692 (2.10.2) with refreshed 9882703f3105363ddc37eca9f97aa09a1d387112 (2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7; unchanged hashes, compiler/OS/hardware and effective options in environment.json. Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.
Durable report and instructions, native test, oracle probe. The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
cargo test --all-features --bin ds4-server issue103_mtp_final_receipts --no-run
cargo build --bin test-supervisor
exportDS4_QWEN38_ARTIFACTS='/path/to/unchanged/qwen3.8-flash-next'exportDS4_ISSUE103_ORACLE="$PWD/docs/issue103-analysis/latest-oracle.jsonl"# Resolve the emitted test executable; do not pass cargo to the watchdog.
target/debug/test-supervisor 10035260120 --command /path/to/ds4_server-test-binary issue103_mtp_final_receipts --ignored --nocapture --test-threads=1
Immutable upstream change and latest test. The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
Validation and review boundary
Implement production behavior in Rust with existing Metal/Objective-C bridge only; Python remains an isolated oracle.
Run cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, make bundle, cargo test --all-features before every future commit, plus the owned native/oracle and product regressions above.
Implementation awaits the user's independent-agent review and subsequent instruction. Creating this issue or closing #103 does not authorize a fix. Review the corrected model-specific oracle, active-route applicability and state ownership first; preserve DS4-backed families.
No arbitrary dependencies on sibling issues: coordinate shared source changes where needed; each issue owns its concrete failure and acceptance.
Publication status: analysis commit `dcc4c516f0e030566c67472f8f1be0baa21ab629` is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under `/Users/gb/Projects/DS4Server/docs/issue103-analysis/` and the committed test/tool paths, not yet at the remote file URLs.
## Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the [independent review](https://git.rfc1437.de/hugo/DS4Server/issues/106#issuecomment-3068) and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
## Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
- Qwen 3.8 Flash Next: behavioral/numerical parity with the pinned MTPLX 2.12.2 revision `9882703f3105363ddc37eca9f97aa09a1d387112`, including token processing, RNG/sampling, context accounting, inference and cache/session state.
- DSv4 Flash and GLM 5.3 Flash: preserve DS4 behavior. A Qwen fix must not change their token IDs, sampler/RNG sequences, numerical inference, context accounting or cache behavior.
- Keep changes in the Qwen family/phase/dispatch where applicable. Before changing any shared helper, trace every caller; preserve DS4-backed callers exactly and provide DS4 regression controls. A helper shared by Qwen and GLM must not silently impose MTPLX semantics on GLM. Do not globally change the DS4 sampler, shared RNG, Unicode tables, sigmoid/fast-math rules, or non-Qwen checkpoint formats for this ticket.
This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
## Broken behavior and evidence
`speculative::decode_mtp` takes the pending primary, emits the next primary, then exits at round head for a stop or token limit without re-arming it for the final commit. The final pending-token commit consequently does nothing. `TurnDecoded::final_arrays`/capture_session regard `pending=None, final_state_error=None` as safe; restore_session trusts the banked token-key length. Active caller: Generator → QwenExecutor → LiveSession → generate_turn → decode_mtp → capture_session → LiveSession.current/persistence → next restore_session.
Real unchanged Bare-Speed model, raw prompt IDs [9707,11,1075], chunk 2048, SSD PLE, seed 0, depth 1/2/3, temperature 0/.6: both max_tokens=1 and first-primary stop (max_tokens=8, stop ID 11 greedy or 314 sampled) return a 4-token bank key, reported restore length 4, twelve trunk QSA KV offsets all 3, MTP offset 2, pending=None and no final-state error. Emitted primary is 11 or 314. Thus the saved recurrent/PLE/QSA state remains the prompt state. Next suffix [1075] differs from properly teacher-forced [314,1075] at 247,643 logits. Bad/good F32-byte SHA256: 9c22c019eeb07fead0760dc06ef796e4fbf3f5b4946b02ad808f430ecc6cee5c / 9a556dd9d3063864b70c3338cb0cdf76cc578a01de1c7771b1da0abf3e776feb. Peak physical memory 73,909,577,088 bytes; watchdog error null. See native-model.jsonl and native-final.log.
## Native implementation scope
Start with the minimal round-head fix matching upstream: re-arm the emitted primary (`result.pending = Some(cycle.primary)` or equivalent) before the stop/max-token break, so the existing final-pending commit advances trunk state and committed MTP history exactly once. Preserve acceptance/RNG and other exit behavior. Do not shorten the bank key to disguise missing state. Keep final_arrays/capture_session/request/owner redesign outside scope unless an owned failing test proves an additional change necessary.
Handle already-persisted bad Qwen snapshots separately: current `QWMTKV01` files may carry a key longer than the saved state. Before reuse after the fix, reject/invalidate pre-fix affected snapshots through a Qwen checkpoint format/policy version or a validated state/key consistency check that detects this defect. Also prevent stale in-memory reuse during any supported live upgrade. A repaired new snapshot alone does not repair an old persisted one. No change to DS4-family checkpoint formats or unrelated session owners.
## Acceptance
Convert the owned first-primary stop/max-token diagnostic to complete-state assertions and teacher-forced next-turn comparison at depth 1/2/3, greedy/native .6. Assert trunk KV, GDN/conv, PLE/QSA and committed MTP history match the banked IDs; final commit must occur exactly once. Retain the upstream exit matrix for fresh primary, deferred greedy correction, bonus, accepted-draft stop and sampled correction as currently unmeasured coverage. Cancellation/final-capture failure must not publish a falsely complete state. Produce a pre-fix bad QWMTKV01 checkpoint with this reproduction, then prove post-fix load rejects/invalidate it and cold recovery is correct; persist/reopen a new correct snapshot and compare the next turn. DSv4/GLM checkpoint/state behavior stays unchanged. No broad snapshot redesign without reproduced need.
## Severity
High: real-model warm-state corruption and next-turn logit divergence are reproduced.
## Shared evidence and reproduction
Analysis parent: [#103](https://git.rfc1437.de/hugo/DS4Server/issues/103). Affected DS4Server revision: `cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983`. Compared MTPLX old `e652d55e2652137a4abcf1312357abbf3eb9d692` (2.10.2) with refreshed `9882703f3105363ddc37eca9f97aa09a1d387112` (2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`; unchanged hashes, compiler/OS/hardware and effective options in [environment.json](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/docs/issue103-analysis/environment.json). Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.
Durable [report and instructions](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/docs/issue103-analysis/README.md), [native test](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/src/engine/metal/qwen_mtplx/issue103_analysis.rs), [oracle probe](https://git.rfc1437.de/hugo/DS4Server/src/commit/dcc4c516f0e030566c67472f8f1be0baa21ab629/tools/issue103-reference.py). The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
```sh
cargo test --all-features --bin ds4-server issue103_mtp_final_receipts --no-run
cargo build --bin test-supervisor
export DS4_QWEN38_ARTIFACTS='/path/to/unchanged/qwen3.8-flash-next'
export DS4_ISSUE103_ORACLE="$PWD/docs/issue103-analysis/latest-oracle.jsonl"
# Resolve the emitted test executable; do not pass cargo to the watchdog.
target/debug/test-supervisor 100352 60 120 --command /path/to/ds4_server-test-binary issue103_mtp_final_receipts --ignored --nocapture --test-threads=1
```
Immutable upstream [change](https://github.com/youssofal/mtplx/commit/1a977b65b002d7efcdfeed69c209d2129abc2b67) and [latest test](https://github.com/youssofal/mtplx/blob/9882703f3105363ddc37eca9f97aa09a1d387112/tests/test_mtp_final_state_covers_committed_tokens.py). The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
## Validation and review boundary
- Implement production behavior in Rust with existing Metal/Objective-C bridge only; Python remains an isolated oracle.
- Run `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `make bundle`, `cargo test --all-features` before every future commit, plus the owned native/oracle and product regressions above.
- Implementation awaits the user's independent-agent review and subsequent instruction. Creating this issue or closing #103 does not authorize a fix. Review the corrected model-specific oracle, active-route applicability and state ownership first; preserve DS4-backed families.
- No arbitrary dependencies on sibling issues: coordinate shared source changes where needed; each issue owns its concrete failure and acceptance.
hugo
added the bug label 2026-10-03 09:20:31 +00:00
Reviewed against DS4Server dcc4c51 and MTPLX 2.12.2 9882703f3105363ddc37eca9f97aa09a1d387112. Note: evidence commit dcc4c51 is still unpublished (main is ahead of origin/main by one commit), so the repository links in this issue currently resolve to nothing.
Defect claim: confirmed. In decode_mtp the round head (speculative.rs:722-737) takes pending or samples a fresh primary, pushes it into result.tokens, then breaks on stop or max_tokens with pending == None. The final commit (if let Some(token) = result.pending) therefore never forwards it. This is precisely upstream defect 2 of 1a977b65.
Proposed fix: correct, but broader than needed, and one gap.
The upstream fix is a single re-arm before the break (pending_primary = int(primary)), so the existing final-pending commit extends trunk cache and committed MTP history by exactly that token. The native equivalent is result.pending = Some(cycle.primary) in the round-head exit branch. Keep the broader final_arrays/capture_session redesign out of scope unless a test shows it is needed.
Missing: snapshots already persisted through session_checkpoint (magic QWMTKV01) under this defect stay corrupt after the fix (key one token longer than state). Acceptance should require invalidating them, e.g. a checkpoint format/version bump, not only "persist/reopen the completed snapshot".
The remaining exit branches (accepted-draft stop at speculative.rs:~887, bonus, correction) are rightly listed as unmeasured. The upstream regression matrix in tests/test_mtp_final_state_covers_committed_tokens.py is the right template.
## Independent review (2026-10-03)
Reviewed against DS4Server `dcc4c51` and MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`. Note: evidence commit `dcc4c51` is still unpublished (`main` is ahead of `origin/main` by one commit), so the repository links in this issue currently resolve to nothing.
**Defect claim: confirmed.** In `decode_mtp` the round head (speculative.rs:722-737) takes `pending` or samples a fresh primary, pushes it into `result.tokens`, then breaks on stop or `max_tokens` with `pending == None`. The final commit (`if let Some(token) = result.pending`) therefore never forwards it. This is precisely upstream defect 2 of `1a977b65`.
**Proposed fix: correct, but broader than needed, and one gap.**
- The upstream fix is a single re-arm before the break (`pending_primary = int(primary)`), so the existing final-pending commit extends trunk cache and committed MTP history by exactly that token. The native equivalent is `result.pending = Some(cycle.primary)` in the round-head exit branch. Keep the broader `final_arrays`/`capture_session` redesign out of scope unless a test shows it is needed.
- **Missing:** snapshots already persisted through `session_checkpoint` (magic `QWMTKV01`) under this defect stay corrupt after the fix (key one token longer than state). Acceptance should require invalidating them, e.g. a checkpoint format/version bump, not only "persist/reopen the completed snapshot".
- The remaining exit branches (accepted-draft stop at speculative.rs:~887, bonus, correction) are rightly listed as unmeasured. The upstream regression matrix in `tests/test_mtp_final_state_covers_committed_tokens.py` is the right template.
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the minimal pending re-arm reaches the existing final commit once, compare all banked-state boundaries, and verify the proposed Qwen-only compatibility mechanism rejects a real pre-fix corrupt checkpoint.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit dcc4c51 remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.
## Description corrected after independent review (2026-10-03)
Updated the issue body in response to [your review](https://git.rfc1437.de/hugo/DS4Server/issues/106#issuecomment-3068). Changes:
- Narrowed the production fix to re-arming the pending primary before the round-head exit; retained existing final commit.
- Removed speculative final_arrays/capture_session redesign from mandatory scope.
- Added explicit invalidation/detection of already-persisted bad QWMTKV01 state and stale live reuse.
- Added pre-fix checkpoint → post-fix rejection/recovery acceptance, alongside new-checkpoint roundtrip.
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the minimal pending re-arm reaches the existing final commit once, compare all banked-state boundaries, and verify the proposed Qwen-only compatibility mechanism rejects a real pre-fix corrupt checkpoint.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit dcc4c51 remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.
Implemented and verified in commit a8cad8810060d22ee7ccac82c210ef40d4e08785 (pushed to main).
The round-head stop/token-limit exit now keeps the emitted primary pending, matching MTPLX 2.12.2 9882703f3105363ddc37eca9f97aa09a1d387112. The existing final commit advances trunk and committed MTP history exactly once. Qwen checkpoints now use QWMTKV02; old V1 files are rejected before tensor allocation, removed by the existing Generator incompatibility path, and rebuilt cold. No snapshot-owner redesign or DS4/GLM checkpoint/inference change was needed. Resident state is process-local and cannot survive the executable restart used for upgrades.
Validation performed in isolation:
Reproduced all 12 original installed-model cases and saved an actual bad V1 checkpoint (116,442,383 bytes, SHA256 8cd38359515e9d30b191992cc477393ee59b5cac7b6d610aa5838e589ce02c64).
All 12 corrected depth 1/2/3, temperature 0/.6, stop/max-token cases match independently teacher-forced state tensor-for-tensor, including GDN/conv, PLE, QSA/KV, logits, hidden state and committed MTP history. Trunk offsets are 4, MTP offset 3, key/reused length 4.
Each corrected snapshot persists/reopens exactly and matches the next suffix's complete state. Real old checkpoint rejection/removal and cold product recovery match an independent cold request.
All eight upstream exit cases match native tokens, stop route and cache coverage on real model caches with scripted logits. Upstream AR control passes.
Cancellation, failed final forward and disabled final capture cannot expose a falsely complete final snapshot. Existing checkpoint roundtrip/corruption/abort/atomic-write controls pass.
All required gates pass: fmt, Clippy with warnings denied, signed app bundle, all-feature tests (308 application tests plus 8 tool tests).
Reviewed the complete diff against this issue, pinned MTPLX and DS4's separate committed-state/checkpoint behavior. No remaining acceptance gap found. Durable evidence and reproduction instructions: issue106-final-primary.
Implemented and verified in commit `a8cad8810060d22ee7ccac82c210ef40d4e08785` (pushed to main).
The round-head stop/token-limit exit now keeps the emitted primary pending, matching MTPLX 2.12.2 `9882703f3105363ddc37eca9f97aa09a1d387112`. The existing final commit advances trunk and committed MTP history exactly once. Qwen checkpoints now use `QWMTKV02`; old V1 files are rejected before tensor allocation, removed by the existing Generator incompatibility path, and rebuilt cold. No snapshot-owner redesign or DS4/GLM checkpoint/inference change was needed. Resident state is process-local and cannot survive the executable restart used for upgrades.
Validation performed in isolation:
- Reproduced all 12 original installed-model cases and saved an actual bad V1 checkpoint (116,442,383 bytes, SHA256 `8cd38359515e9d30b191992cc477393ee59b5cac7b6d610aa5838e589ce02c64`).
- All 12 corrected depth 1/2/3, temperature 0/.6, stop/max-token cases match independently teacher-forced state tensor-for-tensor, including GDN/conv, PLE, QSA/KV, logits, hidden state and committed MTP history. Trunk offsets are 4, MTP offset 3, key/reused length 4.
- Each corrected snapshot persists/reopens exactly and matches the next suffix's complete state. Real old checkpoint rejection/removal and cold product recovery match an independent cold request.
- All eight upstream exit cases match native tokens, stop route and cache coverage on real model caches with scripted logits. Upstream AR control passes.
- Cancellation, failed final forward and disabled final capture cannot expose a falsely complete final snapshot. Existing checkpoint roundtrip/corruption/abort/atomic-write controls pass.
- All required gates pass: fmt, Clippy with warnings denied, signed app bundle, all-feature tests (308 application tests plus 8 tool tests).
Reviewed the complete diff against this issue, pinned MTPLX and DS4's separate committed-state/checkpoint behavior. No remaining acceptance gap found. Durable evidence and reproduction instructions: [issue106-final-primary](https://git.rfc1437.de/hugo/DS4Server/src/commit/a8cad8810060d22ee7ccac82c210ef40d4e08785/docs/issue106-final-primary/README.md).
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Publication status: analysis commit
dcc4c516f0e030566c67472f8f1be0baa21ab629is durable locally; its push to origin/main awaits explicit approval after automatic review refused the default-branch push. Until published, linked repository evidence files are available locally under/Users/gb/Projects/DS4Server/docs/issue103-analysis/and the committed test/tool paths, not yet at the remote file URLs.Review status (updated 2026-10-03)
Confirmed defect or numerical parity difference; implementation still awaits verification. This description incorporates the independent review and the user's explicit model-specific oracle clarification. It supersedes the original issue scope and conflicting conclusions in the initial evidence report. Original raw receipts/diagnostics remain unchanged; no production fix or new runtime/performance result is claimed. The report and AGENTS.md have not been rewritten in this issue-only update.
Model-specific parity boundary
The user explicitly clarified the oracle assignment after independent review:
9882703f3105363ddc37eca9f97aa09a1d387112, including token processing, RNG/sampling, context accounting, inference and cache/session state.This is the user's model-specific clarification of the blanket DS4-oracle wording in AGENTS.md, not permission to weaken DS4 parity. Production remains native Rust/Metal with the existing Objective-C bridge; MTPLX Python is an isolated oracle only.
Broken behavior and evidence
speculative::decode_mtptakes the pending primary, emits the next primary, then exits at round head for a stop or token limit without re-arming it for the final commit. The final pending-token commit consequently does nothing.TurnDecoded::final_arrays/capture_session regardpending=None, final_state_error=Noneas safe; restore_session trusts the banked token-key length. Active caller: Generator → QwenExecutor → LiveSession → generate_turn → decode_mtp → capture_session → LiveSession.current/persistence → next restore_session.Real unchanged Bare-Speed model, raw prompt IDs [9707,11,1075], chunk 2048, SSD PLE, seed 0, depth 1/2/3, temperature 0/.6: both max_tokens=1 and first-primary stop (max_tokens=8, stop ID 11 greedy or 314 sampled) return a 4-token bank key, reported restore length 4, twelve trunk QSA KV offsets all 3, MTP offset 2, pending=None and no final-state error. Emitted primary is 11 or 314. Thus the saved recurrent/PLE/QSA state remains the prompt state. Next suffix [1075] differs from properly teacher-forced [314,1075] at 247,643 logits. Bad/good F32-byte SHA256: 9c22c019eeb07fead0760dc06ef796e4fbf3f5b4946b02ad808f430ecc6cee5c / 9a556dd9d3063864b70c3338cb0cdf76cc578a01de1c7771b1da0abf3e776feb. Peak physical memory 73,909,577,088 bytes; watchdog error null. See native-model.jsonl and native-final.log.
Native implementation scope
Start with the minimal round-head fix matching upstream: re-arm the emitted primary (
result.pending = Some(cycle.primary)or equivalent) before the stop/max-token break, so the existing final-pending commit advances trunk state and committed MTP history exactly once. Preserve acceptance/RNG and other exit behavior. Do not shorten the bank key to disguise missing state. Keep final_arrays/capture_session/request/owner redesign outside scope unless an owned failing test proves an additional change necessary.Handle already-persisted bad Qwen snapshots separately: current
QWMTKV01files may carry a key longer than the saved state. Before reuse after the fix, reject/invalidate pre-fix affected snapshots through a Qwen checkpoint format/policy version or a validated state/key consistency check that detects this defect. Also prevent stale in-memory reuse during any supported live upgrade. A repaired new snapshot alone does not repair an old persisted one. No change to DS4-family checkpoint formats or unrelated session owners.Acceptance
Convert the owned first-primary stop/max-token diagnostic to complete-state assertions and teacher-forced next-turn comparison at depth 1/2/3, greedy/native .6. Assert trunk KV, GDN/conv, PLE/QSA and committed MTP history match the banked IDs; final commit must occur exactly once. Retain the upstream exit matrix for fresh primary, deferred greedy correction, bonus, accepted-draft stop and sampled correction as currently unmeasured coverage. Cancellation/final-capture failure must not publish a falsely complete state. Produce a pre-fix bad QWMTKV01 checkpoint with this reproduction, then prove post-fix load rejects/invalidate it and cold recovery is correct; persist/reopen a new correct snapshot and compare the next turn. DSv4/GLM checkpoint/state behavior stays unchanged. No broad snapshot redesign without reproduced need.
Severity
High: real-model warm-state corruption and next-turn logit divergence are reproduced.
Shared evidence and reproduction
Analysis parent: #103. Affected DS4Server revision:
cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983. Compared MTPLX olde652d55e2652137a4abcf1312357abbf3eb9d692(2.10.2) with refreshed9882703f3105363ddc37eca9f97aa09a1d387112(2.12.2), on MLX exactly 0.32.2. Same Bare-Speed artifact revision74559cdf34fbfc0b593de72d17e93f37fd4f9ea7; unchanged hashes, compiler/OS/hardware and effective options in environment.json. Hardware: M5 Max128GiB/macOS26.6.2. Synthetic model-free cases use explicit shapes/dtypes in the diagnostic, no template/image/prompt or context setting; model case uses raw IDs, seed0 and restored identical prompt snapshots.Durable report and instructions, native test, oracle probe. The test module is cfg(test), ignored by ordinary runs, and initially asserts/prints the affected behavior; passing this diagnostic means reproduction, not a fix. Convert only this issue's owned cases into correct-contract regressions during implementation. Shared vision test has separate case ownership listed in the report.
Immutable upstream change and latest test. The probe's independent expected values and controls are described above. Extra readbacks/synchronization and external memory samples make these correctness receipts unsuitable for speed claims.
Validation and review boundary
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings,make bundle,cargo test --all-featuresbefore every future commit, plus the owned native/oracle and product regressions above.Independent review (2026-10-03)
Reviewed against DS4Server
dcc4c51and MTPLX 2.12.29882703f3105363ddc37eca9f97aa09a1d387112. Note: evidence commitdcc4c51is still unpublished (mainis ahead oforigin/mainby one commit), so the repository links in this issue currently resolve to nothing.Defect claim: confirmed. In
decode_mtpthe round head (speculative.rs:722-737) takespendingor samples a fresh primary, pushes it intoresult.tokens, then breaks on stop ormax_tokenswithpending == None. The final commit (if let Some(token) = result.pending) therefore never forwards it. This is precisely upstream defect 2 of1a977b65.Proposed fix: correct, but broader than needed, and one gap.
pending_primary = int(primary)), so the existing final-pending commit extends trunk cache and committed MTP history by exactly that token. The native equivalent isresult.pending = Some(cycle.primary)in the round-head exit branch. Keep the broaderfinal_arrays/capture_sessionredesign out of scope unless a test shows it is needed.session_checkpoint(magicQWMTKV01) under this defect stay corrupt after the fix (key one token longer than state). Acceptance should require invalidating them, e.g. a checkpoint format/version bump, not only "persist/reopen the completed snapshot".tests/test_mtp_final_state_covers_committed_tokens.pyis the right template.Description corrected after independent review (2026-10-03)
Updated the issue body in response to your review. Changes:
All eight issue descriptions now state the user's contract explicitly: Qwen 3.8 Flash Next follows pinned MTPLX; DSv4 Flash and GLM 5.3 Flash retain DS4 parity. Shared helpers must preserve the DS4-backed callers with exact regression controls.
For renewed verification: Verify the minimal pending re-arm reaches the existing final commit once, compare all banked-state boundaries, and verify the proposed Qwen-only compatibility mechanism rejects a real pre-fix corrupt checkpoint.
This update changes tracker descriptions only, not production code, AGENTS.md or the committed raw receipts. The initial evidence report is historical where the corrected body says otherwise. Evidence commit
dcc4c51remains local/unpublished, with exact local paths and reproduction instructions in the body; no new test/performance result or implementation approval is claimed. Please record any remaining applicability/contract/acceptance gap before implementation.Implemented and verified in commit
a8cad8810060d22ee7ccac82c210ef40d4e08785(pushed to main).The round-head stop/token-limit exit now keeps the emitted primary pending, matching MTPLX 2.12.2
9882703f3105363ddc37eca9f97aa09a1d387112. The existing final commit advances trunk and committed MTP history exactly once. Qwen checkpoints now useQWMTKV02; old V1 files are rejected before tensor allocation, removed by the existing Generator incompatibility path, and rebuilt cold. No snapshot-owner redesign or DS4/GLM checkpoint/inference change was needed. Resident state is process-local and cannot survive the executable restart used for upgrades.Validation performed in isolation:
8cd38359515e9d30b191992cc477393ee59b5cac7b6d610aa5838e589ce02c64).Reviewed the complete diff against this issue, pinned MTPLX and DS4's separate committed-state/checkpoint behavior. No remaining acceptance gap found. Durable evidence and reproduction instructions: issue106-final-primary.