Implement the model's native one-layer MTP head and exact speculative decoding, including proposal generation, probability-ratio acceptance, rejection repair, cache commit, bounded history, and user-visible metrics.
Sequence
Step 6 of 7. Previous: #97. Next: #99. Start only after long-context rollback is correct; #99 optimizes the completed MTP path.
Implementation
Load and validate the separate MTP artifact produced by #93. MTP remains optional at runtime but is a required supported capability.
Implement the Qwen MTP input contract: pre-mixer four-stream trunk hidden state plus token embedding, normalization/projection, one full-attention QSA+MoE+HC layer, stream combination, and shared LM head.
Reuse DS4Server's GLM 5.3 MTP sampling, metrics, and checkpoint machinery where the model semantics match.
Implement exact probability-ratio acceptance and residual correction for official non-greedy sampling.
Capture verification-row GDN and PLE data, trim rejected QSA rows, and commit/replay only accepted recurrent rows. Retain a correct ordinary-forward repair fallback when capture is unavailable or incomplete.
Bound draft history, verify width, allocator caches, and temporary buffers in memory admission. Report drafted tokens, accepted tokens, verifier passes, acceptance depth, and verifier/repair timing.
Handle EOS inside a proposal block without retaining tokens or state beyond EOS.
Verification
Match MTP hidden states, logits, and proposal vectors against pinned references.
Test greedy and sampled decoding with rejection at every proposal position, EOS at every position, disabled-MTP parity, and deterministic seeded sampling.
Compare capture-commit with full repair at the logit, token, and complete cache-state levels.
Exercise multi-turn, cache-hit, long-context, checkpoint/resume, and long-generation history-reset paths.
Completion requirements
MTP preserves the target distribution and can be disabled without changing target-model output.
Rejected rows leave every cache and recurrent component identical to ordinary decoding.
Capture-commit and fallback repair are both correct.
Real server workloads show a wall-clock benefit before MTP becomes the default.
Non-goals
External draft models, prompt-copy speculation, vision, or unrelated speculative-decoding redesign.
## Area
Implement the model's native one-layer MTP head and exact speculative decoding, including proposal generation, probability-ratio acceptance, rejection repair, cache commit, bounded history, and user-visible metrics.
## Sequence
Step 6 of 7. Previous: #97. Next: #99. Start only after long-context rollback is correct; #99 optimizes the completed MTP path.
## Implementation
- Load and validate the separate MTP artifact produced by #93. MTP remains optional at runtime but is a required supported capability.
- Implement the Qwen MTP input contract: pre-mixer four-stream trunk hidden state plus token embedding, normalization/projection, one full-attention QSA+MoE+HC layer, stream combination, and shared LM head.
- Reuse DS4Server's GLM 5.3 MTP sampling, metrics, and checkpoint machinery where the model semantics match.
- Implement exact probability-ratio acceptance and residual correction for official non-greedy sampling.
- Capture verification-row GDN and PLE data, trim rejected QSA rows, and commit/replay only accepted recurrent rows. Retain a correct ordinary-forward repair fallback when capture is unavailable or incomplete.
- Bound draft history, verify width, allocator caches, and temporary buffers in memory admission. Report drafted tokens, accepted tokens, verifier passes, acceptance depth, and verifier/repair timing.
- Handle EOS inside a proposal block without retaining tokens or state beyond EOS.
## Verification
- Match MTP hidden states, logits, and proposal vectors against pinned references.
- Test greedy and sampled decoding with rejection at every proposal position, EOS at every position, disabled-MTP parity, and deterministic seeded sampling.
- Compare capture-commit with full repair at the logit, token, and complete cache-state levels.
- Exercise multi-turn, cache-hit, long-context, checkpoint/resume, and long-generation history-reset paths.
## Completion requirements
- MTP preserves the target distribution and can be disabled without changing target-model output.
- Rejected rows leave every cache and recurrent component identical to ordinary decoding.
- Capture-commit and fallback repair are both correct.
- Real server workloads show a wall-clock benefit before MTP becomes the default.
## Non-goals
- External draft models, prompt-copy speculation, vision, or unrelated speculative-decoding redesign.
hugo
added the idea label 2026-09-01 18:24:31 +00:00
Implemented in commit bf82df7 (Add native Qwen MTP speculation).
Implementation:
Added the complete Rust Qwen3.8 one-layer MTP graph: four-stream hidden and token projections, full QSA attention, routed and shared MoE, hyper-connection mixing, and the shared language head.
Added greedy and exact sampled speculation with DS4-compatible probability-ratio acceptance and max(0, p-q) residual correction.
Added bounded proposal generation, deterministic EOS truncation, verification-row GDN and PLE snapshots, rejected QSA and KV trimming, accepted-tail MTP commit, and a full ordinary-forward Rust replay fallback.
Added optional sidecar loading and admission accounting for the MTP artifact, attention cache, verifier snapshots, hidden/logit captures, and temporary state.
Added MTP checkpoint and resident-session state, disable parity, reset behavior, acceptance-depth and verifier/repair telemetry, and Qwen MTP controls/stats in the macOS UI.
Corrected the pinned MTP routed-expert contract to the artifact bytes: 4-bit affine weights with 32-value groups. Updated its representative digest, inventory digest, and embedded manifest trust anchor.
Review:
DS4 has no Qwen3.8 execution path, so the model graph is a new DS4Server feature. Shared sampling behavior was reconciled against DS4, including u*q <= p acceptance, residual repair, and zero/non-finite probability handling.
The implementation adds no C or Python code and modifies no Objective-C or Metal source.
cargo test --all-features: 237 passed, 27 fixture-dependent ignored
Pinned 105 GB artifact identity verification: passed for all 29 selected files
Pinned Apple M5 Max Metal test: passed; covers disabled-target parity, deterministic MTP proposal/logit/hidden digests, all four accepted depths, complete active target state, rejected cache trimming, ordinary-forward fallback, checkpoint/resume, metrics, and EOS with unchanged MTP state.
Implemented in commit bf82df7 (Add native Qwen MTP speculation).
Implementation:
- Added the complete Rust Qwen3.8 one-layer MTP graph: four-stream hidden and token projections, full QSA attention, routed and shared MoE, hyper-connection mixing, and the shared language head.
- Added greedy and exact sampled speculation with DS4-compatible probability-ratio acceptance and max(0, p-q) residual correction.
- Added bounded proposal generation, deterministic EOS truncation, verification-row GDN and PLE snapshots, rejected QSA and KV trimming, accepted-tail MTP commit, and a full ordinary-forward Rust replay fallback.
- Added optional sidecar loading and admission accounting for the MTP artifact, attention cache, verifier snapshots, hidden/logit captures, and temporary state.
- Added MTP checkpoint and resident-session state, disable parity, reset behavior, acceptance-depth and verifier/repair telemetry, and Qwen MTP controls/stats in the macOS UI.
- Corrected the pinned MTP routed-expert contract to the artifact bytes: 4-bit affine weights with 32-value groups. Updated its representative digest, inventory digest, and embedded manifest trust anchor.
Review:
- DS4 has no Qwen3.8 execution path, so the model graph is a new DS4Server feature. Shared sampling behavior was reconciled against DS4, including u*q <= p acceptance, residual repair, and zero/non-finite probability handling.
- The implementation adds no C or Python code and modifies no Objective-C or Metal source.
Verification:
- cargo fmt --all -- --check
- cargo clippy --all-targets --all-features -- -D warnings
- make bundle
- cargo test --all-features: 237 passed, 27 fixture-dependent ignored
- Pinned 105 GB artifact identity verification: passed for all 29 selected files
- Pinned Apple M5 Max Metal test: passed; covers disabled-target parity, deterministic MTP proposal/logit/hidden digests, all four accepted depths, complete active target state, rejected cache trimming, ordinary-forward fallback, checkpoint/resume, metrics, and EOS with unchanged MTP state.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Area
Implement the model's native one-layer MTP head and exact speculative decoding, including proposal generation, probability-ratio acceptance, rejection repair, cache commit, bounded history, and user-visible metrics.
Sequence
Step 6 of 7. Previous: #97. Next: #99. Start only after long-context rollback is correct; #99 optimizes the completed MTP path.
Implementation
Verification
Completion requirements
Non-goals
hugo referenced this issue2026-09-01 18:27:01 +00:00
Implemented in commit
bf82df7(Add native Qwen MTP speculation).Implementation:
Review:
Verification: