Add Qwen3.8 Flash Next native MTP speculation #98

New Issue

hugo · 2026-09-01T18:24:31Z

hugo commented

2026-09-01 18:24:31 +00:00

Area

Implement the model's native one-layer MTP head and exact speculative decoding, including proposal generation, probability-ratio acceptance, rejection repair, cache commit, bounded history, and user-visible metrics.

Sequence

Step 6 of 7. Previous: #97. Next: #99. Start only after long-context rollback is correct; #99 optimizes the completed MTP path.

Implementation

Load and validate the separate MTP artifact produced by #93. MTP remains optional at runtime but is a required supported capability.
Implement the Qwen MTP input contract: pre-mixer four-stream trunk hidden state plus token embedding, normalization/projection, one full-attention QSA+MoE+HC layer, stream combination, and shared LM head.
Reuse DS4Server's GLM 5.3 MTP sampling, metrics, and checkpoint machinery where the model semantics match.
Implement exact probability-ratio acceptance and residual correction for official non-greedy sampling.
Capture verification-row GDN and PLE data, trim rejected QSA rows, and commit/replay only accepted recurrent rows. Retain a correct ordinary-forward repair fallback when capture is unavailable or incomplete.
Bound draft history, verify width, allocator caches, and temporary buffers in memory admission. Report drafted tokens, accepted tokens, verifier passes, acceptance depth, and verifier/repair timing.
Handle EOS inside a proposal block without retaining tokens or state beyond EOS.

Verification

Match MTP hidden states, logits, and proposal vectors against pinned references.
Test greedy and sampled decoding with rejection at every proposal position, EOS at every position, disabled-MTP parity, and deterministic seeded sampling.
Compare capture-commit with full repair at the logit, token, and complete cache-state levels.
Exercise multi-turn, cache-hit, long-context, checkpoint/resume, and long-generation history-reset paths.

Completion requirements

MTP preserves the target distribution and can be disabled without changing target-model output.
Rejected rows leave every cache and recurrent component identical to ordinary decoding.
Capture-commit and fallback repair are both correct.
Real server workloads show a wall-clock benefit before MTP becomes the default.

Non-goals

External draft models, prompt-copy speculation, vision, or unrelated speculative-decoding redesign.

## Area Implement the model's native one-layer MTP head and exact speculative decoding, including proposal generation, probability-ratio acceptance, rejection repair, cache commit, bounded history, and user-visible metrics. ## Sequence Step 6 of 7. Previous: #97. Next: #99. Start only after long-context rollback is correct; #99 optimizes the completed MTP path. ## Implementation - Load and validate the separate MTP artifact produced by #93. MTP remains optional at runtime but is a required supported capability. - Implement the Qwen MTP input contract: pre-mixer four-stream trunk hidden state plus token embedding, normalization/projection, one full-attention QSA+MoE+HC layer, stream combination, and shared LM head. - Reuse DS4Server's GLM 5.3 MTP sampling, metrics, and checkpoint machinery where the model semantics match. - Implement exact probability-ratio acceptance and residual correction for official non-greedy sampling. - Capture verification-row GDN and PLE data, trim rejected QSA rows, and commit/replay only accepted recurrent rows. Retain a correct ordinary-forward repair fallback when capture is unavailable or incomplete. - Bound draft history, verify width, allocator caches, and temporary buffers in memory admission. Report drafted tokens, accepted tokens, verifier passes, acceptance depth, and verifier/repair timing. - Handle EOS inside a proposal block without retaining tokens or state beyond EOS. ## Verification - Match MTP hidden states, logits, and proposal vectors against pinned references. - Test greedy and sampled decoding with rejection at every proposal position, EOS at every position, disabled-MTP parity, and deterministic seeded sampling. - Compare capture-commit with full repair at the logit, token, and complete cache-state levels. - Exercise multi-turn, cache-hit, long-context, checkpoint/resume, and long-generation history-reset paths. ## Completion requirements - MTP preserves the target distribution and can be disabled without changing target-model output. - Rejected rows leave every cache and recurrent component identical to ordinary decoding. - Capture-commit and fallback repair are both correct. - Real server workloads show a wall-clock benefit before MTP becomes the default. ## Non-goals - External draft models, prompt-copy speculation, vision, or unrelated speculative-decoding redesign.

hugo added the idea label 2026-09-01 18:24:31 +00:00

~~hugo referenced this issue 2026-09-01 18:27:01 +00:00~~

Prepare exact Qwen3.8 Flash Next Bare Speed artifacts #93

hugo referenced this issue

2026-09-01 18:36:52 +00:00

Implement exact Qwen Sparse Attention and long-context state #97

hugo referenced this issue

2026-09-01 18:36:52 +00:00

Optimize and benchmark Qwen3.8 Flash Next on Apple silicon #99

hugo added enhancement and removed idea labels 2026-09-01 18:42:35 +00:00

hugo commented

2026-09-03 20:39:44 +00:00

Implemented in commit bf82df7 (Add native Qwen MTP speculation).

Implementation:

Added the complete Rust Qwen3.8 one-layer MTP graph: four-stream hidden and token projections, full QSA attention, routed and shared MoE, hyper-connection mixing, and the shared language head.
Added greedy and exact sampled speculation with DS4-compatible probability-ratio acceptance and max(0, p-q) residual correction.
Added bounded proposal generation, deterministic EOS truncation, verification-row GDN and PLE snapshots, rejected QSA and KV trimming, accepted-tail MTP commit, and a full ordinary-forward Rust replay fallback.
Added optional sidecar loading and admission accounting for the MTP artifact, attention cache, verifier snapshots, hidden/logit captures, and temporary state.
Added MTP checkpoint and resident-session state, disable parity, reset behavior, acceptance-depth and verifier/repair telemetry, and Qwen MTP controls/stats in the macOS UI.
Corrected the pinned MTP routed-expert contract to the artifact bytes: 4-bit affine weights with 32-value groups. Updated its representative digest, inventory digest, and embedded manifest trust anchor.

Review:

DS4 has no Qwen3.8 execution path, so the model graph is a new DS4Server feature. Shared sampling behavior was reconciled against DS4, including u*q <= p acceptance, residual repair, and zero/non-finite probability handling.
The implementation adds no C or Python code and modifies no Objective-C or Metal source.

Verification:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
make bundle
cargo test --all-features: 237 passed, 27 fixture-dependent ignored
Pinned 105 GB artifact identity verification: passed for all 29 selected files
Pinned Apple M5 Max Metal test: passed; covers disabled-target parity, deterministic MTP proposal/logit/hidden digests, all four accepted depths, complete active target state, rejected cache trimming, ordinary-forward fallback, checkpoint/resume, metrics, and EOS with unchanged MTP state.

Implemented in commit bf82df7 (Add native Qwen MTP speculation). Implementation: - Added the complete Rust Qwen3.8 one-layer MTP graph: four-stream hidden and token projections, full QSA attention, routed and shared MoE, hyper-connection mixing, and the shared language head. - Added greedy and exact sampled speculation with DS4-compatible probability-ratio acceptance and max(0, p-q) residual correction. - Added bounded proposal generation, deterministic EOS truncation, verification-row GDN and PLE snapshots, rejected QSA and KV trimming, accepted-tail MTP commit, and a full ordinary-forward Rust replay fallback. - Added optional sidecar loading and admission accounting for the MTP artifact, attention cache, verifier snapshots, hidden/logit captures, and temporary state. - Added MTP checkpoint and resident-session state, disable parity, reset behavior, acceptance-depth and verifier/repair telemetry, and Qwen MTP controls/stats in the macOS UI. - Corrected the pinned MTP routed-expert contract to the artifact bytes: 4-bit affine weights with 32-value groups. Updated its representative digest, inventory digest, and embedded manifest trust anchor. Review: - DS4 has no Qwen3.8 execution path, so the model graph is a new DS4Server feature. Shared sampling behavior was reconciled against DS4, including u*q <= p acceptance, residual repair, and zero/non-finite probability handling. - The implementation adds no C or Python code and modifies no Objective-C or Metal source. Verification: - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - make bundle - cargo test --all-features: 237 passed, 27 fixture-dependent ignored - Pinned 105 GB artifact identity verification: passed for all 29 selected files - Pinned Apple M5 Max Metal test: passed; covers disabled-target parity, deterministic MTP proposal/logit/hidden digests, all four accepted depths, complete active target state, rejected cache trimming, ordinary-forward fallback, checkpoint/resume, metrics, and EOS with unchanged MTP state.

hugo closed this issue

2026-09-03 20:39:48 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#98