Implement the Qwen3.8 Flash Next Rust and Metal text core #95

New Issue

hugo · 2026-09-01T18:24:30Z

hugo commented

2026-09-01 18:24:30 +00:00

Area

Implement the Qwen3.8 Flash Next transformer core in Rust and Metal: embeddings, hyper-connections, Gated DeltaNet layers, dense full-attention behavior within the sparse budget, MoE, final projection, and recurrent state ownership.

Sequence

Step 3 of 7. Previous: #94. Next: #96. Start only after the exact loader contract in #94 is complete; #96 builds on this executor.

Implementation

Add a dedicated qwen4exp Rust executor and dispatch it only after #94 validation succeeds.
Reuse the GLM 5.3 BF16 bridge, recurrent/conv cache ownership, checkpoint plumbing, command-buffer reuse, MoE scheduling, and prefill/small-batch infrastructure where semantics match.
Implement the Qwen-specific 48-layer 3:1 schedule, 16-key/48-value-head GDN geometry, convolution width 4, fp32 recurrent state, sigmoid output gate, and exact L2-normalization epsilon placement.
Implement four-stream rank-320 low-rank hyper-connection mix/combine, zero-centered norms, and the absence of a conventional output norm exactly as specified.
Implement 512-expert top-10 routing plus shared expert with the Bare Speed tensor formats. Reuse existing quantized matmul paths before adding kernels.
Implement Qwen RoPE and an exact dense 24Q/2KV full-attention path for histories where the 2,048-token sparse budget covers all available tokens. Sparse selection beyond that budget belongs to #97.
Persist every GDN convolution and recurrent component in session checkpoints with versioned validation.
Keep orchestration in Rust and native work in .metal plus the existing small Objective-C bridge.

Verification

Add primitive vectors for zero-centered norms, hyper-connection mix/combine, GDN projection/conv/recurrence/output gate, router/top-10 selection, shared+routed MoE, RoPE, and dense attention.
Compare layer-boundary hidden states and logits against pinned reference vectors.
Prove prefill/decode equivalence across chunk sizes and checkpoint/resume equivalence for recurrent state.
Run existing DeepSeek and GLM regression vectors to prove unchanged output.

Completion requirements

Every transformer-core primitive matches reference vectors within the agreed dtype tolerance.
Dense short-context forward passes produce the expected logits with explicit errors for the still-missing PLE and sparse-QSA stages.
GDN state survives save/restore without drift.
No other model path changes behavior.

Non-goals

PLE lookup, sparse attention beyond the dense budget, MTP, vision, or final optimization.

## Area Implement the Qwen3.8 Flash Next transformer core in Rust and Metal: embeddings, hyper-connections, Gated DeltaNet layers, dense full-attention behavior within the sparse budget, MoE, final projection, and recurrent state ownership. ## Sequence Step 3 of 7. Previous: #94. Next: #96. Start only after the exact loader contract in #94 is complete; #96 builds on this executor. ## Implementation - Add a dedicated `qwen4exp` Rust executor and dispatch it only after #94 validation succeeds. - Reuse the GLM 5.3 BF16 bridge, recurrent/conv cache ownership, checkpoint plumbing, command-buffer reuse, MoE scheduling, and prefill/small-batch infrastructure where semantics match. - Implement the Qwen-specific 48-layer 3:1 schedule, 16-key/48-value-head GDN geometry, convolution width 4, fp32 recurrent state, sigmoid output gate, and exact L2-normalization epsilon placement. - Implement four-stream rank-320 low-rank hyper-connection mix/combine, zero-centered norms, and the absence of a conventional output norm exactly as specified. - Implement 512-expert top-10 routing plus shared expert with the Bare Speed tensor formats. Reuse existing quantized matmul paths before adding kernels. - Implement Qwen RoPE and an exact dense 24Q/2KV full-attention path for histories where the 2,048-token sparse budget covers all available tokens. Sparse selection beyond that budget belongs to #97. - Persist every GDN convolution and recurrent component in session checkpoints with versioned validation. - Keep orchestration in Rust and native work in `.metal` plus the existing small Objective-C bridge. ## Verification - Add primitive vectors for zero-centered norms, hyper-connection mix/combine, GDN projection/conv/recurrence/output gate, router/top-10 selection, shared+routed MoE, RoPE, and dense attention. - Compare layer-boundary hidden states and logits against pinned reference vectors. - Prove prefill/decode equivalence across chunk sizes and checkpoint/resume equivalence for recurrent state. - Run existing DeepSeek and GLM regression vectors to prove unchanged output. ## Completion requirements - Every transformer-core primitive matches reference vectors within the agreed dtype tolerance. - Dense short-context forward passes produce the expected logits with explicit errors for the still-missing PLE and sparse-QSA stages. - GDN state survives save/restore without drift. - No other model path changes behavior. ## Non-goals - PLE lookup, sparse attention beyond the dense budget, MTP, vision, or final optimization.

hugo added the idea label 2026-09-01 18:24:30 +00:00

~~hugo referenced this issue 2026-09-01 18:27:01 +00:00~~

Prepare exact Qwen3.8 Flash Next Bare Speed artifacts #93

hugo referenced this issue

2026-09-01 18:36:51 +00:00

Integrate Qwen3.8 Flash Next loading and tokenizer behavior #94

hugo referenced this issue

2026-09-01 18:36:51 +00:00

Stream Qwen3.8 Flash Next PLE from mapped storage #96

hugo added enhancement and removed idea labels 2026-09-01 18:42:06 +00:00

hugo commented

2026-09-03 18:32:28 +00:00

Implemented in c414640 (Implement Qwen text core).

The server now opens the frozen Qwen safetensors set through a dedicated Rust model/executor path and runs the 48-layer 3:1 hybrid text graph with MLX affine Q2/Q4/Q8 and BF16 projection kernels, four-stream rank-320 hyper-connections, BF16 width-4 convolution history, fp32 Gated DeltaNet recurrence, 24Q/2KV dense causal attention through the 2,048-token budget, top-10-of-512 routed experts plus the shared expert, Qwen RoPE, final hyper mixer, and LM projection. Rust owns model maps, graph scheduling, session state, resident-state swaps, and versioned checkpoints; native additions are limited to Metal kernels and the existing Objective-C dispatch bridge. Missing layer-2 PLE and over-budget sparse QSA fail before unsafe or partial execution and remain explicitly assigned to #96 and #97.

Verification completed in isolation on Apple M5 Max against the pinned 105 GB artifact set:

Metal primitive vectors cover affine dequantization, zero-centered grouped RMS, hyper mix/combine, convolution, fp32 delta recurrence, sigmoid output gate, routed/shared MoE, top-10 normalization, Qwen RoPE, BF16 KV storage, and dense causal attention.
The pinned layer-0 boundary and projected logits match frozen values.
Direct and prefill entry points return the identical missing-PLE boundary without mutating session state.
A computed recurrent continuation produces identical logits, BF16 convolution bytes, and fp32 recurrent bytes after checkpoint/resume.
The sparse boundary returns the explicit #97 error.
The installed MLX runtime was used only as an external packing oracle; no Python or C implementation code was added.

Required gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (233 passed, 22 ignored, plus 4/4 artifact-tool tests). The initial full test run encountered the unrelated randomized dev_brain YAML fixture flake; its isolated rerun and the complete rerun both passed. DS4 has no Qwen3.8 path, so this is a new DS4Server feature while preserving existing DeepSeek and GLM behavior.

Implemented in c414640 (Implement Qwen text core). The server now opens the frozen Qwen safetensors set through a dedicated Rust model/executor path and runs the 48-layer 3:1 hybrid text graph with MLX affine Q2/Q4/Q8 and BF16 projection kernels, four-stream rank-320 hyper-connections, BF16 width-4 convolution history, fp32 Gated DeltaNet recurrence, 24Q/2KV dense causal attention through the 2,048-token budget, top-10-of-512 routed experts plus the shared expert, Qwen RoPE, final hyper mixer, and LM projection. Rust owns model maps, graph scheduling, session state, resident-state swaps, and versioned checkpoints; native additions are limited to Metal kernels and the existing Objective-C dispatch bridge. Missing layer-2 PLE and over-budget sparse QSA fail before unsafe or partial execution and remain explicitly assigned to #96 and #97. Verification completed in isolation on Apple M5 Max against the pinned 105 GB artifact set: - Metal primitive vectors cover affine dequantization, zero-centered grouped RMS, hyper mix/combine, convolution, fp32 delta recurrence, sigmoid output gate, routed/shared MoE, top-10 normalization, Qwen RoPE, BF16 KV storage, and dense causal attention. - The pinned layer-0 boundary and projected logits match frozen values. - Direct and prefill entry points return the identical missing-PLE boundary without mutating session state. - A computed recurrent continuation produces identical logits, BF16 convolution bytes, and fp32 recurrent bytes after checkpoint/resume. - The sparse boundary returns the explicit #97 error. - The installed MLX runtime was used only as an external packing oracle; no Python or C implementation code was added. Required gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (233 passed, 22 ignored, plus 4/4 artifact-tool tests). The initial full test run encountered the unrelated randomized dev_brain YAML fixture flake; its isolated rerun and the complete rerun both passed. DS4 has no Qwen3.8 path, so this is a new DS4Server feature while preserving existing DeepSeek and GLM behavior.

hugo closed this issue

2026-09-03 18:33:05 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#95