Stream Qwen3.8 Flash Next PLE from mapped storage #96

New Issue

hugo · 2026-09-01T18:24:30Z

hugo commented

2026-09-01 18:24:30 +00:00

Area

Implement the mandatory layer-2 PLE/n-gram embedding end to end, including exact hashing, mapped storage, row gathering, bounded caching, session state, and short-context generation parity.

Sequence

Step 4 of 7. Previous: #95. Next: #97. Start only after the core executor is complete; #97 starts after PLE state and generation parity are stable.

Implementation

Implement the official 3-gram hash contract in Rust with exact 64-bit multipliers, 20,000,000 base vocabulary, 128-way layout, offsets, per-head vocabulary sizes, EOS reset, and the PLE convolution/norm path.
Bind the verified PLE artifact from #93 as read-only mapped storage and gather only required rows. Reuse DS4Server's existing mmap/pread/cache and memory-accounting facilities before introducing new storage machinery.
Stage token IDs and PLE row indices on the host before GPU submission so lookup does not require a graph synchronization round trip.
Add a bounded hot-row cache and safe prefetch only when profiling proves a benefit. Capacity and page-cache effects must be visible to memory admission and pressure handling.
Save and restore n-gram history plus PLE convolution state in the session checkpoint.
Complete end-to-end autoregressive generation at contexts where dense attention is still exact.

Verification

Add hash/index golden vectors covering BOS/EOS, repeated n-grams, chunk boundaries, multi-turn reset, and every PLE head/shard boundary.
Match row gathering, dequantization, convolution, normalization, layer output, logits, and greedy token hashes against pinned references.
Prove identical output for cold/warm page cache, cache disabled/enabled, different prefill chunk sizes, and checkpoint/resume.
Measure resident core, mapped bytes, hot cache, observed page-cache residency, prefill peak, and swap pressure separately on the 128 GB M5 Max.

Completion requirements

PLE outputs and short-context generated token hashes match the reference.
A 128 GB M5 Max runs the resident core plus streamed PLE without unaccounted swap pressure.
Missing, truncated, or mismatched PLE data fails before generation.
Caching and prefetching cannot change model output.

Non-goals

Sparse QSA beyond 2,048 tokens, MTP, full-context performance, or vision.

## Area Implement the mandatory layer-2 PLE/n-gram embedding end to end, including exact hashing, mapped storage, row gathering, bounded caching, session state, and short-context generation parity. ## Sequence Step 4 of 7. Previous: #95. Next: #97. Start only after the core executor is complete; #97 starts after PLE state and generation parity are stable. ## Implementation - Implement the official 3-gram hash contract in Rust with exact 64-bit multipliers, 20,000,000 base vocabulary, 128-way layout, offsets, per-head vocabulary sizes, EOS reset, and the PLE convolution/norm path. - Bind the verified PLE artifact from #93 as read-only mapped storage and gather only required rows. Reuse DS4Server's existing mmap/pread/cache and memory-accounting facilities before introducing new storage machinery. - Stage token IDs and PLE row indices on the host before GPU submission so lookup does not require a graph synchronization round trip. - Add a bounded hot-row cache and safe prefetch only when profiling proves a benefit. Capacity and page-cache effects must be visible to memory admission and pressure handling. - Save and restore n-gram history plus PLE convolution state in the session checkpoint. - Complete end-to-end autoregressive generation at contexts where dense attention is still exact. ## Verification - Add hash/index golden vectors covering BOS/EOS, repeated n-grams, chunk boundaries, multi-turn reset, and every PLE head/shard boundary. - Match row gathering, dequantization, convolution, normalization, layer output, logits, and greedy token hashes against pinned references. - Prove identical output for cold/warm page cache, cache disabled/enabled, different prefill chunk sizes, and checkpoint/resume. - Measure resident core, mapped bytes, hot cache, observed page-cache residency, prefill peak, and swap pressure separately on the 128 GB M5 Max. ## Completion requirements - PLE outputs and short-context generated token hashes match the reference. - A 128 GB M5 Max runs the resident core plus streamed PLE without unaccounted swap pressure. - Missing, truncated, or mismatched PLE data fails before generation. - Caching and prefetching cannot change model output. ## Non-goals - Sparse QSA beyond 2,048 tokens, MTP, full-context performance, or vision.

hugo added the idea label 2026-09-01 18:24:30 +00:00

~~hugo referenced this issue 2026-09-01 18:27:01 +00:00~~

Prepare exact Qwen3.8 Flash Next Bare Speed artifacts #93

hugo referenced this issue

2026-09-01 18:36:51 +00:00

Implement the Qwen3.8 Flash Next Rust and Metal text core #95

hugo referenced this issue

2026-09-01 18:36:52 +00:00

Implement exact Qwen Sparse Attention and long-context state #97

hugo added enhancement and removed idea labels 2026-09-01 18:42:16 +00:00

hugo referenced this issue

2026-09-03 18:32:28 +00:00

Implement the Qwen3.8 Flash Next Rust and Metal text core #95

hugo commented

2026-09-03 19:13:12 +00:00

Implemented fully in commit 87ccf67 (pushed to origin/main).

Implementation:

Added the official Qwen3.8 Flash Next 2/3-gram PLE hash contract in Rust, including the exact splitmix multipliers, 20M prime head ranges, offsets, 128-row padding contract, host-side row indexing, and EOS history reset.
Mapped the verified 32,000,154,008-byte ngram sidecar read-only and gather only the sixteen required 100-byte quantized rows per token before GPU submission.
Added native Metal PLE dequantization, key/value projection flow, zero-centered normalization, signed-sqrt sigmoid gating, dilation-3 convolution, and layer-2 hyper-stream injection. Rust owns all graph orchestration and state; only kernels and the existing Objective-C Metal binding are non-Rust.
Fixed the shared Qwen Metal binding for arbitrary safetensors data alignment. Odd-byte tensor offsets are now carried explicitly into the kernels instead of being silently rounded down by Metal.
Added PLE n-gram history and the full 184,320-byte BF16 convolution state to reset, resident-session swapping, checkpoint save/load, and memory admission. Checkpoint format is versioned accordingly.
Completed exact dense autoregressive execution through 2,048 tokens; the next-token boundary fails explicitly into #97 QSA work.

Cache/prefetch decision:

A release microbenchmark over 200,000 repeated accesses measured direct mapped row gathers at 4.36 ms and the proposed 4,096-row LRU at 16.12 ms. Because the cache was 3.7x slower, no hot-row cache or prefetch was retained, exactly as required by the profiling gate. This also leaves no extra cache residency or cache-dependent output mode.

Verification:

Official hash/index goldens cover all 16 head ranges, repeated n-grams, EOS/BOS reset, and chunk groupings of 1, 2, and 5 tokens.
Row gathering rejects truncation and copies only the selected packed/scales/biases row.
Metal reference vectors cover deliberately odd safetensors offsets plus affine dequantization, PLE dequantization, signed-sqrt gating, normalization, dilation-3 convolution, GDN, hyper connections, MoE routing, RoPE, and dense attention.
The pinned 105 GB artifact test passes, including complete metadata/checksum validation and fail-closed artifact boundaries.
The full 48-layer test pins logits SHA-256 89f485fdc9d6c490f9821c3f7c4b20281094947b05325aa5652cdfa43e36a456, samples [6.406557, 2.0828183, -3.0580456, -0.1210103], and greedy token 89648. It also verifies checkpoint/resume state, reset/replay parity, EOS reset, and the #97 boundary.
M5 Max measurements: planned resident core 71,742,682,599 bytes; mapped PLE 32,000,154,008 bytes; KV/recurrent at full context 6,558,093,312 bytes; prefill transient 27,262,976 bytes. Observed mapped residency was 67,535,148,793 core bytes and 32,000,154,008 PLE bytes. The isolated run peaked at 1,071,235,072-byte RSS, reported zero process swaps, and system swap usage was unchanged at 1,565.56 MiB before/after.
Mandatory gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (235 main tests + 4 artifact-tool tests passed; 22 documented hardware/browser fixtures ignored).

DS4 review:
DS4 has no Qwen3.8/PLE implementation to mirror. This is therefore a new DS4Server feature, reviewed against the official Qwen model behavior and the issue contract.

Implemented fully in commit 87ccf67 (pushed to origin/main). Implementation: - Added the official Qwen3.8 Flash Next 2/3-gram PLE hash contract in Rust, including the exact splitmix multipliers, 20M prime head ranges, offsets, 128-row padding contract, host-side row indexing, and EOS history reset. - Mapped the verified 32,000,154,008-byte ngram sidecar read-only and gather only the sixteen required 100-byte quantized rows per token before GPU submission. - Added native Metal PLE dequantization, key/value projection flow, zero-centered normalization, signed-sqrt sigmoid gating, dilation-3 convolution, and layer-2 hyper-stream injection. Rust owns all graph orchestration and state; only kernels and the existing Objective-C Metal binding are non-Rust. - Fixed the shared Qwen Metal binding for arbitrary safetensors data alignment. Odd-byte tensor offsets are now carried explicitly into the kernels instead of being silently rounded down by Metal. - Added PLE n-gram history and the full 184,320-byte BF16 convolution state to reset, resident-session swapping, checkpoint save/load, and memory admission. Checkpoint format is versioned accordingly. - Completed exact dense autoregressive execution through 2,048 tokens; the next-token boundary fails explicitly into #97 QSA work. Cache/prefetch decision: - A release microbenchmark over 200,000 repeated accesses measured direct mapped row gathers at 4.36 ms and the proposed 4,096-row LRU at 16.12 ms. Because the cache was 3.7x slower, no hot-row cache or prefetch was retained, exactly as required by the profiling gate. This also leaves no extra cache residency or cache-dependent output mode. Verification: - Official hash/index goldens cover all 16 head ranges, repeated n-grams, EOS/BOS reset, and chunk groupings of 1, 2, and 5 tokens. - Row gathering rejects truncation and copies only the selected packed/scales/biases row. - Metal reference vectors cover deliberately odd safetensors offsets plus affine dequantization, PLE dequantization, signed-sqrt gating, normalization, dilation-3 convolution, GDN, hyper connections, MoE routing, RoPE, and dense attention. - The pinned 105 GB artifact test passes, including complete metadata/checksum validation and fail-closed artifact boundaries. - The full 48-layer test pins logits SHA-256 89f485fdc9d6c490f9821c3f7c4b20281094947b05325aa5652cdfa43e36a456, samples [6.406557, 2.0828183, -3.0580456, -0.1210103], and greedy token 89648. It also verifies checkpoint/resume state, reset/replay parity, EOS reset, and the #97 boundary. - M5 Max measurements: planned resident core 71,742,682,599 bytes; mapped PLE 32,000,154,008 bytes; KV/recurrent at full context 6,558,093,312 bytes; prefill transient 27,262,976 bytes. Observed mapped residency was 67,535,148,793 core bytes and 32,000,154,008 PLE bytes. The isolated run peaked at 1,071,235,072-byte RSS, reported zero process swaps, and system swap usage was unchanged at 1,565.56 MiB before/after. - Mandatory gates passed: cargo fmt --all -- --check; cargo clippy --all-targets --all-features -- -D warnings; make bundle; cargo test --all-features (235 main tests + 4 artifact-tool tests passed; 22 documented hardware/browser fixtures ignored). DS4 review: DS4 has no Qwen3.8/PLE implementation to mirror. This is therefore a new DS4Server feature, reviewed against the official Qwen model behavior and the issue contract.

hugo closed this issue

2026-09-03 19:13:16 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#96