Optimize and benchmark Qwen3.8 Flash Next on Apple silicon #99

New Issue

hugo · 2026-09-01T18:24:31Z

hugo commented

2026-09-01 18:24:31 +00:00

Area

Optimize the completed Qwen3.8 Flash Next path on the user's M5 Max MacBook Pro, meet the same-model MTPLX target, publish parity/performance receipts, and enable the model's production defaults.

Sequence

Step 7 of 7. Previous: #98. This is the final issue. Start only after correctness, long-context state, and MTP are complete.

Baseline contract

Run mtplx serve and DS4Server on the same M5 Max MacBook Pro with the exact pinned Bare Speed model/precision map, single stream, fans verified at maximum, identical prompts/context, official sampling at temperature 1.0/top-p 0.95/top-k 20, and controlled thermal/load order.
Capture same-process three-run medians for prefill, first token, plain AR, and default MTP. Preserve prompt and generated token IDs/hashes so speed cannot be gained by changing behavior.
Retain the pinned DS4 parity baseline wherever the DS4 Qwen path is available.

Optimization order

First reduce Rust/bridge scheduling overhead through pipeline, command-buffer, and stable-allocation reuse plus contiguous GDN-layer scheduling.
Profile and A/B fused GDN input projection, conv+SiLU+L2 normalization, single-row decay/delta recurrence/gated norm, and verify-width conv/norm.
Profile and A/B fused hyper-connection read and merged MoE gate/up projections.
Profile and A/B sparse-QSA row gathering and large-prefill kernels with capability-specific fallbacks.
Tune MTP capture-commit, proposal depth, verifier width, and PLE hot-cache/prefetch using complete wall-clock receipts.
Do not implement MTPLX experiments already measured flat/slower—custom MoE verify, fused GDN output projection, or async AR—unless a DS4Server profile proves the corresponding bottleneck.

Performance target

The primary target is the exact MTPLX Bare Speed pack. Its public same-task M5 Max result is 47.0 tok/s plain AR and 75.9 tok/s MTP; MTPLX 2.10 also reports approximately 61 tok/s AR and 63–76 tok/s MTP depending on workload.

Default-MTP sampled decode must reach at least 60 tok/s on the user's machine.
DS4Server must remain within 5% of the freshly measured same-machine MTPLX median on the matched coding workload; the published 75.9 tok/s MTP result is the goal.
Plain AR and prefill must separately remain within 5% of the matched MTPLX baselines.
DS4Server must remain within 5% of the DS4 oracle where that oracle is available, with identical model behavior.

Verification and receipts

Publish raw CSV, commands, hardware/OS, artifact hashes, prompts, contexts, sampler, cold/warm state, token hashes, prefill/first-token/steady throughput, p50/p95 latency, peak memory, PLE-cache statistics, and MTP acceptance/timing.
Measure short context plus 16K, 64K, 131K, and the admitted native maximum.
Require A/B/A or B/A/B evidence and exact fallback for every promoted fast path.
Run the full existing model regression suite, bundle gate, and Qwen parity/performance gates before enabling production defaults.

Completion requirements

The exact Bare Speed model is selectable and runs with correct official sampling, streamed PLE, long-context state, checkpoint/resume, and default MTP.
Default-MTP decode reaches at least 60 tok/s and stays within 5% of the matched MTPLX result on the user's M5 Max MacBook Pro.
The report accounts for any gap to 75.9 tok/s and demonstrates matched model data, quantization, workload, and memory policy.
Every enabled optimization wins its measured workload, preserves tokens/distribution, and falls back safely.
All parity, regression, packaging, and release gates pass.

Non-goals

Performance promises on non-M5 or unmatched artifacts/workloads, benchmark-only defaults, MLX/Python runtime integration, CUDA/ROCm, vision, or context beyond 262,144.

## Area Optimize the completed Qwen3.8 Flash Next path on the user's M5 Max MacBook Pro, meet the same-model MTPLX target, publish parity/performance receipts, and enable the model's production defaults. ## Sequence Step 7 of 7. Previous: #98. This is the final issue. Start only after correctness, long-context state, and MTP are complete. ## Baseline contract - Run `mtplx serve` and DS4Server on the same M5 Max MacBook Pro with the exact pinned Bare Speed model/precision map, single stream, fans verified at maximum, identical prompts/context, official sampling at temperature 1.0/top-p 0.95/top-k 20, and controlled thermal/load order. - Capture same-process three-run medians for prefill, first token, plain AR, and default MTP. Preserve prompt and generated token IDs/hashes so speed cannot be gained by changing behavior. - Retain the pinned DS4 parity baseline wherever the DS4 Qwen path is available. ## Optimization order - First reduce Rust/bridge scheduling overhead through pipeline, command-buffer, and stable-allocation reuse plus contiguous GDN-layer scheduling. - Profile and A/B fused GDN input projection, conv+SiLU+L2 normalization, single-row decay/delta recurrence/gated norm, and verify-width conv/norm. - Profile and A/B fused hyper-connection read and merged MoE gate/up projections. - Profile and A/B sparse-QSA row gathering and large-prefill kernels with capability-specific fallbacks. - Tune MTP capture-commit, proposal depth, verifier width, and PLE hot-cache/prefetch using complete wall-clock receipts. - Do not implement MTPLX experiments already measured flat/slower—custom MoE verify, fused GDN output projection, or async AR—unless a DS4Server profile proves the corresponding bottleneck. ## Performance target The primary target is the exact MTPLX Bare Speed pack. Its public same-task M5 Max result is 47.0 tok/s plain AR and 75.9 tok/s MTP; MTPLX 2.10 also reports approximately 61 tok/s AR and 63–76 tok/s MTP depending on workload. - Default-MTP sampled decode must reach at least 60 tok/s on the user's machine. - DS4Server must remain within 5% of the freshly measured same-machine MTPLX median on the matched coding workload; the published 75.9 tok/s MTP result is the goal. - Plain AR and prefill must separately remain within 5% of the matched MTPLX baselines. - DS4Server must remain within 5% of the DS4 oracle where that oracle is available, with identical model behavior. ## Verification and receipts - Publish raw CSV, commands, hardware/OS, artifact hashes, prompts, contexts, sampler, cold/warm state, token hashes, prefill/first-token/steady throughput, p50/p95 latency, peak memory, PLE-cache statistics, and MTP acceptance/timing. - Measure short context plus 16K, 64K, 131K, and the admitted native maximum. - Require A/B/A or B/A/B evidence and exact fallback for every promoted fast path. - Run the full existing model regression suite, bundle gate, and Qwen parity/performance gates before enabling production defaults. ## Completion requirements - The exact Bare Speed model is selectable and runs with correct official sampling, streamed PLE, long-context state, checkpoint/resume, and default MTP. - Default-MTP decode reaches at least 60 tok/s and stays within 5% of the matched MTPLX result on the user's M5 Max MacBook Pro. - The report accounts for any gap to 75.9 tok/s and demonstrates matched model data, quantization, workload, and memory policy. - Every enabled optimization wins its measured workload, preserves tokens/distribution, and falls back safely. - All parity, regression, packaging, and release gates pass. ## Non-goals - Performance promises on non-M5 or unmatched artifacts/workloads, benchmark-only defaults, MLX/Python runtime integration, CUDA/ROCm, vision, or context beyond 262,144.

hugo added the idea label 2026-09-01 18:24:31 +00:00

~~hugo referenced this issue 2026-09-01 18:27:01 +00:00~~

Prepare exact Qwen3.8 Flash Next Bare Speed artifacts #93

hugo referenced this issue

2026-09-01 18:36:52 +00:00

Add Qwen3.8 Flash Next native MTP speculation #98

hugo added enhancement and removed idea labels 2026-09-01 18:42:43 +00:00

hugo commented

2026-09-04 21:28:11 +00:00

Implemented and delivered in commit bd63538 (pushed to origin/main).

The Qwen3.8 Flash Next path now follows the pinned MTPLX/MLX execution contract end to end: exact affine QMV/QMM and BF16 rounding, fused GDN input/conv/recurrent/verifier capture, row-owned router and merged gate/up, MTPLX qsa_flash_skip long-context attention, streamed group-32 PLE with bounded hot cache and telemetry, NumPy-compatible PCG64 plus official temperature 1.0/top-p 0.95/top-k 20 sampling, exact MTP accept/reject/cache repair and full-accept bonus, stable allocation/pipeline reuse, and device-side sparse sampling. Experimental NAX, custom verifier MoE, scalar QMM, altered Metal math, and shortened recurrence paths were removed after oracle failures or lack of a measured win. MTP is now the Qwen production default while an explicit user opt-out remains authoritative. Non-Qwen sampling remains on the DS4 behavior.

Measured on Mac17,7 / Apple M5 Max / 128 GiB / macOS 26.6.2. The matched 4096-context, 26-prompt-token, 64-output-token acceptance sequence reached a 60.825779 tok/s three-run MTP median with stable token SHA-256 46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72, 44 accepted of 60 drafts, and 20 verifier passes. A later confirmation sequence under the then-current load measured 58.019436 tok/s with the identical token hash. Both are within 5% of the fresh matched MTPLX median 59.751170 tok/s. Plain AR median was 50.671077 tok/s. Peak MTP RSS is 52.61 GB, down from the original 102.65 GB path. Long-context oracle coverage passed at 2K, 16K, 64K, 131K, and 262K.

The public 75.9 tok/s figure is not the matched workload result. The current installed MTPLX package/source combination later refused the Flash Next pack because of source-package drift, so no invalid fresh AR number was substituted. Fan boost was also unavailable because neither supported fan utility is installed; both engines were compared under the same no-boost constraint.

Receipts: docs/QWEN38_PERFORMANCE.md and docs/qwen38-issue-99.csv.

Verification passed:

exact MTP token/cache restoration oracle
exact MLX QMV/QMM reference comparison
Qwen long-context logical-depth oracle
release three-run AR/MTP performance gates
cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
make bundle
cargo test --all-features: 239 main tests plus 4 artifact-tool tests passed; 37 declared hardware/artifact tests ignored

Implemented and delivered in commit bd63538 (pushed to origin/main). The Qwen3.8 Flash Next path now follows the pinned MTPLX/MLX execution contract end to end: exact affine QMV/QMM and BF16 rounding, fused GDN input/conv/recurrent/verifier capture, row-owned router and merged gate/up, MTPLX qsa_flash_skip long-context attention, streamed group-32 PLE with bounded hot cache and telemetry, NumPy-compatible PCG64 plus official temperature 1.0/top-p 0.95/top-k 20 sampling, exact MTP accept/reject/cache repair and full-accept bonus, stable allocation/pipeline reuse, and device-side sparse sampling. Experimental NAX, custom verifier MoE, scalar QMM, altered Metal math, and shortened recurrence paths were removed after oracle failures or lack of a measured win. MTP is now the Qwen production default while an explicit user opt-out remains authoritative. Non-Qwen sampling remains on the DS4 behavior. Measured on Mac17,7 / Apple M5 Max / 128 GiB / macOS 26.6.2. The matched 4096-context, 26-prompt-token, 64-output-token acceptance sequence reached a 60.825779 tok/s three-run MTP median with stable token SHA-256 46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72, 44 accepted of 60 drafts, and 20 verifier passes. A later confirmation sequence under the then-current load measured 58.019436 tok/s with the identical token hash. Both are within 5% of the fresh matched MTPLX median 59.751170 tok/s. Plain AR median was 50.671077 tok/s. Peak MTP RSS is 52.61 GB, down from the original 102.65 GB path. Long-context oracle coverage passed at 2K, 16K, 64K, 131K, and 262K. The public 75.9 tok/s figure is not the matched workload result. The current installed MTPLX package/source combination later refused the Flash Next pack because of source-package drift, so no invalid fresh AR number was substituted. Fan boost was also unavailable because neither supported fan utility is installed; both engines were compared under the same no-boost constraint. Receipts: docs/QWEN38_PERFORMANCE.md and docs/qwen38-issue-99.csv. Verification passed: - exact MTP token/cache restoration oracle - exact MLX QMV/QMM reference comparison - Qwen long-context logical-depth oracle - release three-run AR/MTP performance gates - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - make bundle - cargo test --all-features: 239 main tests plus 4 artifact-tool tests passed; 37 declared hardware/artifact tests ignored

hugo closed this issue

2026-09-04 21:28:16 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#99