Optimize the completed Qwen3.8 Flash Next path on the user's M5 Max MacBook Pro, meet the same-model MTPLX target, publish parity/performance receipts, and enable the model's production defaults.
Sequence
Step 7 of 7. Previous: #98. This is the final issue. Start only after correctness, long-context state, and MTP are complete.
Baseline contract
Run mtplx serve and DS4Server on the same M5 Max MacBook Pro with the exact pinned Bare Speed model/precision map, single stream, fans verified at maximum, identical prompts/context, official sampling at temperature 1.0/top-p 0.95/top-k 20, and controlled thermal/load order.
Capture same-process three-run medians for prefill, first token, plain AR, and default MTP. Preserve prompt and generated token IDs/hashes so speed cannot be gained by changing behavior.
Retain the pinned DS4 parity baseline wherever the DS4 Qwen path is available.
Optimization order
First reduce Rust/bridge scheduling overhead through pipeline, command-buffer, and stable-allocation reuse plus contiguous GDN-layer scheduling.
Profile and A/B fused GDN input projection, conv+SiLU+L2 normalization, single-row decay/delta recurrence/gated norm, and verify-width conv/norm.
Profile and A/B fused hyper-connection read and merged MoE gate/up projections.
Profile and A/B sparse-QSA row gathering and large-prefill kernels with capability-specific fallbacks.
Tune MTP capture-commit, proposal depth, verifier width, and PLE hot-cache/prefetch using complete wall-clock receipts.
Do not implement MTPLX experiments already measured flat/slower—custom MoE verify, fused GDN output projection, or async AR—unless a DS4Server profile proves the corresponding bottleneck.
Performance target
The primary target is the exact MTPLX Bare Speed pack. Its public same-task M5 Max result is 47.0 tok/s plain AR and 75.9 tok/s MTP; MTPLX 2.10 also reports approximately 61 tok/s AR and 63–76 tok/s MTP depending on workload.
Default-MTP sampled decode must reach at least 60 tok/s on the user's machine.
DS4Server must remain within 5% of the freshly measured same-machine MTPLX median on the matched coding workload; the published 75.9 tok/s MTP result is the goal.
Plain AR and prefill must separately remain within 5% of the matched MTPLX baselines.
DS4Server must remain within 5% of the DS4 oracle where that oracle is available, with identical model behavior.
Measure short context plus 16K, 64K, 131K, and the admitted native maximum.
Require A/B/A or B/A/B evidence and exact fallback for every promoted fast path.
Run the full existing model regression suite, bundle gate, and Qwen parity/performance gates before enabling production defaults.
Completion requirements
The exact Bare Speed model is selectable and runs with correct official sampling, streamed PLE, long-context state, checkpoint/resume, and default MTP.
Default-MTP decode reaches at least 60 tok/s and stays within 5% of the matched MTPLX result on the user's M5 Max MacBook Pro.
The report accounts for any gap to 75.9 tok/s and demonstrates matched model data, quantization, workload, and memory policy.
Every enabled optimization wins its measured workload, preserves tokens/distribution, and falls back safely.
All parity, regression, packaging, and release gates pass.
Non-goals
Performance promises on non-M5 or unmatched artifacts/workloads, benchmark-only defaults, MLX/Python runtime integration, CUDA/ROCm, vision, or context beyond 262,144.
## Area
Optimize the completed Qwen3.8 Flash Next path on the user's M5 Max MacBook Pro, meet the same-model MTPLX target, publish parity/performance receipts, and enable the model's production defaults.
## Sequence
Step 7 of 7. Previous: #98. This is the final issue. Start only after correctness, long-context state, and MTP are complete.
## Baseline contract
- Run `mtplx serve` and DS4Server on the same M5 Max MacBook Pro with the exact pinned Bare Speed model/precision map, single stream, fans verified at maximum, identical prompts/context, official sampling at temperature 1.0/top-p 0.95/top-k 20, and controlled thermal/load order.
- Capture same-process three-run medians for prefill, first token, plain AR, and default MTP. Preserve prompt and generated token IDs/hashes so speed cannot be gained by changing behavior.
- Retain the pinned DS4 parity baseline wherever the DS4 Qwen path is available.
## Optimization order
- First reduce Rust/bridge scheduling overhead through pipeline, command-buffer, and stable-allocation reuse plus contiguous GDN-layer scheduling.
- Profile and A/B fused GDN input projection, conv+SiLU+L2 normalization, single-row decay/delta recurrence/gated norm, and verify-width conv/norm.
- Profile and A/B fused hyper-connection read and merged MoE gate/up projections.
- Profile and A/B sparse-QSA row gathering and large-prefill kernels with capability-specific fallbacks.
- Tune MTP capture-commit, proposal depth, verifier width, and PLE hot-cache/prefetch using complete wall-clock receipts.
- Do not implement MTPLX experiments already measured flat/slower—custom MoE verify, fused GDN output projection, or async AR—unless a DS4Server profile proves the corresponding bottleneck.
## Performance target
The primary target is the exact MTPLX Bare Speed pack. Its public same-task M5 Max result is 47.0 tok/s plain AR and 75.9 tok/s MTP; MTPLX 2.10 also reports approximately 61 tok/s AR and 63–76 tok/s MTP depending on workload.
- Default-MTP sampled decode must reach at least 60 tok/s on the user's machine.
- DS4Server must remain within 5% of the freshly measured same-machine MTPLX median on the matched coding workload; the published 75.9 tok/s MTP result is the goal.
- Plain AR and prefill must separately remain within 5% of the matched MTPLX baselines.
- DS4Server must remain within 5% of the DS4 oracle where that oracle is available, with identical model behavior.
## Verification and receipts
- Publish raw CSV, commands, hardware/OS, artifact hashes, prompts, contexts, sampler, cold/warm state, token hashes, prefill/first-token/steady throughput, p50/p95 latency, peak memory, PLE-cache statistics, and MTP acceptance/timing.
- Measure short context plus 16K, 64K, 131K, and the admitted native maximum.
- Require A/B/A or B/A/B evidence and exact fallback for every promoted fast path.
- Run the full existing model regression suite, bundle gate, and Qwen parity/performance gates before enabling production defaults.
## Completion requirements
- The exact Bare Speed model is selectable and runs with correct official sampling, streamed PLE, long-context state, checkpoint/resume, and default MTP.
- Default-MTP decode reaches at least 60 tok/s and stays within 5% of the matched MTPLX result on the user's M5 Max MacBook Pro.
- The report accounts for any gap to 75.9 tok/s and demonstrates matched model data, quantization, workload, and memory policy.
- Every enabled optimization wins its measured workload, preserves tokens/distribution, and falls back safely.
- All parity, regression, packaging, and release gates pass.
## Non-goals
- Performance promises on non-M5 or unmatched artifacts/workloads, benchmark-only defaults, MLX/Python runtime integration, CUDA/ROCm, vision, or context beyond 262,144.
hugo
added the idea label 2026-09-01 18:24:31 +00:00
Implemented and delivered in commit bd63538 (pushed to origin/main).
The Qwen3.8 Flash Next path now follows the pinned MTPLX/MLX execution contract end to end: exact affine QMV/QMM and BF16 rounding, fused GDN input/conv/recurrent/verifier capture, row-owned router and merged gate/up, MTPLX qsa_flash_skip long-context attention, streamed group-32 PLE with bounded hot cache and telemetry, NumPy-compatible PCG64 plus official temperature 1.0/top-p 0.95/top-k 20 sampling, exact MTP accept/reject/cache repair and full-accept bonus, stable allocation/pipeline reuse, and device-side sparse sampling. Experimental NAX, custom verifier MoE, scalar QMM, altered Metal math, and shortened recurrence paths were removed after oracle failures or lack of a measured win. MTP is now the Qwen production default while an explicit user opt-out remains authoritative. Non-Qwen sampling remains on the DS4 behavior.
Measured on Mac17,7 / Apple M5 Max / 128 GiB / macOS 26.6.2. The matched 4096-context, 26-prompt-token, 64-output-token acceptance sequence reached a 60.825779 tok/s three-run MTP median with stable token SHA-256 46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72, 44 accepted of 60 drafts, and 20 verifier passes. A later confirmation sequence under the then-current load measured 58.019436 tok/s with the identical token hash. Both are within 5% of the fresh matched MTPLX median 59.751170 tok/s. Plain AR median was 50.671077 tok/s. Peak MTP RSS is 52.61 GB, down from the original 102.65 GB path. Long-context oracle coverage passed at 2K, 16K, 64K, 131K, and 262K.
The public 75.9 tok/s figure is not the matched workload result. The current installed MTPLX package/source combination later refused the Flash Next pack because of source-package drift, so no invalid fresh AR number was substituted. Fan boost was also unavailable because neither supported fan utility is installed; both engines were compared under the same no-boost constraint.
Receipts: docs/QWEN38_PERFORMANCE.md and docs/qwen38-issue-99.csv.
cargo test --all-features: 239 main tests plus 4 artifact-tool tests passed; 37 declared hardware/artifact tests ignored
Implemented and delivered in commit bd63538 (pushed to origin/main).
The Qwen3.8 Flash Next path now follows the pinned MTPLX/MLX execution contract end to end: exact affine QMV/QMM and BF16 rounding, fused GDN input/conv/recurrent/verifier capture, row-owned router and merged gate/up, MTPLX qsa_flash_skip long-context attention, streamed group-32 PLE with bounded hot cache and telemetry, NumPy-compatible PCG64 plus official temperature 1.0/top-p 0.95/top-k 20 sampling, exact MTP accept/reject/cache repair and full-accept bonus, stable allocation/pipeline reuse, and device-side sparse sampling. Experimental NAX, custom verifier MoE, scalar QMM, altered Metal math, and shortened recurrence paths were removed after oracle failures or lack of a measured win. MTP is now the Qwen production default while an explicit user opt-out remains authoritative. Non-Qwen sampling remains on the DS4 behavior.
Measured on Mac17,7 / Apple M5 Max / 128 GiB / macOS 26.6.2. The matched 4096-context, 26-prompt-token, 64-output-token acceptance sequence reached a 60.825779 tok/s three-run MTP median with stable token SHA-256 46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72, 44 accepted of 60 drafts, and 20 verifier passes. A later confirmation sequence under the then-current load measured 58.019436 tok/s with the identical token hash. Both are within 5% of the fresh matched MTPLX median 59.751170 tok/s. Plain AR median was 50.671077 tok/s. Peak MTP RSS is 52.61 GB, down from the original 102.65 GB path. Long-context oracle coverage passed at 2K, 16K, 64K, 131K, and 262K.
The public 75.9 tok/s figure is not the matched workload result. The current installed MTPLX package/source combination later refused the Flash Next pack because of source-package drift, so no invalid fresh AR number was substituted. Fan boost was also unavailable because neither supported fan utility is installed; both engines were compared under the same no-boost constraint.
Receipts: docs/QWEN38_PERFORMANCE.md and docs/qwen38-issue-99.csv.
Verification passed:
- exact MTP token/cache restoration oracle
- exact MLX QMV/QMM reference comparison
- Qwen long-context logical-depth oracle
- release three-run AR/MTP performance gates
- cargo fmt --all -- --check
- cargo clippy --all-targets --all-features -- -D warnings
- make bundle
- cargo test --all-features: 239 main tests plus 4 artifact-tool tests passed; 37 declared hardware/artifact tests ignored
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Area
Optimize the completed Qwen3.8 Flash Next path on the user's M5 Max MacBook Pro, meet the same-model MTPLX target, publish parity/performance receipts, and enable the model's production defaults.
Sequence
Step 7 of 7. Previous: #98. This is the final issue. Start only after correctness, long-context state, and MTP are complete.
Baseline contract
mtplx serveand DS4Server on the same M5 Max MacBook Pro with the exact pinned Bare Speed model/precision map, single stream, fans verified at maximum, identical prompts/context, official sampling at temperature 1.0/top-p 0.95/top-k 20, and controlled thermal/load order.Optimization order
Performance target
The primary target is the exact MTPLX Bare Speed pack. Its public same-task M5 Max result is 47.0 tok/s plain AR and 75.9 tok/s MTP; MTPLX 2.10 also reports approximately 61 tok/s AR and 63–76 tok/s MTP depending on workload.
Verification and receipts
Completion requirements
Non-goals
hugo referenced this issue2026-09-01 18:27:01 +00:00
Implemented and delivered in commit
bd63538(pushed to origin/main).The Qwen3.8 Flash Next path now follows the pinned MTPLX/MLX execution contract end to end: exact affine QMV/QMM and BF16 rounding, fused GDN input/conv/recurrent/verifier capture, row-owned router and merged gate/up, MTPLX qsa_flash_skip long-context attention, streamed group-32 PLE with bounded hot cache and telemetry, NumPy-compatible PCG64 plus official temperature 1.0/top-p 0.95/top-k 20 sampling, exact MTP accept/reject/cache repair and full-accept bonus, stable allocation/pipeline reuse, and device-side sparse sampling. Experimental NAX, custom verifier MoE, scalar QMM, altered Metal math, and shortened recurrence paths were removed after oracle failures or lack of a measured win. MTP is now the Qwen production default while an explicit user opt-out remains authoritative. Non-Qwen sampling remains on the DS4 behavior.
Measured on Mac17,7 / Apple M5 Max / 128 GiB / macOS 26.6.2. The matched 4096-context, 26-prompt-token, 64-output-token acceptance sequence reached a 60.825779 tok/s three-run MTP median with stable token SHA-256 46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72, 44 accepted of 60 drafts, and 20 verifier passes. A later confirmation sequence under the then-current load measured 58.019436 tok/s with the identical token hash. Both are within 5% of the fresh matched MTPLX median 59.751170 tok/s. Plain AR median was 50.671077 tok/s. Peak MTP RSS is 52.61 GB, down from the original 102.65 GB path. Long-context oracle coverage passed at 2K, 16K, 64K, 131K, and 262K.
The public 75.9 tok/s figure is not the matched workload result. The current installed MTPLX package/source combination later refused the Flash Next pack because of source-package drift, so no invalid fresh AR number was substituted. Fan boost was also unavailable because neither supported fan utility is installed; both engines were compared under the same no-boost constraint.
Receipts: docs/QWEN38_PERFORMANCE.md and docs/qwen38-issue-99.csv.
Verification passed: