Optimize Qwen3.8 inference on Apple silicon
This commit is contained in:
Georg Bauer
146
docs/QWEN38_PERFORMANCE.md
Normal file
146
docs/QWEN38_PERFORMANCE.md
Normal file
@@ -0,0 +1,146 @@
|
||||
# Qwen3.8 Flash Next performance and parity receipt
|
||||
|
||||
Issue #99 optimizes the pinned `Youssofal/Qwen3.8-Flash-Next-MTPLX-Bare-Speed`
|
||||
text model without changing its artifact precision, sampler, or token behavior.
|
||||
All application and inference work remains Rust; native additions are confined to
|
||||
the existing Objective-C Metal bridge and Qwen `.metal` kernels. Integrated MTP
|
||||
is the Qwen production default; an explicitly stored user opt-out remains
|
||||
authoritative.
|
||||
|
||||
## Result
|
||||
|
||||
The acceptance B1 median is **60.825779 tok/s MTP**. A later B2 confirmation,
|
||||
after adding complete repair telemetry and under the then-current machine load,
|
||||
measured **58.019436 tok/s**. Both sequences produced the same 64-token SHA-256
|
||||
`46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72`, with 44
|
||||
accepted drafts from 60 proposals in 20 verifier passes. The fresh matched MTPLX
|
||||
median was 59.751170 tok/s, putting B1 1.8% above and B2 2.9% below the oracle;
|
||||
both are inside the issue's 5% parity band. Raw per-run data is in
|
||||
`docs/qwen38-issue-99.csv`.
|
||||
|
||||
Plain AR reached a 50.671077 tok/s median with stable token hash
|
||||
`15373689bcb1c667123accfcfbe49736466f28ce7beb89d2cbe5e4120416056d`.
|
||||
The published 47 tok/s MTPLX figure is contextual rather than a fresh matched
|
||||
AR baseline: after the valid MTP oracle run, the installed MTPLX package and
|
||||
checked-out source drifted apart. The current CLI rejects this Flash Next pack
|
||||
as unsupported and the source path fails at the missing
|
||||
`DecoderLayer.input_layernorm`, so no invalid number is presented as a fresh AR
|
||||
comparison.
|
||||
|
||||
The published 75.9 tok/s MTP result is from a different MTPLX runtime/workload.
|
||||
With identical pinned artifacts, 4,096 context, prompt, sampling, seed, and
|
||||
64-token measurement window, the current usable oracle result is 59.751170
|
||||
tok/s. The remaining gap is therefore not explained by quantization drift or a
|
||||
different output distribution.
|
||||
|
||||
Maximum fan control could not be established because neither ThermalForge nor
|
||||
TG Pro is installed; MTPLX reports its fan boost as unavailable. Both compared
|
||||
engines ran under the same no-boost constraint. No system utility was installed
|
||||
or substituted for the missing control.
|
||||
|
||||
## Fixed workload
|
||||
|
||||
- Hardware: MacBook Pro `Mac17,7`, Apple M5 Max, 18 CPU cores, 128 GiB unified
|
||||
memory.
|
||||
- OS: macOS 26.6.2, build 25G83, arm64.
|
||||
- Model revision: `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`.
|
||||
- Model manifest contract SHA-256:
|
||||
`6f1172de47fa30b9602e13fc7ad14e578a813bb5b1fab8f0ef320041aac6c19a`.
|
||||
- Tensor inventory SHA-256:
|
||||
`b5731e6febcf865d276a0e7b144da02375f2d7e3129594f271ee9c6c351f4c8f`.
|
||||
- `config.json` SHA-256:
|
||||
`826302af9ff3293cfc9975c9a5a96546fa0aa9f0808505adfc30a62fef806965`.
|
||||
- `mtplx_runtime.json` SHA-256:
|
||||
`381c7666872f878542494a48a874bb0784c72a955d18cc077324d3c9e74b8cdb`.
|
||||
- Prompt: `Write a concise Rust implementation of binary search and explain its invariants.`
|
||||
- Prompt tokens: 26; context: 4,096; generated tokens: 64.
|
||||
- Sampling: temperature 1.0, top-p 0.95, top-k 20, min-p 0, seed 12,345.
|
||||
- One process, three runs: first run cold, following runs warm.
|
||||
|
||||
The exact DS4Server commands were:
|
||||
|
||||
```sh
|
||||
DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
|
||||
DS4SERVER_QWEN38_MEASURED=64 DS4SERVER_QWEN38_RUNS=3 \
|
||||
cargo test --release qwen_m5_performance_gate -- --ignored --nocapture
|
||||
|
||||
DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
|
||||
DS4SERVER_QWEN38_MTP=1 DS4SERVER_QWEN38_MEASURED=64 \
|
||||
DS4SERVER_QWEN38_RUNS=3 \
|
||||
cargo test --release qwen_m5_performance_gate -- --ignored --nocapture
|
||||
|
||||
DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
|
||||
cargo test --release qwen_mtp_matches_target_and_restores_its_cache \
|
||||
-- --ignored --nocapture
|
||||
```
|
||||
|
||||
The MTPLX comparison used its normal `serve` path with the same local Bare Speed
|
||||
directory, generation mode, prompt, context, sampler, seed, and single request
|
||||
stream. A later retry is not a valid benchmark because the local MTPLX
|
||||
source/package compatibility gate fails as described above.
|
||||
|
||||
## MTPLX logic review
|
||||
|
||||
The production path now follows the MTPLX/MLX graph rather than retaining the
|
||||
temporary Qwen experiments used while earlier issues established correctness:
|
||||
|
||||
- PLE uses the artifact's 4-bit, group-32 rows, streams them with `pread`, keeps
|
||||
a bounded hot cache, and publishes hit, miss, eviction, byte, and read-time
|
||||
counters. The warm MTP runs had 2,080 hits, no misses, and no reads.
|
||||
- GDN uses fused input projection, MTPLX-equivalent conv+SiLU+L2 normalization,
|
||||
BF16 rounding points, decay/delta recurrence, gated norm, and direct verifier
|
||||
capture buffers. Rejected rows restore or advance authoritative recurrence
|
||||
rather than inventing state.
|
||||
- QSA uses the MTPLX `qsa_flash_skip` row ownership, online-softmax merge, and
|
||||
native 256-thread/eight-SIMD layout. Long-context state remains linear and
|
||||
admitted through 262,144 tokens without a gather-copy fallback.
|
||||
- MoE uses the exact router ownership and merged gate/up projection. The custom
|
||||
verifier MoE experiment and NAX path were removed because they are not the
|
||||
oracle's promoted path.
|
||||
- MTP uses the same checkpoint, history, proposal, verifier, rejection-repair,
|
||||
full-accept bonus, and NumPy-compatible PCG64 sampling rules. The exact cache
|
||||
restoration test passes after the final telemetry change.
|
||||
- Sparse top-k sampling stays on device for the supported official sampler and
|
||||
uses the existing exact CPU path only when the requested sampler is outside
|
||||
that kernel's contract. Full production logits are not copied to the CPU.
|
||||
- Capability fallbacks remain exact implementations, not alternate Qwen
|
||||
behavior. Experiments that changed tokens (normal Metal math, wider Tensor
|
||||
QMM, and recurrence-only repair) were removed rather than left dormant.
|
||||
|
||||
The 4-bit/group-64 core, declared 8-bit router/QSA matrices, BF16 recurrent and
|
||||
normalization tensors, 4-bit/group-32 PLE, and published 2/4/8-bit MTP tensors
|
||||
are consumed as stored. No blanket up-conversion or replacement quantization is
|
||||
used. Peak RSS in the final MTP sequence was 52.61 GB, down from the original
|
||||
102.65 GB path; the 105.44 GB artifact size includes a 32.00 GB file-backed PLE
|
||||
table that is not resident wholesale.
|
||||
|
||||
## A/B evidence
|
||||
|
||||
The optimization work used exact hashes or tensor comparisons before promotion:
|
||||
|
||||
| Area | Before | After | Decision |
|
||||
| --- | ---: | ---: | --- |
|
||||
| Resident memory | 102.65 GB RSS | 52.61 GB RSS | Reused staging and streamed PLE promoted |
|
||||
| Router | 49.7 ms GPU | 18.7 ms GPU, 23.4 ms wall | Row-owned reduced router promoted |
|
||||
| 16K QSA AR | 2.6 tok/s | 45.7 tok/s | MTPLX flash-skip rewrite promoted |
|
||||
| Shared verifier gate/up | 54.7 tok/s | 58.3 tok/s | Exact fused projection promoted |
|
||||
| Direct GDN verifier capture | 58.5/58.7 tok/s | 60.8 tok/s | Direct capture buffers promoted |
|
||||
|
||||
The direct-capture B1 acceptance was followed by the B2 confirmation in the CSV;
|
||||
both retained the same token hash. Normal Metal math, Tensor QMM at verifier
|
||||
width, and shortened recurrence repair failed exact-oracle checks and were
|
||||
removed. Custom MoE verify, fused GDN output projection, and async AR were not
|
||||
promoted because the MTPLX evidence did not show a win and DS4Server profiles did
|
||||
not justify overriding that result.
|
||||
|
||||
## Context and verification
|
||||
|
||||
The long-context gate exercised 2,048, 16,384, 65,536, 131,072, and the native
|
||||
262,144-token maximum. AR remained 43.488 tok/s at the maximum, and live state
|
||||
grew from 58,195,968 bytes at 2K to 7,449,083,904 bytes at 262K without collapse
|
||||
or over-admission. The CSV records every context result and state size.
|
||||
|
||||
Final issue verification comprises the exact MTP cache/token oracle, affine
|
||||
QMV/QMM MLX reference comparisons, focused Qwen tests, the long-context state
|
||||
gate, the release performance gate, formatting, Clippy with warnings denied, the
|
||||
macOS application bundle, and the full all-features Rust test suite.
|
||||
18
docs/qwen38-issue-99.csv
Normal file
18
docs/qwen38-issue-99.csv
Normal file
@@ -0,0 +1,18 @@
|
||||
date,engine,mode,sequence,run,state,context,prompt_tokens,generated_tokens,temperature,top_p,top_k,seed,prefill_tps,first_token_ms,steady_tps,decode_tps,p50_cycle_ms,p95_cycle_ms,peak_rss_bytes,token_sha256,cycles,drafted,accepted,verifier_passes,verifier_ms,repair_ms,ple_entries,ple_hits,ple_misses,ple_evictions,ple_pread_bytes,ple_pread_ms,notes
|
||||
2026-09-04,DS4Server,MTP,B1,1,cold,4096,26,64,1.0,0.95,20,12345,6.360708,59.484667,57.518099,58.042801,54.731042,57.838709,52720156672,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,1000,0,1648,432,1648,0,164800,,repair timing counter not yet wired
|
||||
2026-09-04,DS4Server,MTP,B1,2,warm,4096,26,64,1.0,0.95,20,12345,51.095361,52.229584,60.307983,61.119864,52.427709,54.273459,52720156672,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,946,0,1648,2080,0,0,0,0,repair timing counter not yet wired
|
||||
2026-09-04,DS4Server,MTP,B1,3,warm,4096,26,64,1.0,0.95,20,12345,51.152493,52.422375,60.014218,60.825779,52.679334,54.147250,52720156672,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,951,0,1648,2080,0,0,0,0,acceptance median 60.825779
|
||||
2026-09-04,DS4Server,MTP,B2,1,cold,4096,26,64,1.0,0.95,20,12345,6.527240,62.605750,54.320764,54.834156,57.909625,60.828916,52605190144,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,1056,30,1648,432,1648,0,164800,63,confirmation after repair telemetry
|
||||
2026-09-04,DS4Server,MTP,B2,2,warm,4096,26,64,1.0,0.95,20,12345,50.178762,54.061500,57.196397,58.019436,55.002750,57.242875,52605190144,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,993,29,1648,2080,0,0,0,0,confirmation median 58.019436
|
||||
2026-09-04,DS4Server,MTP,B2,3,warm,4096,26,64,1.0,0.95,20,12345,49.976289,55.289000,57.556598,58.301539,54.853000,56.619333,52605190144,46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72,20,60,44,20,988,30,1648,2080,0,0,0,0,confirmation after repair telemetry
|
||||
2026-09-04,DS4Server,AR,A1,1,cold,4096,26,64,1.0,0.95,20,12345,7.796453,21.087125,49.845055,49.805297,18.894042,19.426917,53962735616,15373689bcb1c667123accfcfbe49736466f28ce7beb89d2cbe5e4120416056d,0,0,0,0,0,0,1648,16,1424,0,,,plain autoregressive
|
||||
2026-09-04,DS4Server,AR,A1,2,warm,4096,26,64,1.0,0.95,20,12345,51.386057,18.606375,51.034286,51.074537,18.392375,18.953542,53962735616,15373689bcb1c667123accfcfbe49736466f28ce7beb89d2cbe5e4120416056d,0,0,0,0,0,0,1648,,,,,,plain autoregressive
|
||||
2026-09-04,DS4Server,AR,A1,3,warm,4096,26,64,1.0,0.95,20,12345,51.424682,18.256166,50.610874,50.671077,18.575958,19.129667,53962735616,15373689bcb1c667123accfcfbe49736466f28ce7beb89d2cbe5e4120416056d,0,0,0,0,0,0,1648,,,,,,plain autoregressive median 50.671077
|
||||
2026-09-04,DS4Server,AR,LONG,1,warm,2048,,1,,,,,0.778000,,,29.259000,,,52158349312,b6d28407,,,,,,,,,,,,,synthetic long-context state gate token=5 live_state_bytes=58195968
|
||||
2026-09-04,DS4Server,AR,LONG,1,warm,16384,,1,,,,,12.101000,,,46.486000,,,,79d71772,,,,,,,,,,,,,synthetic long-context state gate token=5 live_state_bytes=465567744
|
||||
2026-09-04,DS4Server,AR,LONG,1,warm,65536,,1,,,,,10.037000,,,45.739000,,,,0a1131fe,,,,,,,,,,,,,synthetic long-context state gate token=5 live_state_bytes=1862270976
|
||||
2026-09-04,DS4Server,AR,LONG,1,warm,131072,,1,,,,,3.277000,,,45.145000,,,,fb678d10,,,,,,,,,,,,,synthetic long-context state gate token=5 live_state_bytes=3724541952
|
||||
2026-09-04,DS4Server,AR,LONG,1,warm,262144,,1,,,,,0.777000,,,43.488000,,,,01ae755c,,,,,,,,,,,,,synthetic native-maximum state gate token=5 live_state_bytes=7449083904
|
||||
2026-09-04,MTPLX,MTP,ORACLE,1,warm,4096,26,64,1.0,0.95,20,12345,,,,59.170000,,,,,20,58,44,20,950,,,,,,,,fresh matched run before local source-package drift
|
||||
2026-09-04,MTPLX,MTP,ORACLE,2,warm,4096,26,64,1.0,0.95,20,12345,,,,59.820000,,,,,20,58,44,20,950,,,,,,,,fresh matched run before local source-package drift
|
||||
2026-09-04,MTPLX,MTP,ORACLE,3,warm,4096,26,64,1.0,0.95,20,12345,,,,59.751170,,,,,20,58,44,20,950,,,,,,,,matched median 59.751170
|
||||
|
Reference in New Issue
Block a user