121 lines
8.5 KiB
Markdown
121 lines
8.5 KiB
Markdown
# Apple Metal parity for issue #78
|
|
|
|
## Frozen comparison
|
|
|
|
- DS4 oracle: `8db89fe083ae4d17c9a2428ccd29803d3ae8f577`
|
|
- DS4Server pre-issue baseline: `79468c65b68c28e1539acf0a2a14bd42e000e365`
|
|
- DeepSeek V4 Flash 0731 IQ2XXS/w2Q2K model SHA-256:
|
|
`ca22ae2f838e14077c22bc1c1417b71b45b5e5a3687bd96c2ac6e17fdb6261c0`
|
|
- DSpark support-model SHA-256:
|
|
`7e319924541db3f7a163ed7e11d7532a70d48228ab59d36cb81e1d4511885360`
|
|
- Machine: MacBook Pro `Mac17,7`, Apple M5 Max, 128 GiB unified memory,
|
|
macOS 26.6.2, power 100%
|
|
- Context and prefill allocation: 4096 tokens; quality, steering, speculative
|
|
decoding, and DSpark disabled for the throughput gate
|
|
- SSD mode: 4096 cached experts, 4096 popularity-preloaded experts, ordinary
|
|
(not cold) streaming
|
|
- Prompt: `Count from one to two hundred, spelling out every number.` rendered
|
|
to 16 tokens; 128 greedy non-EOS tokens are measured
|
|
|
|
The longer prompt is deliberate: all 128 generated tokens are reachable before
|
|
EOS and are identical between DS4 and DS4Server in each mode. A short-response
|
|
fixture reached EOS early and made a forced post-EOS speed run compare
|
|
unreachable, mode-dependent states.
|
|
|
|
## DS4 performance-change matrix
|
|
|
|
The audited range is
|
|
`efdadd41e20134af4f3381e1ed90e96fe4faef6f..8db89fe083ae4d17c9a2428ccd29803d3ae8f577`.
|
|
Documentation-only, parser/server-only, CUDA, ROCm, tensor-parallel, model
|
|
download, and merge commits do not affect the Rust-owned single-session Metal
|
|
graph. The applicable performance/runtime commits are accounted for below.
|
|
|
|
| DS4 commit(s) | DS4 call site or behavior | DS4Server result |
|
|
| --- | --- | --- |
|
|
| `3ce6777`, `8e0f997` | Native streamed-expert cache pinning and safe fallback | Active in the byte-matched native Metal cache; Rust preserves the cache budget and fallback. |
|
|
| `df04887` | `ds4_gpu_attention_indexed_mixed_batch_heads_tensor` | Active through `encode_layer_with_cache_rows`; the long-context fixture crosses the indexed boundary. |
|
|
| `74e8f11`, `4591cb1`, `a51e6ec`, `f284165`, `af80694`, `7fb2830`, `0e89a0e`, `769a8ba`, `8d683d6` | DSpark block commit, deterministic scheduling, acceptance replay/direct commit, greedy identity, and exact/opportunistic sampling | Active in `Dspark::propose` and the verifier commit paths; greedy and exact stochastic fixtures cover the frozen support model. |
|
|
| `96c3ba4`, `d69a017`, `81f8534`, `ac5cb2c`, `48392e3`, `222b2cb`, `532ec8b` | Metal 4 Q4 projection, compact routed-MoE, aligned Q8, indexed-attention, and routed prefill kernels | Active through the existing `encode_batch_layer` Q4/indexed/routed entry points; no second prefill graph was added. |
|
|
| `24903f3` | `ds4_gpu_head_rms_norm_rope_tail_tensor` | Existing Rust Q-head norm/RoPE fusion is retained and used. |
|
|
| `4893e0c`, `d14ce35` | Mapped-prefill expert seeding and consistent expert/non-expert maps | Rust seeds with `ds4_gpu_stream_expert_cache_seed_experts_gpu_copy`, falls back to CPU/pread, and restores one cached static decode map with the same non-routed spans. |
|
|
| `0dd0d36` | Release speed regression gate | Rust has an ignored release gate with retained raw CSV, DS4 reference threshold, baseline threshold, p50/p95, cache metrics, and token hashes. |
|
|
| `4e401a1`, `0e125e6` | Automatic pre-M5/M5 admission for exact decode fusions | Rust binds both device predicates and mirrors per-generation rollback variables without a UI preference. |
|
|
| `71d8c2a` | Q/KV pair plus ratio-4/128 compressor projection/store | `try_qkv_pair_compressor_fusion` calls the native combined entry point only for DS4's resident full-decode shapes. |
|
|
| `b64d3e0` | Emit-path compressor finalizer | `update_compression_pair` defers both updates only when the resident finalizer is eligible, then requires exact success. |
|
|
| `3826f52`, `ad4d05c` | Exact compressor softmax/product and ratio-4 pool | Native kernels are active; Rust's compressor ABI now passes the upstream `decode_one_token` and `defer_finalize` flags exactly. |
|
|
| `92d83e6` | HC norm/mix producer clustering | `decode_hc_mix` uses split-norm producer fusion, with the established norm/mix/split sequence as fallback. |
|
|
| `1be41c8` | Packed exact split-K attention reduction | Active in the vendored native attention kernel reached by Rust's decode attention calls. |
|
|
| `0e8fb54`, `e6a6ab5` | Router/select, shared gate/up, QKV norm/KV RoPE/store, parallel full FFN, inverse-attention RoPE, and pre-M5 ports | All applicable entry points are called with DS4 shape, tensor, quality, steering, profile, SSD, and device guards. Parallel FFN aborts on partial failure; inverse RoPE falls back when the attention kernel does not consume it. |
|
|
| `0658dac` | Long-context prefill/decode correctness | The 4096-crossing ignored fixture covers compressed/indexed state and the next decode token. |
|
|
| `023614e`, `0ad494e`, `42033ee` | Remove stream-512 experiment; make exact paths automatic; verify DFlash pipeline | No stream-512 or opt-in experiment remains. Feature rollbacks retain identical token hashes. |
|
|
| `24fa85e` | Keep successfully loaded disk KV checkpoints | Existing Rust checkpoint load/round-trip behavior is verified; only invalid checkpoints follow deletion policy. |
|
|
| `7694112` | Reuse a shorter exact GLM live prefix | `live_prefix_rewind_target` rewinds to `prompt_len - 1`, clears pending speculation, and reevaluates the last prompt token. |
|
|
| `5186e2c` | Keep Metal decode changes portable | Device checks and unavailable returns preserve non-M5 and unfused execution. |
|
|
|
|
The Metal MXFP4 commits (`725b084`, `1f862bd`, `7bec128`, `bcd1e57`,
|
|
`f75a065`, and their merge/format work) are intentionally excluded: MXFP4 is
|
|
issue #79. `ds4_gpu_set_decode_pipeline_fast_lookup` is reset to disabled,
|
|
matching DS4 because that cache is enabled only for the pre-M5 MXFP4 path.
|
|
|
|
## Results
|
|
|
|
Three implementations were run sequentially, three times per mode, with a
|
|
five-second cooldown. Medians are from
|
|
[`metal-parity-issue-78.csv`](metal-parity-issue-78.csv).
|
|
|
|
| Implementation | Mode | Prefill tok/s | TTFT ms | First decode ms | Stable tok/s | p50 ms | p95 ms |
|
|
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
| DS4Server baseline | resident | 69.63 | 256.07 | 26.37 | 39.53 | 24.88 | 26.61 |
|
|
| DS4 | resident | 56.49 | 305.94 | 22.71 | 45.60 | 21.91 | 22.32 |
|
|
| DS4Server issue #78 | resident | 69.76 | 252.01 | 22.66 | 45.60 | 21.94 | 22.33 |
|
|
| DS4Server baseline | SSD | 6.81 | 2476.94 | 125.30 | 8.63 | 114.93 | 124.28 |
|
|
| DS4 | SSD | 11.88 | 1393.54 | 46.75 | 26.11 | 35.62 | 46.86 |
|
|
| DS4Server issue #78 | SSD | 11.49 | 1440.79 | 48.84 | 25.07 | 37.70 | 48.85 |
|
|
|
|
TTFT is the per-run prompt prefill duration plus first decode latency; its
|
|
median is reported separately from the first decode kernel latency.
|
|
|
|
Resident stable decode improves 15.4% over the baseline and matches DS4.
|
|
SSD stable decode improves 190.3% over the baseline and is 4.0% below
|
|
DS4, inside the required 5% boundary. SSD cache accounting is identical to
|
|
DS4 for the measured token sequence: 4096 entries, 32,958 hits, 4,194 misses,
|
|
and 29,684,662,272 pread bytes. The resident and SSD token SHA-256 values are,
|
|
respectively, `ed5461476a82824c3e0440e943a0b2f6b8a8f19dc26a550a1bc255003851ab8d`
|
|
and `2bc33a04bb88f9f3c56f8bae0124b7bf25fd2f32aee81a1bdaf6e979448da052`;
|
|
DS4 emitted the same 128-token sequences in the corresponding modes.
|
|
|
|
Representative `/usr/bin/time -l` measurements report:
|
|
|
|
| Implementation | Mode | Maximum RSS bytes | Peak footprint bytes | Process swaps |
|
|
| --- | --- | ---: | ---: | ---: |
|
|
| DS4 | resident | 78,790,656 | 4,746,511,440 | 0 |
|
|
| DS4Server issue #78 | resident | 129,024,000 | 104,071,816 | 0 |
|
|
| DS4 | SSD | 29,066,280,960 | 29,595,620,664 | 0 |
|
|
| DS4Server issue #78 | SSD | 29,098,393,600 | 104,350,344 | 0 |
|
|
|
|
The post-run system memory-pressure reading was 92% free. Every measured
|
|
process reported zero swaps; the post-run host counters were 0 swap-ins and 20
|
|
historical swap-outs.
|
|
|
|
## Reproduction
|
|
|
|
DS4 uses the exact 16 Rust-rendered prompt token IDs with its speed-benchmark
|
|
format. DS4Server uses the installed-model fixture:
|
|
|
|
```sh
|
|
DS4_REFERENCE_TPS=45.60 \
|
|
DS4SERVER_BASELINE_TPS=39.526849 \
|
|
cargo test --release --all-features flash_0731_m5_decode_performance_gate \
|
|
-- --ignored --nocapture --test-threads=1
|
|
|
|
DS4SERVER_BENCH_SSD=1 \
|
|
DS4_REFERENCE_TPS=26.11 \
|
|
DS4SERVER_BASELINE_TPS=8.634659 \
|
|
cargo test --release --all-features flash_0731_m5_decode_performance_gate \
|
|
-- --ignored --nocapture --test-threads=1
|
|
```
|
|
|
|
The gate fails unless throughput improves over the frozen DS4Server baseline
|
|
and remains at least 95% of DS4. `DS4SERVER_BENCH_TOKENS=1` prints the prompt
|
|
and generated sequences for direct DS4 comparison.
|