185 lines
12 KiB
Markdown
185 lines
12 KiB
Markdown
# Apple Metal parity for issue #78
|
|
|
|
## GLM 5.3 Flash parity target
|
|
|
|
- Previous DS4 oracle: `8db89fe083ae4d17c9a2428ccd29803d3ae8f577`
|
|
- Current DS4 oracle: `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`
|
|
- Main artifact: `GLM-5.3-Flash-Q2.gguf`, SHA-256
|
|
`e81fd6241c6e55a64e1e14e47a3eab61a173fa8d7e4b5c1d1848827119705b32`
|
|
- Vision artifact: `GLM-5.3-Flash-Vision-Encoder.gguf`, SHA-256
|
|
`ae23e14c6979e889051b2e4a39351abcdafb161e18e606fae4d8c40095a4bf3a`
|
|
- Machine: MacBook Pro `Mac17,7`, Apple M5 Max, 128 GiB unified memory,
|
|
resident Q2 weights
|
|
|
|
The applicable commits after the previous oracle are:
|
|
|
|
| DS4 commit(s) | Scope carried into DS4Server |
|
|
| --- | --- |
|
|
| `147109a` | GLM 5.3 shape, GGUF binding, KDA/DSA attention, hyperconnections, routed MoE, BF16 tensors, output head, integrated MTP, and Metal kernels |
|
|
| `4771329`, `9f95d9f` | Vision encoder, image preprocessing, prompt overlays, and compact-prefill correctness |
|
|
| `b0c31af` | Pooled indexer cache, bounded attention memory, and batched prefill |
|
|
| `747eafc`, `1df8ff0` | Fixed-image encoder oracle, replay/logit checks, and multimodal quality gates |
|
|
| `8adb221` | Directional attention and FFN steering at the DS4 residual points |
|
|
| `639d4eb` | Continued-prefill cache and position behavior |
|
|
| `6cf658a` | BF16 decode projection schedule and Metal kernel specialization |
|
|
| `01f3316`, `81c8c39`, `5da3206` | Current Metal compact-cache ABI, Q2/full-checkpoint binding, and GLM streaming compatibility |
|
|
|
|
CUDA- and ROCm-only commits in the range are excluded. The Rust release gates
|
|
compare three-run medians against this exact DS4 build, require identical
|
|
greedy token hashes, and fail only when Rust is more than 2% slower than DS4;
|
|
there is no upper bound when Rust is faster. Text measures an
|
|
8192-token frontier plus steady decode, both with and without integrated MTP.
|
|
Vision compares the fixed encoder embedding (cosine at least 0.995, mean
|
|
absolute error at most 0.001, maximum error at most 0.06), encoder latency,
|
|
multimodal prefill, steady decode, image-conditioned logits, and replay.
|
|
|
|
### GLM 5.3 Flash acceptance results
|
|
|
|
The text fixture is `ds4.c`, rendered as one no-system chat turn (`-sys ''` in
|
|
DS4), truncated to exactly 8192 tokens, with a 32768-token allocation and 64
|
|
greedy output tokens. Same-process three-run medians are used because both DS4
|
|
and DS4Server exhibit a load-order/thermal artifact with this 96 GB model.
|
|
|
|
| Capability | Metric | DS4 | DS4Server | Result |
|
|
| --- | --- | ---: | ---: | --- |
|
|
| Text | Prefill tok/s | 288.91 | 295.21 | Rust 2.18% faster |
|
|
| Text | Decode tok/s | 22.58 | 22.78 | Rust 0.89% faster |
|
|
| Integrated MTP | Decode tok/s | 26.55 | 26.80 | Rust 0.93% faster |
|
|
| Vision encoder | Fixed-image seconds | 2.6255 | 2.4354 | Rust 7.24% faster |
|
|
| Vision prompt | Prefill tok/s | 185.81 | 201.51 | Rust 8.45% faster |
|
|
| Vision prompt | Decode tok/s | 26.40 | 27.21 | Rust 3.09% faster |
|
|
|
|
Text and MTP produced the same token SHA-256,
|
|
`0b61e613708d8367fc87f5a32fbfd4ba098b40ee920645b1e38381e867131108`.
|
|
The 1094-token fixed-image prompt produced the same vision token SHA-256,
|
|
`424fdadc10c0a2bd95b405a3985151e84aa844a209d2a2ca72cd01c39dfa53db`.
|
|
The Rust vision embedding compared with the DS4 oracle at cosine
|
|
`0.999958726`, mean absolute error `0.000036702`, and maximum error
|
|
`0.006153620`.
|
|
|
|
The checkpoint advertises a 1048576-token maximum. On this 128 GiB machine,
|
|
32768 is the default manageable context and 50000 is the extended option: the
|
|
50000-token run used 3.51 GiB of live runtime tensors in addition to 90.61 GiB
|
|
of mapped model wrappers and retained the exact output hash. Larger allocations
|
|
leave too little operating-system and application headroom for a reliable
|
|
default.
|
|
|
|
## Frozen comparison
|
|
|
|
- DS4 oracle: `8db89fe083ae4d17c9a2428ccd29803d3ae8f577`
|
|
- DS4Server pre-issue baseline: `79468c65b68c28e1539acf0a2a14bd42e000e365`
|
|
- DeepSeek V4 Flash 0731 IQ2XXS/w2Q2K model SHA-256:
|
|
`ca22ae2f838e14077c22bc1c1417b71b45b5e5a3687bd96c2ac6e17fdb6261c0`
|
|
- DSpark support-model SHA-256:
|
|
`7e319924541db3f7a163ed7e11d7532a70d48228ab59d36cb81e1d4511885360`
|
|
- Machine: MacBook Pro `Mac17,7`, Apple M5 Max, 128 GiB unified memory,
|
|
macOS 26.6.2, power 100%
|
|
- Context and prefill allocation: 4096 tokens; quality, steering, speculative
|
|
decoding, and DSpark disabled for the throughput gate
|
|
- SSD mode: 4096 cached experts, 4096 popularity-preloaded experts, ordinary
|
|
(not cold) streaming
|
|
- Prompt: `Count from one to two hundred, spelling out every number.` rendered
|
|
to 16 tokens; 128 greedy non-EOS tokens are measured
|
|
|
|
The longer prompt is deliberate: all 128 generated tokens are reachable before
|
|
EOS and are identical between DS4 and DS4Server in each mode. A short-response
|
|
fixture reached EOS early and made a forced post-EOS speed run compare
|
|
unreachable, mode-dependent states.
|
|
|
|
## DS4 performance-change matrix
|
|
|
|
The audited range is
|
|
`efdadd41e20134af4f3381e1ed90e96fe4faef6f..8db89fe083ae4d17c9a2428ccd29803d3ae8f577`.
|
|
Documentation-only, parser/server-only, CUDA, ROCm, tensor-parallel, model
|
|
download, and merge commits do not affect the Rust-owned single-session Metal
|
|
graph. The applicable performance/runtime commits are accounted for below.
|
|
|
|
| DS4 commit(s) | DS4 call site or behavior | DS4Server result |
|
|
| --- | --- | --- |
|
|
| `3ce6777`, `8e0f997` | Native streamed-expert cache pinning and safe fallback | Active in the byte-matched native Metal cache; Rust preserves the cache budget and fallback. |
|
|
| `df04887` | `ds4_gpu_attention_indexed_mixed_batch_heads_tensor` | Active through `encode_layer_with_cache_rows`; the long-context fixture crosses the indexed boundary. |
|
|
| `74e8f11`, `4591cb1`, `a51e6ec`, `f284165`, `af80694`, `7fb2830`, `0e89a0e`, `769a8ba`, `8d683d6` | DSpark block commit, deterministic scheduling, acceptance replay/direct commit, greedy identity, and exact/opportunistic sampling | Active in `Dspark::propose` and the verifier commit paths; greedy and exact stochastic fixtures cover the frozen support model. |
|
|
| `96c3ba4`, `d69a017`, `81f8534`, `ac5cb2c`, `48392e3`, `222b2cb`, `532ec8b` | Metal 4 Q4 projection, compact routed-MoE, aligned Q8, indexed-attention, and routed prefill kernels | Active through the existing `encode_batch_layer` Q4/indexed/routed entry points; no second prefill graph was added. |
|
|
| `24903f3` | `ds4_gpu_head_rms_norm_rope_tail_tensor` | Existing Rust Q-head norm/RoPE fusion is retained and used. |
|
|
| `4893e0c`, `d14ce35` | Mapped-prefill expert seeding and consistent expert/non-expert maps | Rust seeds with `ds4_gpu_stream_expert_cache_seed_experts_gpu_copy`, falls back to CPU/pread, and restores one cached static decode map with the same non-routed spans. |
|
|
| `0dd0d36` | Release speed regression gate | Rust has an ignored release gate with retained raw CSV, DS4 reference threshold, baseline threshold, p50/p95, cache metrics, and token hashes. |
|
|
| `4e401a1`, `0e125e6` | Automatic pre-M5/M5 admission for exact decode fusions | Rust binds both device predicates and mirrors per-generation rollback variables without a UI preference. |
|
|
| `71d8c2a` | Q/KV pair plus ratio-4/128 compressor projection/store | `try_qkv_pair_compressor_fusion` calls the native combined entry point only for DS4's resident full-decode shapes. |
|
|
| `b64d3e0` | Emit-path compressor finalizer | `update_compression_pair` defers both updates only when the resident finalizer is eligible, then requires exact success. |
|
|
| `3826f52`, `ad4d05c` | Exact compressor softmax/product and ratio-4 pool | Native kernels are active; Rust's compressor ABI now passes the upstream `decode_one_token` and `defer_finalize` flags exactly. |
|
|
| `92d83e6` | HC norm/mix producer clustering | `decode_hc_mix` uses split-norm producer fusion, with the established norm/mix/split sequence as fallback. |
|
|
| `1be41c8` | Packed exact split-K attention reduction | Active in the vendored native attention kernel reached by Rust's decode attention calls. |
|
|
| `0e8fb54`, `e6a6ab5` | Router/select, shared gate/up, QKV norm/KV RoPE/store, parallel full FFN, inverse-attention RoPE, and pre-M5 ports | All applicable entry points are called with DS4 shape, tensor, quality, steering, profile, SSD, and device guards. Parallel FFN aborts on partial failure; inverse RoPE falls back when the attention kernel does not consume it. |
|
|
| `0658dac` | Long-context prefill/decode correctness | The 4096-crossing ignored fixture covers compressed/indexed state and the next decode token. |
|
|
| `023614e`, `0ad494e`, `42033ee` | Remove stream-512 experiment; make exact paths automatic; verify DFlash pipeline | No stream-512 or opt-in experiment remains. Feature rollbacks retain identical token hashes. |
|
|
| `24fa85e` | Keep successfully loaded disk KV checkpoints | Existing Rust checkpoint load/round-trip behavior is verified; only invalid checkpoints follow deletion policy. |
|
|
| `7694112` | Reuse a shorter exact GLM live prefix | `live_prefix_rewind_target` rewinds to `prompt_len - 1`, clears pending speculation, and reevaluates the last prompt token. |
|
|
| `5186e2c` | Keep Metal decode changes portable | Device checks and unavailable returns preserve non-M5 and unfused execution. |
|
|
|
|
The Metal MXFP4 commits (`725b084`, `1f862bd`, `7bec128`, `bcd1e57`,
|
|
`f75a065`, and their merge/format work) are intentionally excluded: MXFP4 is
|
|
issue #79. `ds4_gpu_set_decode_pipeline_fast_lookup` is reset to disabled,
|
|
matching DS4 because that cache is enabled only for the pre-M5 MXFP4 path.
|
|
|
|
## Results
|
|
|
|
Three implementations were run sequentially, three times per mode, with a
|
|
five-second cooldown. Medians are from
|
|
[`metal-parity-issue-78.csv`](metal-parity-issue-78.csv).
|
|
|
|
| Implementation | Mode | Prefill tok/s | TTFT ms | First decode ms | Stable tok/s | p50 ms | p95 ms |
|
|
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
| DS4Server baseline | resident | 69.63 | 256.07 | 26.37 | 39.53 | 24.88 | 26.61 |
|
|
| DS4 | resident | 56.49 | 305.94 | 22.71 | 45.60 | 21.91 | 22.32 |
|
|
| DS4Server issue #78 | resident | 69.76 | 252.01 | 22.66 | 45.60 | 21.94 | 22.33 |
|
|
| DS4Server baseline | SSD | 6.81 | 2476.94 | 125.30 | 8.63 | 114.93 | 124.28 |
|
|
| DS4 | SSD | 11.88 | 1393.54 | 46.75 | 26.11 | 35.62 | 46.86 |
|
|
| DS4Server issue #78 | SSD | 11.49 | 1440.79 | 48.84 | 25.07 | 37.70 | 48.85 |
|
|
|
|
TTFT is the per-run prompt prefill duration plus first decode latency; its
|
|
median is reported separately from the first decode kernel latency.
|
|
|
|
Resident stable decode improves 15.4% over the baseline and matches DS4.
|
|
SSD stable decode improves 190.3% over the baseline and is 4.0% below
|
|
DS4, inside the required 5% boundary. SSD cache accounting is identical to
|
|
DS4 for the measured token sequence: 4096 entries, 32,958 hits, 4,194 misses,
|
|
and 29,684,662,272 pread bytes. The resident and SSD token SHA-256 values are,
|
|
respectively, `ed5461476a82824c3e0440e943a0b2f6b8a8f19dc26a550a1bc255003851ab8d`
|
|
and `2bc33a04bb88f9f3c56f8bae0124b7bf25fd2f32aee81a1bdaf6e979448da052`;
|
|
DS4 emitted the same 128-token sequences in the corresponding modes.
|
|
|
|
Representative `/usr/bin/time -l` measurements report:
|
|
|
|
| Implementation | Mode | Maximum RSS bytes | Peak footprint bytes | Process swaps |
|
|
| --- | --- | ---: | ---: | ---: |
|
|
| DS4 | resident | 78,790,656 | 4,746,511,440 | 0 |
|
|
| DS4Server issue #78 | resident | 129,024,000 | 104,071,816 | 0 |
|
|
| DS4 | SSD | 29,066,280,960 | 29,595,620,664 | 0 |
|
|
| DS4Server issue #78 | SSD | 29,098,393,600 | 104,350,344 | 0 |
|
|
|
|
The post-run system memory-pressure reading was 92% free. Every measured
|
|
process reported zero swaps; the post-run host counters were 0 swap-ins and 20
|
|
historical swap-outs.
|
|
|
|
## Reproduction
|
|
|
|
DS4 uses the exact 16 Rust-rendered prompt token IDs with its speed-benchmark
|
|
format. DS4Server uses the installed-model fixture:
|
|
|
|
```sh
|
|
DS4_REFERENCE_TPS=45.60 \
|
|
DS4SERVER_BASELINE_TPS=39.526849 \
|
|
cargo test --release --all-features flash_0731_m5_decode_performance_gate \
|
|
-- --ignored --nocapture --test-threads=1
|
|
|
|
DS4SERVER_BENCH_SSD=1 \
|
|
DS4_REFERENCE_TPS=26.11 \
|
|
DS4SERVER_BASELINE_TPS=8.634659 \
|
|
cargo test --release --all-features flash_0731_m5_decode_performance_gate \
|
|
-- --ignored --nocapture --test-threads=1
|
|
```
|
|
|
|
The gate fails unless throughput improves over the frozen DS4Server baseline
|
|
and remains at least 95% of DS4. `DS4SERVER_BENCH_TOKENS=1` prints the prompt
|
|
and generated sequences for direct DS4 comparison.
|