Reach current DS4 performance parity on Apple Metal from the Rust execution path #78

New Issue

hugo · 2026-08-30T10:56:57Z

hugo commented

2026-08-30 10:56:57 +00:00

Goal

Bring DS4Server's Rust-owned Metal execution path to current DS4 performance parity, with particular emphasis on the exact M5 fast paths when DS4Server runs on an M5-class Apple Silicon device.

DS4 remains the behavioral oracle. Device-specific paths must be selected automatically using the same capability and shape gates as DS4: M5 optimizations run only on supported M5 hardware and matching graph shapes, pre-M5 paths remain limited to their supported devices, and every fused operation retains the exact unfused fallback.

Audit evidence

The comparison range is DS4 efdadd41e20134af4f3381e1ed90e96fe4faef6f through 8db89fe083ae4d17c9a2428ccd29803d3ae8f577.

DS4Server already vendors the current native Metal implementation:

all 19 Metal kernel files are byte-identical to DS4 at 8db89fe;
ds4.h and ds4_ssd.h are byte-identical;
ds4_gpu.h contains the upstream API plus DS4Server's cache statistics;
ds4_metal.m contains the upstream implementation plus narrow DS4Server diagnostics.

The remaining gap is host scheduling. Rust owns the model graph and does not currently bind or call several APIs that current ds4.c uses. The kernels are present but cannot deliver their intended gains while Rust continues to dispatch the older unfused sequence.

Relevant upstream work includes 4893e0c, 4e401a1, 0e125e6, 71d8c2a, b64d3e0, 0e8fb54, e6a6ab5, 0ad494e, and 42033ee. The implementation must recheck the final upstream range before coding rather than treating this list as exhaustive.

Phase 1: freeze the comparison and establish an M5 baseline

Record the exact DS4 and DS4Server commits, model hashes, context sizes, sampling settings, DSpark settings, SSD/resident mode, expert-cache budget, prefill chunk, macOS power mode, and machine identity.
Build a parity matrix for every applicable performance commit since the old reference. Classify each as active in Rust, missing, superseded, or not applicable, with the responsible DS4 call site and DS4Server call site.
Measure DS4 and DS4Server sequentially on the same M5 machine. They must not run concurrently because of memory pressure.
Use identical GGUFs and prompts and report prefill throughput, time to first token, first-token latency, stable decode tokens/s, per-token p50/p95, peak memory, memory pressure, swap delta, and SSD/cache statistics.
Include resident and SSD-streaming runs for DeepSeek V4 Flash 0731. Use at least 128 greedy decode tokens and repeated runs with cooldown.

The benchmark format should be compatible with DS4's speed benchmark so results can be compared directly rather than inferred from whole-turn UI timing.

Phase 2: bind device policy and exact decode fusions

Add the smallest missing Rust FFI surface and mirror current DS4 eligibility rules for:

ds4_gpu_device_is_m5_apple_silicon
ds4_gpu_device_is_pre_m5_apple_silicon
ds4_gpu_set_decode_pipeline_fast_lookup
ds4_gpu_hc_rms_norm_mix_f16_available
ds4_gpu_hc_rms_norm_mix_f16_tensor
ds4_gpu_hc_rms_norm_mix_split_norm_f16_tensor
ds4_gpu_qkv_pair_quad_compressor_store_tensor
ds4_gpu_matmul_f16_quad_compressor_store_tensor
ds4_gpu_dsv4_comp_row_finalize_tensor
ds4_gpu_dsv4_qkv_rms_norm_kv_rope_fp8_store_tensor
ds4_gpu_router_project_select_fused_tensor
ds4_gpu_router_shared_gate_up_q8_0_tensor
ds4_gpu_parallel_ffn_start
ds4_gpu_parallel_ffn_finish
ds4_gpu_parallel_ffn_abort

Port the scheduling in dependency order:

HC normalization/mix producer fusions.
Q/KV projection, normalization, RoPE, FP8-store, and compressor fusions.
Compressor finalization.
Router projection/selection and shared-expert gate/up fusions.
Parallel full-FFN scheduling and cleanup.
Decode pipeline fast lookup where DS4 enables it.

Preserve DS4's precise guards for model shape, tensor type, quality mode, SSD mode, tensor parallelism, retained outputs, and device generation. A fused function returning unavailable or failure must execute the established unfused path without changing tokens, state, or cache accounting. Do not add a user-facing preference or enable an M5 path merely because its symbol exists.

The already-ported Q-head normalization plus RoPE fusion must be verified and reused, not implemented a second time.

Phase 3: streamed experts and prefill

Bind and use ds4_gpu_stream_expert_cache_seed_experts_gpu_copy where current DS4 seeds the streamed cache from already mapped prefill layers.
Keep the existing CPU/pread seed path as the exact fallback when GPU-copy seeding is ineligible or fails.
Verify that the current indexed-attention and routed-MoE prefill entry points actually reach the newer native kernels and use consistent non-expert mapping spans.
Preserve cache budgets, in-flight command-buffer lifetimes, cancellation, teardown, metrics, and deterministic expert selection.
Do not fold issue #77's experimental mmap-view paging mode into this work.

Phase 4: remaining applicable runtime gains

Port the GLM live-prefix rewind from 7694112: when a shorter incoming GLM prompt exactly matches the live token prefix, rewind to prompt_len - 1 and reevaluate the final prompt token instead of rebuilding the complete prompt.

Verify rather than reimplement the already-present behavior that keeps successfully loaded disk KV checkpoints. Delete only invalid checkpoints according to the existing policy.

Phase 5: correctness and performance gates

Add focused coverage for:

M5 versus pre-M5 device selection and graph-shape eligibility;
every fused-call fallback;
partial failure and parallel-FFN abort/cleanup;
resident and SSD-streaming execution;
cache GPU-copy seeding and fallback;
GLM live-prefix rewind;
long-context prefill/decode correctness;
deterministic greedy token and state parity with current DS4 fixtures;
DSpark greedy and exact stochastic sampling parity.

Add a release performance gate based on the Phase 1 harness. It must compare like for like and retain the raw run data.

Acceptance criteria

The final parity matrix accounts for every applicable Metal/runtime performance commit in the audited DS4 range.
Every applicable API listed above is called by the Rust graph under the same conditions as current DS4, or is explicitly documented as inapplicable with evidence.
On M5 hardware, instrumentation or focused tests prove that the M5 fast paths are selected for eligible DeepSeek V4 Flash 0731 workloads.
On unsupported hardware or shapes, those paths are not selected and the existing unfused path remains exact.
Greedy tokens, speculative-decoding behavior, KV/cache state, context accounting, and teardown remain behaviorally identical to DS4.
Median stable decode throughput on the M5 target improves beyond measured run-to-run noise over the pre-change DS4Server baseline.
With identical inputs and settings, DS4Server stable decode throughput is within 5% of current DS4 on the same M5 machine, unless a remaining measured difference is documented and explicitly accepted.
Prefill, time to first token, p95 token latency, memory pressure, swap, and SSD-streaming performance show no unexplained regression beyond measured noise.
Ineffective or unreachable experimental paths are not retained.
cargo fmt --all -- --check, Clippy with -D warnings, make bundle, and cargo test --all-features pass.

Non-goals

MXFP4 GGUF format support; track that separately.
CUDA, ROCm, Mojo, or tensor-parallel backend work.
A port of the C engine or a second graph abstraction.
New UI controls for hardware dispatch.
The experimental zero-copy expert paging design from #77.

## Goal Bring DS4Server's Rust-owned Metal execution path to current DS4 performance parity, with particular emphasis on the exact M5 fast paths when DS4Server runs on an M5-class Apple Silicon device. DS4 remains the behavioral oracle. Device-specific paths must be selected automatically using the same capability and shape gates as DS4: M5 optimizations run only on supported M5 hardware and matching graph shapes, pre-M5 paths remain limited to their supported devices, and every fused operation retains the exact unfused fallback. ## Audit evidence The comparison range is DS4 efdadd41e20134af4f3381e1ed90e96fe4faef6f through 8db89fe083ae4d17c9a2428ccd29803d3ae8f577. DS4Server already vendors the current native Metal implementation: - all 19 Metal kernel files are byte-identical to DS4 at 8db89fe; - ds4.h and ds4_ssd.h are byte-identical; - ds4_gpu.h contains the upstream API plus DS4Server's cache statistics; - ds4_metal.m contains the upstream implementation plus narrow DS4Server diagnostics. The remaining gap is host scheduling. Rust owns the model graph and does not currently bind or call several APIs that current ds4.c uses. The kernels are present but cannot deliver their intended gains while Rust continues to dispatch the older unfused sequence. Relevant upstream work includes 4893e0c, 4e401a1, 0e125e6, 71d8c2a, b64d3e0, 0e8fb54, e6a6ab5, 0ad494e, and 42033ee. The implementation must recheck the final upstream range before coding rather than treating this list as exhaustive. ## Phase 1: freeze the comparison and establish an M5 baseline 1. Record the exact DS4 and DS4Server commits, model hashes, context sizes, sampling settings, DSpark settings, SSD/resident mode, expert-cache budget, prefill chunk, macOS power mode, and machine identity. 2. Build a parity matrix for every applicable performance commit since the old reference. Classify each as active in Rust, missing, superseded, or not applicable, with the responsible DS4 call site and DS4Server call site. 3. Measure DS4 and DS4Server sequentially on the same M5 machine. They must not run concurrently because of memory pressure. 4. Use identical GGUFs and prompts and report prefill throughput, time to first token, first-token latency, stable decode tokens/s, per-token p50/p95, peak memory, memory pressure, swap delta, and SSD/cache statistics. 5. Include resident and SSD-streaming runs for DeepSeek V4 Flash 0731. Use at least 128 greedy decode tokens and repeated runs with cooldown. The benchmark format should be compatible with DS4's speed benchmark so results can be compared directly rather than inferred from whole-turn UI timing. ## Phase 2: bind device policy and exact decode fusions Add the smallest missing Rust FFI surface and mirror current DS4 eligibility rules for: - ds4_gpu_device_is_m5_apple_silicon - ds4_gpu_device_is_pre_m5_apple_silicon - ds4_gpu_set_decode_pipeline_fast_lookup - ds4_gpu_hc_rms_norm_mix_f16_available - ds4_gpu_hc_rms_norm_mix_f16_tensor - ds4_gpu_hc_rms_norm_mix_split_norm_f16_tensor - ds4_gpu_qkv_pair_quad_compressor_store_tensor - ds4_gpu_matmul_f16_quad_compressor_store_tensor - ds4_gpu_dsv4_comp_row_finalize_tensor - ds4_gpu_dsv4_qkv_rms_norm_kv_rope_fp8_store_tensor - ds4_gpu_router_project_select_fused_tensor - ds4_gpu_router_shared_gate_up_q8_0_tensor - ds4_gpu_parallel_ffn_start - ds4_gpu_parallel_ffn_finish - ds4_gpu_parallel_ffn_abort Port the scheduling in dependency order: 1. HC normalization/mix producer fusions. 2. Q/KV projection, normalization, RoPE, FP8-store, and compressor fusions. 3. Compressor finalization. 4. Router projection/selection and shared-expert gate/up fusions. 5. Parallel full-FFN scheduling and cleanup. 6. Decode pipeline fast lookup where DS4 enables it. Preserve DS4's precise guards for model shape, tensor type, quality mode, SSD mode, tensor parallelism, retained outputs, and device generation. A fused function returning unavailable or failure must execute the established unfused path without changing tokens, state, or cache accounting. Do not add a user-facing preference or enable an M5 path merely because its symbol exists. The already-ported Q-head normalization plus RoPE fusion must be verified and reused, not implemented a second time. ## Phase 3: streamed experts and prefill 1. Bind and use ds4_gpu_stream_expert_cache_seed_experts_gpu_copy where current DS4 seeds the streamed cache from already mapped prefill layers. 2. Keep the existing CPU/pread seed path as the exact fallback when GPU-copy seeding is ineligible or fails. 3. Verify that the current indexed-attention and routed-MoE prefill entry points actually reach the newer native kernels and use consistent non-expert mapping spans. 4. Preserve cache budgets, in-flight command-buffer lifetimes, cancellation, teardown, metrics, and deterministic expert selection. 5. Do not fold issue #77's experimental mmap-view paging mode into this work. ## Phase 4: remaining applicable runtime gains Port the GLM live-prefix rewind from 7694112: when a shorter incoming GLM prompt exactly matches the live token prefix, rewind to prompt_len - 1 and reevaluate the final prompt token instead of rebuilding the complete prompt. Verify rather than reimplement the already-present behavior that keeps successfully loaded disk KV checkpoints. Delete only invalid checkpoints according to the existing policy. ## Phase 5: correctness and performance gates Add focused coverage for: - M5 versus pre-M5 device selection and graph-shape eligibility; - every fused-call fallback; - partial failure and parallel-FFN abort/cleanup; - resident and SSD-streaming execution; - cache GPU-copy seeding and fallback; - GLM live-prefix rewind; - long-context prefill/decode correctness; - deterministic greedy token and state parity with current DS4 fixtures; - DSpark greedy and exact stochastic sampling parity. Add a release performance gate based on the Phase 1 harness. It must compare like for like and retain the raw run data. ## Acceptance criteria - The final parity matrix accounts for every applicable Metal/runtime performance commit in the audited DS4 range. - Every applicable API listed above is called by the Rust graph under the same conditions as current DS4, or is explicitly documented as inapplicable with evidence. - On M5 hardware, instrumentation or focused tests prove that the M5 fast paths are selected for eligible DeepSeek V4 Flash 0731 workloads. - On unsupported hardware or shapes, those paths are not selected and the existing unfused path remains exact. - Greedy tokens, speculative-decoding behavior, KV/cache state, context accounting, and teardown remain behaviorally identical to DS4. - Median stable decode throughput on the M5 target improves beyond measured run-to-run noise over the pre-change DS4Server baseline. - With identical inputs and settings, DS4Server stable decode throughput is within 5% of current DS4 on the same M5 machine, unless a remaining measured difference is documented and explicitly accepted. - Prefill, time to first token, p95 token latency, memory pressure, swap, and SSD-streaming performance show no unexplained regression beyond measured noise. - Ineffective or unreachable experimental paths are not retained. - cargo fmt --all -- --check, Clippy with -D warnings, make bundle, and cargo test --all-features pass. ## Non-goals - MXFP4 GGUF format support; track that separately. - CUDA, ROCm, Mojo, or tensor-parallel backend work. - A port of the C engine or a second graph abstraction. - New UI controls for hardware dispatch. - The experimental zero-copy expert paging design from #77.

hugo added the enhancement critical labels 2026-08-30 10:56:57 +00:00

hugo referenced this issue

2026-08-30 10:57:35 +00:00

Add end-to-end MXFP4 GGUF support to the Metal backend #79

hugo commented

2026-08-30 13:27:21 +00:00

Implemented and pushed in 1ac559b (Reach DS4 Metal performance parity).

Implementation:

Bound and scheduled the current DS4 pre-M5/M5 device policy, HC producer, Q/KV compressor, QKV norm/RoPE/store, compressor finalizer, router/shared, parallel FFN, and inverse-attention-RoPE Metal paths with the DS4 graph guards and unfused rollback paths.
Added mapped-prefill GPU expert-cache seeding with exact CPU/pread fallback, token-only prefill mapping, restored static decode spans, deterministic selected-expert loading, and DS4 cache accounting.
Ported the exact GLM shorter-live-prefix rewind and kept speculative state consistent.
Added the audited DS4 performance matrix and retained raw resident/SSD benchmark data in docs/METAL_PARITY.md and docs/metal-parity-issue-78.csv. MXFP4 remains isolated to issue #79.

Parity and performance on Mac17,7 / Apple M5 Max:

Exact 128-token mode-for-mode hashes match DS4: resident ed5461476a82824c3e0440e943a0b2f6b8a8f19dc26a550a1bc255003851ab8d; SSD 2bc33a04bb88f9f3c56f8bae0124b7bf25fd2f32aee81a1bdaf6e979448da052.
Resident median stable decode: 45.602 tok/s versus DS4 45.60 and baseline 39.527.
SSD median stable decode: 25.066 tok/s versus DS4 26.11 and baseline 8.635, 4.0% below DS4 and inside the required 5% gate.
SSD counters match DS4 exactly: 4096 entries, 32958 hits, 4194 misses, 29684662272 pread bytes.
The persisted rollback regression disables every newly scheduled fusion and confirms exact resident and SSD token parity. GPU-copy failure/CPU fallback, long-context decode, target-owned DSpark greedy, exact stochastic DSpark, device policy, and GLM rewind fixtures pass.

Commit gates passed after the final test change:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
make bundle
cargo test --all-features: 188 passed, 0 failed, 16 ignored

The resident and SSD three-run release performance gates also pass with five-second cooldowns and frozen DS4/baseline thresholds.

Implemented and pushed in 1ac559b (Reach DS4 Metal performance parity). Implementation: - Bound and scheduled the current DS4 pre-M5/M5 device policy, HC producer, Q/KV compressor, QKV norm/RoPE/store, compressor finalizer, router/shared, parallel FFN, and inverse-attention-RoPE Metal paths with the DS4 graph guards and unfused rollback paths. - Added mapped-prefill GPU expert-cache seeding with exact CPU/pread fallback, token-only prefill mapping, restored static decode spans, deterministic selected-expert loading, and DS4 cache accounting. - Ported the exact GLM shorter-live-prefix rewind and kept speculative state consistent. - Added the audited DS4 performance matrix and retained raw resident/SSD benchmark data in docs/METAL_PARITY.md and docs/metal-parity-issue-78.csv. MXFP4 remains isolated to issue #79. Parity and performance on Mac17,7 / Apple M5 Max: - Exact 128-token mode-for-mode hashes match DS4: resident ed5461476a82824c3e0440e943a0b2f6b8a8f19dc26a550a1bc255003851ab8d; SSD 2bc33a04bb88f9f3c56f8bae0124b7bf25fd2f32aee81a1bdaf6e979448da052. - Resident median stable decode: 45.602 tok/s versus DS4 45.60 and baseline 39.527. - SSD median stable decode: 25.066 tok/s versus DS4 26.11 and baseline 8.635, 4.0% below DS4 and inside the required 5% gate. - SSD counters match DS4 exactly: 4096 entries, 32958 hits, 4194 misses, 29684662272 pread bytes. - The persisted rollback regression disables every newly scheduled fusion and confirms exact resident and SSD token parity. GPU-copy failure/CPU fallback, long-context decode, target-owned DSpark greedy, exact stochastic DSpark, device policy, and GLM rewind fixtures pass. Commit gates passed after the final test change: - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - make bundle - cargo test --all-features: 188 passed, 0 failed, 16 ignored The resident and SSD three-run release performance gates also pass with five-second cooldowns and frozen DS4/baseline thresholds.

hugo closed this issue

2026-08-30 13:27:28 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#78