Add end-to-end MXFP4 GGUF support to the Metal backend #79

New Issue

hugo · 2026-08-30T10:57:35Z

hugo commented

2026-08-30 10:57:35 +00:00

Goal

Add end-to-end MXFP4 GGUF support to DS4Server's Rust model loader and Metal execution path.

This is an enhancement rather than a blocker for #78. The current repository already contains DS4's portable and optimized MXFP4 Metal kernels, but Rust rejects GGUF tensor type 39 and therefore cannot use them.

Upstream source of truth

Use current DS4 as the behavioral and format oracle, including the work introduced around:

725b084: lossless DeepSeek MXFP4 GGUF conversion
1f862bd: MXFP4 GGUF format and scalar reference
7bec128: portable Metal MXFP4 expert inference
bcd1e57 and f75a065: exact MXFP4 decode and prefill speedups
0ad494e: automatic exact-path policy

Recheck the current DS4 commit before implementation and record the exact source revision. Do not infer the block layout, scale encoding, tensor-size formula, or dispatch guards from kernel names alone.

Phase 1: port the exact GGUF format contract

Add the MXFP4 tensor type constant and exact block-size/byte-size calculation to the Rust GGUF parser.
Port all relevant validation from DS4, including dimensional alignment, checked size arithmetic, file-range validation, supported tensor roles, and clear rejection of malformed or unsupported layouts.
Add focused parser tests using a minimal GGUF fixture or constructed metadata covering:
- valid MXFP4 tensors;
- invalid block alignment;
- truncated tensor data;
- overflowing dimensions or byte counts;
- unsupported MXFP4 placement.
Keep unknown tensor kinds rejected. Do not weaken existing validation to make type 39 load.

Phase 2: integrate model loading and placement

Teach the model-weight validation layer which DeepSeek tensor roles may use MXFP4, matching DS4 exactly.
Support the format in resident Metal loading and the existing SSD-streaming/expert-cache path where DS4 supports it.
Preserve mmap lifetimes, tensor offsets, page/range validation, streaming cache budgets, and admission accounting.
Report an explicit model-load error when an MXFP4 layout is valid GGUF but unsupported by the selected DS4Server model/backend combination.
Add model-catalog/download entries only for verified artifacts with confirmed URLs, sizes, and hashes. Format support must not depend on an invented or unverified artifact.

Phase 3: route the existing Metal implementation

The current native Metal snapshot already contains the MXFP4 kernels and wrapper entry points. Wire the Rust execution graph to them under the same tensor, model-shape, device, quality, prefill/decode, and SSD/resident conditions as current DS4.

Reuse the capability and M5/pre-M5 policy implemented by #78. MXFP4 exact fast paths must activate automatically only on supported hardware and shapes. Unsupported combinations must either use DS4's exact portable path or fail clearly if DS4 itself does not provide a valid fallback.

Do not duplicate general Q2 decode-fusion work from #78. Keep MXFP4-specific scheduling in this issue and share only the already-established Rust FFI and device-policy helpers.

Phase 4: numerical and behavioral validation

Port or reuse the smallest authoritative scalar/reference checks from DS4 for MXFP4 block decoding and dot products.
Compare DS4 and DS4Server with the same hashed MXFP4 GGUF and identical prompts/settings.
Verify deterministic greedy token parity and relevant intermediate/reference outputs for:
- short prefill and decode;
- representative long prefill;
- resident execution;
- SSD-streaming execution when supported;
- DSpark/MTP execution where the model supports it;
- cancellation and teardown.
Exercise both portable and hardware-specific optimized paths when suitable test hardware is available.
Ensure cache accounting, context accounting, KV state, and model teardown remain unchanged.

Phase 5: performance characterization

Benchmark MXFP4 against current DS4 on the same machine and against DS4Server's closest supported quantization using the same methodology as #78:

separate prefill, first-token, and stable decode measurements;
repeated sequential runs with cooldown;
identical context, prompt, sampling, power mode, and cache settings;
peak memory, memory pressure, swap, SSD bytes, and cache metrics;
no simultaneous DS4 and DS4Server run.

The purpose is to verify that DS4Server receives the native MXFP4 gains and does not merely parse the format. Do not make MXFP4 the default artifact solely because it loads; default selection requires measured quality, memory, and speed evidence.

Acceptance criteria

DS4Server accepts valid GGUF tensor type 39 using the exact current DS4 layout and rejects malformed layouts safely.
Supported DeepSeek MXFP4 models load through the Rust model layer without bypassing existing validation.
Resident and supported SSD-streaming execution reach the existing native MXFP4 kernels.
M5/pre-M5 optimized paths follow current DS4 device and shape gates; no M5-only path runs on unsupported hardware.
Deterministic outputs and state transitions match current DS4 for the same MXFP4 artifact and settings.
Portable/reference tests cover MXFP4 block interpretation independently of the optimized Metal path.
Benchmarks demonstrate the resulting prefill/decode performance and memory behavior relative to current DS4 and the nearest existing DS4Server quantization.
Model catalog entries, if added, use verified artifacts and hashes.
cargo fmt --all -- --check, Clippy with -D warnings, make bundle, and cargo test --all-features pass.

Non-goals

Blocking or expanding the critical Q2/M5 performance work in #78.
CUDA, ROCm, Mojo, or tensor-parallel MXFP4 support.
Maintaining a separate DS4Server GGUF conversion tool unless a concrete distribution workflow requires it.
Changing the default model quantization without benchmark and quality evidence.
Replacing the existing GGUF container or splitting models into per-expert files.

## Goal Add end-to-end MXFP4 GGUF support to DS4Server's Rust model loader and Metal execution path. This is an enhancement rather than a blocker for #78. The current repository already contains DS4's portable and optimized MXFP4 Metal kernels, but Rust rejects GGUF tensor type 39 and therefore cannot use them. ## Upstream source of truth Use current DS4 as the behavioral and format oracle, including the work introduced around: - 725b084: lossless DeepSeek MXFP4 GGUF conversion - 1f862bd: MXFP4 GGUF format and scalar reference - 7bec128: portable Metal MXFP4 expert inference - bcd1e57 and f75a065: exact MXFP4 decode and prefill speedups - 0ad494e: automatic exact-path policy Recheck the current DS4 commit before implementation and record the exact source revision. Do not infer the block layout, scale encoding, tensor-size formula, or dispatch guards from kernel names alone. ## Phase 1: port the exact GGUF format contract 1. Add the MXFP4 tensor type constant and exact block-size/byte-size calculation to the Rust GGUF parser. 2. Port all relevant validation from DS4, including dimensional alignment, checked size arithmetic, file-range validation, supported tensor roles, and clear rejection of malformed or unsupported layouts. 3. Add focused parser tests using a minimal GGUF fixture or constructed metadata covering: - valid MXFP4 tensors; - invalid block alignment; - truncated tensor data; - overflowing dimensions or byte counts; - unsupported MXFP4 placement. 4. Keep unknown tensor kinds rejected. Do not weaken existing validation to make type 39 load. ## Phase 2: integrate model loading and placement 1. Teach the model-weight validation layer which DeepSeek tensor roles may use MXFP4, matching DS4 exactly. 2. Support the format in resident Metal loading and the existing SSD-streaming/expert-cache path where DS4 supports it. 3. Preserve mmap lifetimes, tensor offsets, page/range validation, streaming cache budgets, and admission accounting. 4. Report an explicit model-load error when an MXFP4 layout is valid GGUF but unsupported by the selected DS4Server model/backend combination. 5. Add model-catalog/download entries only for verified artifacts with confirmed URLs, sizes, and hashes. Format support must not depend on an invented or unverified artifact. ## Phase 3: route the existing Metal implementation The current native Metal snapshot already contains the MXFP4 kernels and wrapper entry points. Wire the Rust execution graph to them under the same tensor, model-shape, device, quality, prefill/decode, and SSD/resident conditions as current DS4. Reuse the capability and M5/pre-M5 policy implemented by #78. MXFP4 exact fast paths must activate automatically only on supported hardware and shapes. Unsupported combinations must either use DS4's exact portable path or fail clearly if DS4 itself does not provide a valid fallback. Do not duplicate general Q2 decode-fusion work from #78. Keep MXFP4-specific scheduling in this issue and share only the already-established Rust FFI and device-policy helpers. ## Phase 4: numerical and behavioral validation 1. Port or reuse the smallest authoritative scalar/reference checks from DS4 for MXFP4 block decoding and dot products. 2. Compare DS4 and DS4Server with the same hashed MXFP4 GGUF and identical prompts/settings. 3. Verify deterministic greedy token parity and relevant intermediate/reference outputs for: - short prefill and decode; - representative long prefill; - resident execution; - SSD-streaming execution when supported; - DSpark/MTP execution where the model supports it; - cancellation and teardown. 4. Exercise both portable and hardware-specific optimized paths when suitable test hardware is available. 5. Ensure cache accounting, context accounting, KV state, and model teardown remain unchanged. ## Phase 5: performance characterization Benchmark MXFP4 against current DS4 on the same machine and against DS4Server's closest supported quantization using the same methodology as #78: - separate prefill, first-token, and stable decode measurements; - repeated sequential runs with cooldown; - identical context, prompt, sampling, power mode, and cache settings; - peak memory, memory pressure, swap, SSD bytes, and cache metrics; - no simultaneous DS4 and DS4Server run. The purpose is to verify that DS4Server receives the native MXFP4 gains and does not merely parse the format. Do not make MXFP4 the default artifact solely because it loads; default selection requires measured quality, memory, and speed evidence. ## Acceptance criteria - DS4Server accepts valid GGUF tensor type 39 using the exact current DS4 layout and rejects malformed layouts safely. - Supported DeepSeek MXFP4 models load through the Rust model layer without bypassing existing validation. - Resident and supported SSD-streaming execution reach the existing native MXFP4 kernels. - M5/pre-M5 optimized paths follow current DS4 device and shape gates; no M5-only path runs on unsupported hardware. - Deterministic outputs and state transitions match current DS4 for the same MXFP4 artifact and settings. - Portable/reference tests cover MXFP4 block interpretation independently of the optimized Metal path. - Benchmarks demonstrate the resulting prefill/decode performance and memory behavior relative to current DS4 and the nearest existing DS4Server quantization. - Model catalog entries, if added, use verified artifacts and hashes. - cargo fmt --all -- --check, Clippy with -D warnings, make bundle, and cargo test --all-features pass. ## Non-goals - Blocking or expanding the critical Q2/M5 performance work in #78. - CUDA, ROCm, Mojo, or tensor-parallel MXFP4 support. - Maintaining a separate DS4Server GGUF conversion tool unless a concrete distribution workflow requires it. - Changing the default model quantization without benchmark and quality evidence. - Replacing the existing GGUF container or splitting models into per-expert files.

hugo added the enhancement label 2026-08-30 10:57:35 +00:00

hugo referenced this issue

2026-08-30 13:27:21 +00:00

Reach current DS4 performance parity on Apple Metal from the Rust execution path #78

hugo commented

2026-08-30 18:43:43 +00:00

Implemented and verified in commit 28231e1 (Add end-to-end MXFP4 Metal support).

Oracle and artifact

Rechecked clean DS4 revision 8db89fe083ae4d17c9a2428ccd29803d3ae8f577.
Followed the MXFP4 work from 725b084, 1f862bd, 7bec128, bcd1e57, f75a065, and 0ad494e.
Tested artifact: DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf
Size: 155976458848 bytes.
SHA-256: 0e3a161b670f686128ec5f92a601dfde616a37bf5e7e48999fa2d32471b57ec6.
The verified artifact and checksum are preserved in the local DS4Server model catalog. Default model selection was intentionally unchanged.

Implementation

Added exact GGUF type 39 handling with 32 values per 17-byte E8M0/E2M1 block.
Added checked dimension, byte-size, row-alignment, tensor-range, unknown-kind, and supported-role validation. MXFP4 is accepted only for routed DeepSeek expert gate/up/down tensors; dense and unsupported placement still fail clearly.
Routed resident and SSD-streamed expert tensors through the existing DS4 native MXFP4 kernels.
Reproduced DS4 device, quality, support-model, shape, early-decode, and pre-M5 fast-lookup guards.
Preserved span isolation and corrected active-span maximum tensor accounting for Metal mappings.
Matched the DS4 host graph ordering and F16 shared-down/HC expansion path.
Restored DS4 SSD prefill behavior with eight 1 MiB positional-read workers, whole-layer spans, selected-address policy, one-layer-ahead overlap, and matching disable controls.
Matched DS4 native bridge optimization flags: O3, fast-math, and mcpu=native.

Parity and validation

Independent scalar checks cover all MXFP4 code points and E8M0 scale edge values.
DS4 current native MXFP4 harness passed against the DS4Server native snapshot in resident and SSD modes: all 4096 LUT entries exact; gate/up exact; maximum intermediate error 2.98e-08; output error 1.49e-07; prefill output error 1.19e-07; fused MoE PASS.
With identical empty system prompts and settings, short greedy output matched DS4 token-for-token. Long 2048-token prefill plus 128 greedy tokens also matched exactly; DS4Server token-stream SHA-256 was 3c6d07eb712892cd650c1e3e03b395025422b7637f4791070cf96bc6655d129e.
Real SSD long-context and cancellation/reset test passed.
Real MXFP4 plus DSpark greedy target/strict verification and exact-sampling verification passed. No legacy MTP artifact is installed for this checkpoint, so live legacy-MTP execution was not claimed.
The 156 GB checkpoint cannot run resident on this 128 GB machine. Resident dispatch and numerics were therefore verified using the authoritative DS4 synthetic harness; the real model was tested through its supported SSD-streaming path.

Performance and root cause

Same Apple M5 Max, same model, prompt, context, sampling, power mode, and expert-cache methodology; DS4 and DS4Server were run sequentially.
Final DS4Server MXFP4: 148.85 to 150.26 prefill tok/s and 15.91 to 16.00 stable decode tok/s.
Current DS4 reference: about 160.55 prefill tok/s and 16.18 stable decode tok/s; individual prefill runs showed normal SSD variance, while native layer timings matched.
Layer 4 routed gate/up/down timing matched directly: DS4Server gate/up 60.187 ms and down 19.453 ms; DS4 gate/up 61.412 ms and down 18.892 ms.
Closest prior DS4Server Q2 control: about 153 prefill tok/s and 19.585 stable decode tok/s.
The apparent large gap was not caused by different Metal kernels. It came from missing mcpu=native on the Objective-C bridge, page faults instead of DS4 prefetching, a 4096-expert benchmark cache versus DS4 automatic 5737 experts, and different host graph ordering. After fixing those causes and matching cache policy, decode and native stage timings converged.
Memory report: 0.22 GiB runtime, 71.43 GiB expert cache, 71.97 GiB task footprint, 71.46 GiB resident, 154.45 GiB mmap spans without copying the model, and 32.65 GiB expert-cache pread traffic. No full-model residency was requested.

Verification gates

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
make bundle
cargo test --all-features: 204 passed, 0 failed, 17 ignored

The final review against issue #79 and DS4 found no unimplemented acceptance item beyond the explicitly documented lack of enough RAM for a real full-resident 156 GB run.

Implemented and verified in commit 28231e1 (Add end-to-end MXFP4 Metal support). Oracle and artifact - Rechecked clean DS4 revision 8db89fe083ae4d17c9a2428ccd29803d3ae8f577. - Followed the MXFP4 work from 725b084, 1f862bd, 7bec128, bcd1e57, f75a065, and 0ad494e. - Tested artifact: DeepSeek-V4-Flash-MXFP4Experts-F16HC-F16Compressor-F16Indexer-Q8Attn-Q8Shared-Q8Out-chat-v2-mxfp4-0731.gguf - Size: 155976458848 bytes. - SHA-256: 0e3a161b670f686128ec5f92a601dfde616a37bf5e7e48999fa2d32471b57ec6. - The verified artifact and checksum are preserved in the local DS4Server model catalog. Default model selection was intentionally unchanged. Implementation - Added exact GGUF type 39 handling with 32 values per 17-byte E8M0/E2M1 block. - Added checked dimension, byte-size, row-alignment, tensor-range, unknown-kind, and supported-role validation. MXFP4 is accepted only for routed DeepSeek expert gate/up/down tensors; dense and unsupported placement still fail clearly. - Routed resident and SSD-streamed expert tensors through the existing DS4 native MXFP4 kernels. - Reproduced DS4 device, quality, support-model, shape, early-decode, and pre-M5 fast-lookup guards. - Preserved span isolation and corrected active-span maximum tensor accounting for Metal mappings. - Matched the DS4 host graph ordering and F16 shared-down/HC expansion path. - Restored DS4 SSD prefill behavior with eight 1 MiB positional-read workers, whole-layer spans, selected-address policy, one-layer-ahead overlap, and matching disable controls. - Matched DS4 native bridge optimization flags: O3, fast-math, and mcpu=native. Parity and validation - Independent scalar checks cover all MXFP4 code points and E8M0 scale edge values. - DS4 current native MXFP4 harness passed against the DS4Server native snapshot in resident and SSD modes: all 4096 LUT entries exact; gate/up exact; maximum intermediate error 2.98e-08; output error 1.49e-07; prefill output error 1.19e-07; fused MoE PASS. - With identical empty system prompts and settings, short greedy output matched DS4 token-for-token. Long 2048-token prefill plus 128 greedy tokens also matched exactly; DS4Server token-stream SHA-256 was 3c6d07eb712892cd650c1e3e03b395025422b7637f4791070cf96bc6655d129e. - Real SSD long-context and cancellation/reset test passed. - Real MXFP4 plus DSpark greedy target/strict verification and exact-sampling verification passed. No legacy MTP artifact is installed for this checkpoint, so live legacy-MTP execution was not claimed. - The 156 GB checkpoint cannot run resident on this 128 GB machine. Resident dispatch and numerics were therefore verified using the authoritative DS4 synthetic harness; the real model was tested through its supported SSD-streaming path. Performance and root cause - Same Apple M5 Max, same model, prompt, context, sampling, power mode, and expert-cache methodology; DS4 and DS4Server were run sequentially. - Final DS4Server MXFP4: 148.85 to 150.26 prefill tok/s and 15.91 to 16.00 stable decode tok/s. - Current DS4 reference: about 160.55 prefill tok/s and 16.18 stable decode tok/s; individual prefill runs showed normal SSD variance, while native layer timings matched. - Layer 4 routed gate/up/down timing matched directly: DS4Server gate/up 60.187 ms and down 19.453 ms; DS4 gate/up 61.412 ms and down 18.892 ms. - Closest prior DS4Server Q2 control: about 153 prefill tok/s and 19.585 stable decode tok/s. - The apparent large gap was not caused by different Metal kernels. It came from missing mcpu=native on the Objective-C bridge, page faults instead of DS4 prefetching, a 4096-expert benchmark cache versus DS4 automatic 5737 experts, and different host graph ordering. After fixing those causes and matching cache policy, decode and native stage timings converged. - Memory report: 0.22 GiB runtime, 71.43 GiB expert cache, 71.97 GiB task footprint, 71.46 GiB resident, 154.45 GiB mmap spans without copying the model, and 32.65 GiB expert-cache pread traffic. No full-model residency was requested. Verification gates - cargo fmt --all -- --check - cargo clippy --all-targets --all-features -- -D warnings - make bundle - cargo test --all-features: 204 passed, 0 failed, 17 ignored The final review against issue #79 and DS4 found no unimplemented acceptance item beyond the explicitly documented lack of enough RAM for a real full-resident 156 GB run.

hugo closed this issue

2026-08-30 18:44:42 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#79