hugo/DS4Server
1
0
Fork 0
Code Issues 2 Pull Requests Actions Packages Projects Releases Wiki Activity

Build the native-Metal-versus-Mojo parity harness and operation inventory #85

New Issue
Closed
opened 2026-08-31 12:11:29 +00:00 by hugo · 2 comments
hugo commented 2026-08-31 12:11:29 +00:00
Owner
Copy Link

Goal

Create the reproducible Metal correctness and performance framework used by every main Mojo migration issue. DS4 remains the behavioral oracle and DS4Server native Metal is the same-hardware performance oracle.

This work must incorporate #84. It must not switch the normal backend or require CUDA.

Scope

  • Inventory every reachable GPU operation from the Rust Metal executor, GPU bindings, and model executors. Record model consumers, phase, types, shapes, layouts, quantization, and whether the operation is expected to be portable to a later target.
  • Cover allocation and transfers, command lifecycle, embeddings, elementwise and norm work, all matrix and quantized matrix variants, DeepSeek HC and attention, KV paths, routing and experts, GLM sparse/indexed paths, streaming, GPU logit work, MTP, steering, Qwen Gated DeltaNet state, and later vision operations.
  • Add an internal test-only selector for native-metal and mojo-metal. It must never silently fall back or mix backends.
  • Build one shared harness for operation, layer, executor/session, and complete-model comparisons.
  • Record repository commits, toolchains, flags, model and fixture hashes, prompt tokens, hardware, OS, power state, and thermal conditions.
  • Add fixture manifests for real model tensor slices without committing large weights.
  • Capture repeatable native Metal baselines on Apple hardware.

The result schema and operation inventory should be reusable by the later CUDA issue, but no DS4 CUDA adapter, DGX run, Linux support, or CUDA acceptance gate belongs here.

Measurement contract

  • Use release builds, fixed tokenized prompts, greedy decode, fixed RNG state, warm-up, and alternating A/B/B/A or B/A/A/B ordering.
  • Separate compilation, model load, first upload, prefill, time to first token, stable decode, and cold start.
  • Report median, p95, dispersion, paired confidence intervals, peak and resident GPU memory, transfer bytes, and command-buffer counts.
  • Use GPU event time for kernels and Rust wall-clock time for layers and end-to-end execution.
  • Cover short and long cache positions, prefill at 1, 128, 512, 2048, and 8192 tokens where supported, relevant layout transitions, resident and streamed experts, and every production quantization format.

Correctness contract

Compare shapes, strides, quantization decoding, expert indices, cache positions, context counters, checkpoints, logits, argmax, sampled tokens with identical RNG, and full greedy sequences. Use operation-specific tolerances. Reject NaN, infinity, out-of-bounds behavior, silent precision changes, fallback, or extra token-path transfers.

Performance gate

Critical operations, complete-model prefill, and stable decode require a Mojo-Metal versus native-Metal median ratio of at least 1.00 on the same Mac. The paired 95 percent confidence lower bound must be at least 0.97, but that uncertainty band is not an allowed persistent regression. p95 latency must not reproducibly worsen and peak GPU memory must not exceed native without explicit approval.

Acceptance criteria

  • Native results before and after adding the selector are statistically equivalent.
  • Raw JSON or CSV and generated Markdown reports are reproducible from one command.
  • Every native Metal operation has one coverage entry or a documented removed-materialization explanation.
  • Hardware-dependent tests cannot be reported complete without the real Apple run.
  • #84 findings are reflected in fixtures, target boundaries, and thresholds.
  • Future CUDA can add another backend to the same result contract without changing the completed Metal semantics or slowing the Metal harness.

Binding outcome from #84 (2026-08-31)

#84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal DeviceContext. The working max.gpu.host path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. mojo build --emit shared-lib also links @rpath/libKGENCompilerRTShared.dylib for a trivial export.

The operation inventory, native baseline, fixture schema, and backend-neutral result format may proceed. A Mojo backend, Mojo timing comparison, and completion claim are blocked until every resume condition in docs/MOJO_SPIKE_84.md passes.

These constraints are acceptance criteria:

  • keep Mojo pinned to 1.0.0 through the committed uv.lock; do not use nightly or an older release;
  • do not add max, max-core, max-mojo-libs, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component;
  • do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries;
  • target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes;
  • before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in docs/MOJO_SPIKE_84.md;
  • CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision.

If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.

## Goal Create the reproducible Metal correctness and performance framework used by every main Mojo migration issue. DS4 remains the behavioral oracle and DS4Server native Metal is the same-hardware performance oracle. This work must incorporate #84. It must not switch the normal backend or require CUDA. ## Scope - Inventory every reachable GPU operation from the Rust Metal executor, GPU bindings, and model executors. Record model consumers, phase, types, shapes, layouts, quantization, and whether the operation is expected to be portable to a later target. - Cover allocation and transfers, command lifecycle, embeddings, elementwise and norm work, all matrix and quantized matrix variants, DeepSeek HC and attention, KV paths, routing and experts, GLM sparse/indexed paths, streaming, GPU logit work, MTP, steering, Qwen Gated DeltaNet state, and later vision operations. - Add an internal test-only selector for native-metal and mojo-metal. It must never silently fall back or mix backends. - Build one shared harness for operation, layer, executor/session, and complete-model comparisons. - Record repository commits, toolchains, flags, model and fixture hashes, prompt tokens, hardware, OS, power state, and thermal conditions. - Add fixture manifests for real model tensor slices without committing large weights. - Capture repeatable native Metal baselines on Apple hardware. The result schema and operation inventory should be reusable by the later CUDA issue, but no DS4 CUDA adapter, DGX run, Linux support, or CUDA acceptance gate belongs here. ## Measurement contract - Use release builds, fixed tokenized prompts, greedy decode, fixed RNG state, warm-up, and alternating A/B/B/A or B/A/A/B ordering. - Separate compilation, model load, first upload, prefill, time to first token, stable decode, and cold start. - Report median, p95, dispersion, paired confidence intervals, peak and resident GPU memory, transfer bytes, and command-buffer counts. - Use GPU event time for kernels and Rust wall-clock time for layers and end-to-end execution. - Cover short and long cache positions, prefill at 1, 128, 512, 2048, and 8192 tokens where supported, relevant layout transitions, resident and streamed experts, and every production quantization format. ## Correctness contract Compare shapes, strides, quantization decoding, expert indices, cache positions, context counters, checkpoints, logits, argmax, sampled tokens with identical RNG, and full greedy sequences. Use operation-specific tolerances. Reject NaN, infinity, out-of-bounds behavior, silent precision changes, fallback, or extra token-path transfers. ## Performance gate Critical operations, complete-model prefill, and stable decode require a Mojo-Metal versus native-Metal median ratio of at least 1.00 on the same Mac. The paired 95 percent confidence lower bound must be at least 0.97, but that uncertainty band is not an allowed persistent regression. p95 latency must not reproducibly worsen and peak GPU memory must not exceed native without explicit approval. ## Acceptance criteria - Native results before and after adding the selector are statistically equivalent. - Raw JSON or CSV and generated Markdown reports are reproducible from one command. - Every native Metal operation has one coverage entry or a documented removed-materialization explanation. - Hardware-dependent tests cannot be reported complete without the real Apple run. - #84 findings are reflected in fixtures, target boundaries, and thresholds. - Future CUDA can add another backend to the same result contract without changing the completed Metal semantics or slowing the Metal harness. ## Binding outcome from #84 (2026-08-31) #84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal `DeviceContext`. The working `max.gpu.host` path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. `mojo build --emit shared-lib` also links `@rpath/libKGENCompilerRTShared.dylib` for a trivial export. The operation inventory, native baseline, fixture schema, and backend-neutral result format may proceed. A Mojo backend, Mojo timing comparison, and completion claim are blocked until every resume condition in `docs/MOJO_SPIKE_84.md` passes. These constraints are acceptance criteria: - keep Mojo pinned to 1.0.0 through the committed `uv.lock`; do not use nightly or an older release; - do not add `max`, `max-core`, `max-mojo-libs`, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component; - do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries; - target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes; - before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in `docs/MOJO_SPIKE_84.md`; - CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision. If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.
hugo added the idea label 2026-08-31 12:11:29 +00:00
hugo added this to the Mojo GPU Backend Transition milestone 2026-08-31 12:14:25 +00:00
hugo changed title from Build the native-versus-Mojo parity harness and operation coverage inventory to Build the native-Metal-versus-Mojo parity harness and operation inventory 2026-08-31 12:30:21 +00:00
hugo commented 2026-08-31 14:01:01 +00:00
Author
Owner
Copy Link

Based on findings in #84, all work on MOJO migration is stopped. Licene-hell.

Based on findings in #84, all work on MOJO migration is stopped. Licene-hell.
hugo closed this issue 2026-08-31 14:01:01 +00:00
hugo commented 2026-08-31 15:31:50 +00:00
Author
Owner
Copy Link

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in 9a33c61; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in `9a33c61`; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
cannot reproduce
bug
Something is not working
 critical
issues that are more important than others of the same group
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 idea
just an idea
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label idea
Milestone
No items
No Milestone Mojo GPU Backend Transition
Projects
Clear projects
No project
Assignees
Clear assignees
hugo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hugo/DS4Server#85
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?