Optional follow-up: add CUDA to the completed Mojo backend #92

New Issue

hugo · 2026-08-31T12:31:05Z

hugo commented

2026-08-31 12:31:05 +00:00

Goal

After the macOS and Metal transition is complete in #91, extend the proven shared Mojo model and operation code to NVIDIA CUDA without delaying or destabilizing the main work.

This is an optional, last-in-sequence issue. It is blocked by #91 and is not part of the Metal definition of done.

Principles

Preserve the completed Metal behavior and performance. CUDA work may not force speculative abstractions or regress Apple execution.
Reuse the same model and semantic operation source wherever hardware permits.
Add NVIDIA behavior through narrow compile-time target policies and measured micro-kernel specializations, not copied DeepSeek, Qwen, or GLM executors.
DS4 remains the behavioral oracle where it supports the model. The pinned DS4 CUDA path is the same-hardware performance oracle on DGX Spark.
For Qwen3.8, use the official configuration and pinned identical-GGUF CUDA reference established by the Metal issue.

Scope

Revalidate and pin the Mojo toolchain, CUDA compiler, target architecture, and DGX Spark environment independently of the earlier Apple pin.
Extend build.rs to build the same Mojo source as a Linux shared library and add only the target/nvidia policies and CUDA-specific kernels required by measurements.
Reuse the #85 result schema, fixtures, operation inventory, and measurement protocol. Add native-cuda and mojo-cuda adapters without changing completed Metal results.
Implement device lifecycle, memory, streams, events, tensor-core policies, transfers, telemetry, and packaging for Linux.
Port and tune every required production operation for supported DeepSeek, GLM 5.2, and Qwen3.8 Q4 and Q8 paths, including long context, streaming, checkpoints, steering, MTP, and multimodal modes that are already complete on Metal.
Keep common quantization, matrix, attention, cache, state, routing, model composition, and checkpoint semantics shared. Add target-specific micro-kernels only where profiling proves they are necessary.
Produce a reproducible native DS4 CUDA reference run from the same model, prompt, options, and manifest when an in-process adapter is not practical.

MAX and kernel decision

Repeat the #84 component decision for the actual CUDA artifact. A MAX component is usable only if:

its exact source and runtime closure is redistributable under the approved license policy; and
it meets or exceeds the pinned native CUDA reference for the operation, full layer, memory, transfers, and end-to-end model path.

If either gate fails, implement a DS4Server-owned Mojo CUDA kernel. Use the existing DS4 CUDA implementation as the same-hardware performance reference and preserve the layouts and fusion strategy already proven by the Metal implementation where appropriate.

CUDA choices may not introduce a new dependency into the macOS artifact or slow the existing Metal path.

Performance and correctness gates

Run only on real NVIDIA hardware; do not infer CUDA from Metal.
Compare Mojo-CUDA only with native CUDA on the same DGX Spark using identical model hashes, tokenized prompts, options, warm-up, and alternating run order.
Require the #85 thresholds for critical operations, complete layers, prefill, time to first token, stable decode, p95 latency, memory, transfers, and cold start.
Verify shapes, quantization, attention, expert routing, state caches, checkpoints, logits, sampled and greedy tokens, and every supported optional mode.
Reject silent fallback, mixed native and Mojo measurements, extra token-path transfers, precision changes, or persistent regressions.
Re-run the relevant Metal matrix and prove CUDA additions do not alter Metal tokens, artifacts, memory plans, or performance.

Linux packaging and license gate

Produce a Linux artifact from pinned inputs and an offline frozen environment after bootstrap.
Use only relative release RPaths or system libraries and inspect the finished artifact with readelf, ldd, and nm.
Prove execution without the source checkout or project virtual environment.
Inventory every non-system library, CUDA or Mojo runtime file, source component, license, redistribution status, notice, and open alternative.
Unknown, non-redistributable, or unapproved dependencies block release.

Acceptance criteria

DeepSeek, GLM 5.2, and Qwen3.8 supported modes pass the full CUDA correctness and performance matrix on DGX Spark.
Mojo-CUDA is not slower than native CUDA on the same hardware.
The Metal path remains behaviorally and statistically unchanged.
CUDA implementation is confined to target policy, micro-kernel, build, runtime, and packaging differences; model executors are not duplicated.
Raw results, fixtures, binary closure, licenses, and Linux artifacts are reproducible.

## Goal After the macOS and Metal transition is complete in #91, extend the proven shared Mojo model and operation code to NVIDIA CUDA without delaying or destabilizing the main work. This is an optional, last-in-sequence issue. It is blocked by #91 and is not part of the Metal definition of done. ## Principles - Preserve the completed Metal behavior and performance. CUDA work may not force speculative abstractions or regress Apple execution. - Reuse the same model and semantic operation source wherever hardware permits. - Add NVIDIA behavior through narrow compile-time target policies and measured micro-kernel specializations, not copied DeepSeek, Qwen, or GLM executors. - DS4 remains the behavioral oracle where it supports the model. The pinned DS4 CUDA path is the same-hardware performance oracle on DGX Spark. - For Qwen3.8, use the official configuration and pinned identical-GGUF CUDA reference established by the Metal issue. ## Scope - Revalidate and pin the Mojo toolchain, CUDA compiler, target architecture, and DGX Spark environment independently of the earlier Apple pin. - Extend build.rs to build the same Mojo source as a Linux shared library and add only the target/nvidia policies and CUDA-specific kernels required by measurements. - Reuse the #85 result schema, fixtures, operation inventory, and measurement protocol. Add native-cuda and mojo-cuda adapters without changing completed Metal results. - Implement device lifecycle, memory, streams, events, tensor-core policies, transfers, telemetry, and packaging for Linux. - Port and tune every required production operation for supported DeepSeek, GLM 5.2, and Qwen3.8 Q4 and Q8 paths, including long context, streaming, checkpoints, steering, MTP, and multimodal modes that are already complete on Metal. - Keep common quantization, matrix, attention, cache, state, routing, model composition, and checkpoint semantics shared. Add target-specific micro-kernels only where profiling proves they are necessary. - Produce a reproducible native DS4 CUDA reference run from the same model, prompt, options, and manifest when an in-process adapter is not practical. ## MAX and kernel decision Repeat the #84 component decision for the actual CUDA artifact. A MAX component is usable only if: 1. its exact source and runtime closure is redistributable under the approved license policy; and 2. it meets or exceeds the pinned native CUDA reference for the operation, full layer, memory, transfers, and end-to-end model path. If either gate fails, implement a DS4Server-owned Mojo CUDA kernel. Use the existing DS4 CUDA implementation as the same-hardware performance reference and preserve the layouts and fusion strategy already proven by the Metal implementation where appropriate. CUDA choices may not introduce a new dependency into the macOS artifact or slow the existing Metal path. ## Performance and correctness gates - Run only on real NVIDIA hardware; do not infer CUDA from Metal. - Compare Mojo-CUDA only with native CUDA on the same DGX Spark using identical model hashes, tokenized prompts, options, warm-up, and alternating run order. - Require the #85 thresholds for critical operations, complete layers, prefill, time to first token, stable decode, p95 latency, memory, transfers, and cold start. - Verify shapes, quantization, attention, expert routing, state caches, checkpoints, logits, sampled and greedy tokens, and every supported optional mode. - Reject silent fallback, mixed native and Mojo measurements, extra token-path transfers, precision changes, or persistent regressions. - Re-run the relevant Metal matrix and prove CUDA additions do not alter Metal tokens, artifacts, memory plans, or performance. ## Linux packaging and license gate - Produce a Linux artifact from pinned inputs and an offline frozen environment after bootstrap. - Use only relative release RPaths or system libraries and inspect the finished artifact with readelf, ldd, and nm. - Prove execution without the source checkout or project virtual environment. - Inventory every non-system library, CUDA or Mojo runtime file, source component, license, redistribution status, notice, and open alternative. - Unknown, non-redistributable, or unapproved dependencies block release. ## Acceptance criteria - DeepSeek, GLM 5.2, and Qwen3.8 supported modes pass the full CUDA correctness and performance matrix on DGX Spark. - Mojo-CUDA is not slower than native CUDA on the same hardware. - The Metal path remains behaviorally and statistically unchanged. - CUDA implementation is confined to target policy, micro-kernel, build, runtime, and packaging differences; model executors are not duplicated. - Raw results, fixtures, binary closure, licenses, and Linux artifacts are reproducible.

hugo added this to the Mojo GPU Backend Transition milestone 2026-08-31 12:31:05 +00:00

hugo added the idea label 2026-08-31 12:31:05 +00:00

hugo removed this from the Mojo GPU Backend Transition milestone 2026-08-31 12:31:54 +00:00

hugo commented

2026-08-31 14:02:10 +00:00

Based on findings in #84, all work on MOJO migration is stopped. Licene-hell.

hugo closed this issue

2026-08-31 14:02:10 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#92