Add a performance-parity Qwen3.8-27B Q4 and Q8 text backend on Metal #88

New Issue

hugo · 2026-08-31T12:13:27Z

hugo commented

2026-08-31 12:13:27 +00:00

Goal

Add the Qwen3.8-27B text model to the shared Mojo backend on Apple Metal, proving that common operations are reusable and that Q4 and Q8 preserve correctness and native-Metal performance.

Blocked by #84 through #87. CUDA is deferred to the final optional issue.

Model and reference contract

Target the official Qwen/Qwen3.8-27B configuration, including its 64 layers, hidden size 5120, repeated three Gated DeltaNet plus one full-attention pattern, model-specific heads, FFN size 17408, native long context, and separate MTP and vision capabilities.
Generate GGUF artifacts reproducibly from the official model. Pin source revision, converter commit and command, format, and SHA-256.
Use Q4_K_M, or the exact equivalent from the pinned converter, as the primary local performance and memory path.
Implement Q8_0 as the higher-precision reference and operating path. Require full performance parity only where the complete working set stays resident; elsewhere retain smaller correctness fixtures.
Because DS4 has no Qwen3.8 path, use the official model configuration plus a pinned Qwen3.8-capable native Metal reference executor running the identical GGUF.

Scope

Reuse shared embedding, norm, quantized matrix, FFN, RoPE, full attention, KV cache, LM head, telemetry, and checkpoint operations.
Add reusable causal depthwise Conv1D, Gated DeltaNet recurrence, and recurrent-state cache. Keep recurrent state semantically separate from KV cache.
Implement separately tuned GDN prefill and single-step decode behind one semantic interface. Evaluate MAX GDN only under the #84 license and Metal performance gates.
Compose the Qwen-specific hybrid schedule and state mapping under models/qwen3_8 without copying shared operations.
Implement text load, prefill, decode, Q4 and Q8 weight binding, combined recurrent and KV checkpoint mapping, and exact token generation.
Leave MTP and vision disabled for #89.
Preserve target-neutral data contracts where practical, but add no CUDA build, NVIDIA tuning, or Linux requirement.

Measurements and gates

Isolate causal Conv1D, GDN prefill, GDN decode, recurrent-state update, full attention, the three-GDN-plus-one-attention hybrid block, and the complete model. Measure short contexts and decode positions at 64K, 80K, 96K, 128K, and 262K when memory permits, investigating rather than averaging any long-context cliff.

Q4 must pass correctness and complete performance parity against the pinned native Metal reference on the same Mac. Q8 must pass the same gates where resident and the defined smaller correctness suite elsewhere. Tokens, state transitions, checkpoints, and decisive logits must match the identical-GGUF reference.

Common quantization, matrix, norm, attention, cache, and FFN code must be reused. Update coverage, fixture provenance, raw results, binary closure, and licenses.

Upstream material to reassess

Binding outcome from #84 (2026-08-31)

#84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal DeviceContext. The working max.gpu.host path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. mojo build --emit shared-lib also links @rpath/libKGENCompilerRTShared.dylib for a trivial export.

The Qwen text port cannot begin on MAX-backed GPU APIs. Model inventory and native fixtures may proceed, but no Mojo completion or performance claim is possible before the shared runtime gate passes.

These constraints are acceptance criteria:

keep Mojo pinned to 1.0.0 through the committed uv.lock; do not use nightly or an older release;
do not add max, max-core, max-mojo-libs, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component;
do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries;
target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes;
before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in docs/MOJO_SPIKE_84.md;
CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision.

If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.

## Goal Add the Qwen3.8-27B text model to the shared Mojo backend on Apple Metal, proving that common operations are reusable and that Q4 and Q8 preserve correctness and native-Metal performance. Blocked by #84 through #87. CUDA is deferred to the final optional issue. ## Model and reference contract - Target the official Qwen/Qwen3.8-27B configuration, including its 64 layers, hidden size 5120, repeated three Gated DeltaNet plus one full-attention pattern, model-specific heads, FFN size 17408, native long context, and separate MTP and vision capabilities. - Generate GGUF artifacts reproducibly from the official model. Pin source revision, converter commit and command, format, and SHA-256. - Use Q4_K_M, or the exact equivalent from the pinned converter, as the primary local performance and memory path. - Implement Q8_0 as the higher-precision reference and operating path. Require full performance parity only where the complete working set stays resident; elsewhere retain smaller correctness fixtures. - Because DS4 has no Qwen3.8 path, use the official model configuration plus a pinned Qwen3.8-capable native Metal reference executor running the identical GGUF. ## Scope - Reuse shared embedding, norm, quantized matrix, FFN, RoPE, full attention, KV cache, LM head, telemetry, and checkpoint operations. - Add reusable causal depthwise Conv1D, Gated DeltaNet recurrence, and recurrent-state cache. Keep recurrent state semantically separate from KV cache. - Implement separately tuned GDN prefill and single-step decode behind one semantic interface. Evaluate MAX GDN only under the #84 license and Metal performance gates. - Compose the Qwen-specific hybrid schedule and state mapping under models/qwen3_8 without copying shared operations. - Implement text load, prefill, decode, Q4 and Q8 weight binding, combined recurrent and KV checkpoint mapping, and exact token generation. - Leave MTP and vision disabled for #89. - Preserve target-neutral data contracts where practical, but add no CUDA build, NVIDIA tuning, or Linux requirement. ## Measurements and gates Isolate causal Conv1D, GDN prefill, GDN decode, recurrent-state update, full attention, the three-GDN-plus-one-attention hybrid block, and the complete model. Measure short contexts and decode positions at 64K, 80K, 96K, 128K, and 262K when memory permits, investigating rather than averaging any long-context cliff. Q4 must pass correctness and complete performance parity against the pinned native Metal reference on the same Mac. Q8 must pass the same gates where resident and the defined smaller correctness suite elsewhere. Tokens, state transitions, checkpoints, and decisive logits must match the identical-GGUF reference. Common quantization, matrix, norm, attention, cache, and FFN code must be reused. Update coverage, fixture provenance, raw results, binary closure, and licenses. ## Upstream material to reassess - https://huggingface.co/Qwen/Qwen3.8-27B - https://github.com/modular/modular/issues/5772 - https://github.com/modular/modular/blob/main/max/kernels/src/state_space/gated_delta.mojo - https://github.com/modular/modular/blob/main/max/kernels/src/state_space/gated_delta_conv1d.mojo - https://github.com/ggml-org/llama.cpp/issues/27623 ## Binding outcome from #84 (2026-08-31) #84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal `DeviceContext`. The working `max.gpu.host` path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. `mojo build --emit shared-lib` also links `@rpath/libKGENCompilerRTShared.dylib` for a trivial export. The Qwen text port cannot begin on MAX-backed GPU APIs. Model inventory and native fixtures may proceed, but no Mojo completion or performance claim is possible before the shared runtime gate passes. These constraints are acceptance criteria: - keep Mojo pinned to 1.0.0 through the committed `uv.lock`; do not use nightly or an older release; - do not add `max`, `max-core`, `max-mojo-libs`, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component; - do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries; - target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes; - before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in `docs/MOJO_SPIKE_84.md`; - CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision. If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.

hugo added the idea label 2026-08-31 12:13:27 +00:00

hugo referenced this issue

2026-08-31 12:13:28 +00:00

Add Qwen3.8 MTP and multimodal inference on Metal without regressing text #89

hugo added this to the Mojo GPU Backend Transition milestone 2026-08-31 12:14:25 +00:00

hugo changed title from ~~Add a performance-parity Qwen3.8-27B Q4 and Q8 text backend in Mojo~~ to Add a performance-parity Qwen3.8-27B Q4 and Q8 text backend on Metal

2026-08-31 12:29:54 +00:00

hugo commented

2026-08-31 14:01:30 +00:00

Based on findings in #84, all work on MOJO migration is stopped. Licene-hell.

hugo closed this issue

2026-08-31 14:01:31 +00:00

hugo commented

2026-08-31 15:31:51 +00:00

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in 9a33c61; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in `9a33c61`; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#88