Add Qwen3.8 MTP and multimodal inference on Metal without regressing text #89

New Issue

hugo · 2026-08-31T12:13:28Z

hugo commented

2026-08-31 12:13:28 +00:00

Goal

Complete Qwen3.8-27B on Apple Metal by adding MTP and image or video inference as separately measurable extensions after #88.

CUDA is deferred and does not block this issue.

Scope and order

Implement and accept these steps sequentially:

Qwen-specific MTP heads and shared verify scheduling.
Image preprocessing and visual tokenization in Rust.
Vision encoder and projector in Mojo.
Combined visual-token and text prefill through the existing text model.
Video only if it belongs to the DS4Server product scope when implemented.

Keep model weights and heads under models/qwen3_8. Move a vision operation into shared ops only after a second real consumer needs identical semantics. Do not add speculative cross-platform or universal vision abstractions.

Correctness gates

MTP candidate generation, verification, cache updates, rollback, accepted tokens, and output match the pinned Qwen Metal reference.
Image preprocessing, visual-token order, projection, context accounting, checkpoints, and multimodal output match within operation-specific tolerances.
Text-only requests use the existing text path and retain the same tokens and state when MTP or visual input is absent.
Do not describe Qwen as fully multimodal until vision passes end-to-end.

Performance gates

Measure baseline decode, MTP draft and verify work, accepted-token throughput, memory, image preprocessing, vision encoder, projector, combined prefill, transfers, and cold start separately on the same Mac. Mojo-Metal must meet the native Metal reference for each enabled feature. A faster text backend cannot hide a slower MTP or vision stage, and absent visual input may not add text-path work.

Acceptance criteria

Qwen MTP passes Metal correctness and performance parity.
Representative image and image-text prompts pass Metal correctness, memory, and prefill parity.
Video is either covered by equivalent Metal gates or explicitly unsupported.
Coverage, fixture provenance, raw results, binary closure, and licenses are updated.
No native fallback or mixed backend execution appears in a claimed Mojo result.
Data contracts remain suitable for the later optional CUDA issue without adding CUDA work here.

Binding outcome from #84 (2026-08-31)

#84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal DeviceContext. The working max.gpu.host path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. mojo build --emit shared-lib also links @rpath/libKGENCompilerRTShared.dylib for a trivial export.

MTP, multimodal, and Gated DeltaNet design work must retain the same backend boundary. Do not introduce MAX to obtain missing host-side GPU functionality.

These constraints are acceptance criteria:

keep Mojo pinned to 1.0.0 through the committed uv.lock; do not use nightly or an older release;
do not add max, max-core, max-mojo-libs, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component;
do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries;
target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes;
before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in docs/MOJO_SPIKE_84.md;
CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision.

If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.

## Goal Complete Qwen3.8-27B on Apple Metal by adding MTP and image or video inference as separately measurable extensions after #88. CUDA is deferred and does not block this issue. ## Scope and order Implement and accept these steps sequentially: 1. Qwen-specific MTP heads and shared verify scheduling. 2. Image preprocessing and visual tokenization in Rust. 3. Vision encoder and projector in Mojo. 4. Combined visual-token and text prefill through the existing text model. 5. Video only if it belongs to the DS4Server product scope when implemented. Keep model weights and heads under models/qwen3_8. Move a vision operation into shared ops only after a second real consumer needs identical semantics. Do not add speculative cross-platform or universal vision abstractions. ## Correctness gates - MTP candidate generation, verification, cache updates, rollback, accepted tokens, and output match the pinned Qwen Metal reference. - Image preprocessing, visual-token order, projection, context accounting, checkpoints, and multimodal output match within operation-specific tolerances. - Text-only requests use the existing text path and retain the same tokens and state when MTP or visual input is absent. - Do not describe Qwen as fully multimodal until vision passes end-to-end. ## Performance gates Measure baseline decode, MTP draft and verify work, accepted-token throughput, memory, image preprocessing, vision encoder, projector, combined prefill, transfers, and cold start separately on the same Mac. Mojo-Metal must meet the native Metal reference for each enabled feature. A faster text backend cannot hide a slower MTP or vision stage, and absent visual input may not add text-path work. ## Acceptance criteria - Qwen MTP passes Metal correctness and performance parity. - Representative image and image-text prompts pass Metal correctness, memory, and prefill parity. - Video is either covered by equivalent Metal gates or explicitly unsupported. - Coverage, fixture provenance, raw results, binary closure, and licenses are updated. - No native fallback or mixed backend execution appears in a claimed Mojo result. - Data contracts remain suitable for the later optional CUDA issue without adding CUDA work here. ## Binding outcome from #84 (2026-08-31) #84 stopped product integration on Mojo 1.0.0 because the open compiler/standard-library environment has no host-side Metal `DeviceContext`. The working `max.gpu.host` path requires MAX plus AsyncRT/KGEN dylibs whose component-specific redistribution permission could not be established. `mojo build --emit shared-lib` also links `@rpath/libKGENCompilerRTShared.dylib` for a trivial export. MTP, multimodal, and Gated DeltaNet design work must retain the same backend boundary. Do not introduce MAX to obtain missing host-side GPU functionality. These constraints are acceptance criteria: - keep Mojo pinned to 1.0.0 through the committed `uv.lock`; do not use nightly or an older release; - do not add `max`, `max-core`, `max-mojo-libs`, AsyncRT, or KGEN to the product/build/package closure without an authoritative redistribution grant naming every shipped component; - do not treat Apache-licensed MAX Mojo source as permission to redistribute separately delivered runtime binaries; - target Apple silicon and macOS 15 or newer, while preserving native Metal as production until the runtime gate passes; - before implementation resumes, re-run the #84 closure, clean-machine packaging, code-signing, and same-workload parity gates documented in `docs/MOJO_SPIKE_84.md`; - CUDA remains a separate Linux/NVIDIA toolchain, runtime-closure, redistribution, and hardware-validation decision. If the runtime grant or open host API changes, update #84 evidence first so this milestone has one source of truth.

hugo added the idea label 2026-08-31 12:13:28 +00:00

hugo added this to the Mojo GPU Backend Transition milestone 2026-08-31 12:14:26 +00:00

hugo referenced this issue

2026-08-31 12:29:54 +00:00

Add a performance-parity Qwen3.8-27B Q4 and Q8 text backend on Metal #88

hugo changed title from ~~Add Qwen3.8 MTP and multimodal inference without regressing the text path~~ to Add Qwen3.8 MTP and multimodal inference on Metal without regressing text

2026-08-31 12:29:54 +00:00

hugo commented

2026-08-31 14:01:39 +00:00

Based on findings in #84, all work on MOJO migration is stopped. Licene-hell.

hugo closed this issue

2026-08-31 14:01:39 +00:00

hugo commented

2026-08-31 15:31:51 +00:00

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in 9a33c61; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.

Project decision: the Mojo backend effort is canceled. All committed spike tooling, dependencies, lockfiles, and documentation were removed in `9a33c61`; generated probe state was deleted as well. The independent Apple-silicon/macOS-15 product floor remains. This issue is no longer actionable.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#89