Keep this issue as an analysis task for Qwen 3.8 Flash Next. Compare our native Rust/Metal implementation with the relevant changes in the latest MTPLX, reproduce actual defects in our active execution paths, and create independently reviewable bug issues for the defects that are confirmed.
The user will review those bug issues with another agent before implementation. Closing this analysis issue does not authorize implementing its child issues. Production fixes belong to the resulting bug issues and begin only after that review and the user's instruction to implement them.
Analysis code, diagnostic instrumentation, small oracle probes and reproduction/regression tests may be created under this issue. Every retained reproduction must be linked or carried into the bug issue that owns its eventual fix. Do not leave analysis-only work as an untracked implementation obligation.
Our pinned MTPLX reference: e652d55e2652137a4abcf1312357abbf3eb9d692, MTPLX 2.10.2, dated 2026-09-01. The later vision work used the same reference.
Latest inspected MTPLX: release 2.12.2 and HEAD 9882703f3105363ddc37eca9f97aa09a1d387112.
MTPLX itself requires mlx==0.32.2 on Apple Silicon. Its commit 0d56bc6d1940906884fc1ec40f97c180a48ed43b pins that version because 0.32.3 produces different Flash-Next output and its release checks used 0.32.2. Our shader runtime is also 0.32.2.
At analysis start, refresh MTPLX HEAD and release tags, record the immutable commit actually used, and include any newer relevant changes. Keep both the old reference and the refreshed reference available for before/after comparisons. Do not silently change model weights, quantization, tokenizer assets or MLX version to obtain a passing comparison.
Record checkout SHA, model/config/tokenizer hashes, runtime/compiler versions, hardware, OS, effective options, prompt/image digests, sampling parameters, seed, context, chunk size and cold/warm state for every reproduction. Compare identical artifacts and workloads. The primary performance target remains the same-machine Bare-Speed/M5 Max contract; published Optimized-Speed figures are candidate evidence, not our acceptance result.
Scope and boundaries
Analyse the active product path from model loading and request rendering through Rust graph execution, Metal kernels, sampling, state capture, session restore and response publication. Trace callers and effective routing before declaring a defect applicable.
MTPLX Python and its tests may run in an isolated reference environment as an oracle. Production application, loading, inference, cache/session and orchestration code remains Rust; GPU work remains Metal with the existing Objective-C bridge. Do not introduce MTPLX/MLX Python or C/C++ host execution into DS4Server.
Preserve DS4 behavioral parity for DSv4 Flash and GLM 5.3 Flash. Qwen 3.8 Flash Next follows the pinned MTPLX oracle instead. Keep shared helpers model-scoped where those contracts differ, and require exact DS4 regression controls for DS4-backed callers; do not impose MTPLX semantics globally. This is the user's explicit model-specific clarification of the blanket DS4 wording in AGENTS.md.
Under this issue, do not implement or enable production fixes, change default tuning, replace shipped shader artifacts, change managed model revisions, or mark an untested candidate as fixed.
Temporary experiments must stay isolated and reversible. Explain any instrumentation effect on synchronization, memory, scheduling or timing. Keep receipts needed for another agent to reproduce the result in durable repository evidence; /tmp paths alone are insufficient for child issues.
Use direct git and tea from this project for delivery and tracker operations. Use ordinary bullets in issue bodies, not task checkboxes.
Stage 1: Build an applicability and evidence inventory
Read relevant MTPLX commits, actual source diffs, release notes and regression tests between our pinned reference and the refreshed latest reference. Search existing DS4Server issues before creating duplicates.
For each candidate record:
Upstream change/test and immutable source reference.
Root cause, trigger and affected upstream version/route.
Corresponding DS4Server code, full caller chain, active/inactive status, effective settings and hardware/dtype constraints.
Preliminary classification: confirmed defect, suspected defect awaiting reproduction, already covered, not applicable, optimization candidate, or blocked by missing evidence.
Required reproduction and exact expected result.
The observations below are starting hypotheses from source inspection, not completed runtime reproductions. Re-evaluate them against the exact analysis checkout.
Stage 2: Analyse and reproduce the following areas
A. Sampling law and invalid logits
Starting evidence: MTPLX a2d0b4a; tests/test_non_finite_logits_and_lazy_sampler_law.py and the follow-up allowing legitimate grammar-masked negative infinity.
Inspect src/engine/metal/qwen_mtplx/sampling.rs, ar.rs, speculative.rs and API/session error handling.
The active AR pipeline appears to compute top-p from the top-k slice's softmax; ordinary sampling already has a full-vocabulary path. Compare the actual support and normalized probabilities across pipeline AR, ordinary AR, target and draft sampling.
Reproduce with small synthetic rows where full-vocabulary top-p and top-k-renormalized top-p retain different tokens. Cover the official sampler and representative temperatures/top-k/top-p combinations. Compare distributions rather than expecting different RNG implementations to emit the same seeded token.
Exercise NaN, positive infinity, all-negative-infinity and valid finite-winner rows with negative-infinity masks, in greedy, pipeline and sampled paths.
Verify explicit error reporting, no fabricated token, invalid session-state removal and the ability to serve a subsequent valid request.
Keep sampler-law and invalid-logit failures as separate bug issues if both are confirmed: their root causes and acceptance criteria differ.
Inspect speculative.rs, session_cache.rs, request.rs, owner.rs and MTP history append/restore.
A final pending-token commit exists, but the round-head EOS/max-token exit appears not to re-arm the emitted primary as pending.
Reproduce first-primary stop, fresh-primary stop, deferred greedy correction, bonus stop, accepted-draft stop, sampled correction and max-token exits at depths 1, 2 and 3, with temperature zero and the native sampler where applicable.
Check emitted IDs against trunk KV offset, GDN/conv state, QSA state, PLE state, MTP history and the banked token key.
Compare the next warm turn with a correctly teacher-forced/replayed state. A snapshot flagged safe must cover every token represented by its key.
Cover cancellation and final-capture failure so neither publishes a falsely complete state.
C. Recurrent restore boundaries and committed history identity
Starting evidence: MTPLX 1a977b65, the 2.11.2/2.11.3 warm-cache repairs, and 2.12.1 restore fixes.
Our active LiveSession currently selects complete stored token prefixes, so the upstream tiny-gap tolerance defect is not automatically ours. Verify outer checkpoint/bootstrap/session selection as well.
Exercise edits or re-rendering 1, 4, 8, 9 and 40 tokens before a saved end. A partial hybrid restore must use a real GDN/conv boundary and replay its suffix; attention-only trimming cannot justify retaining a later recurrent state.
Compare visible-text-equivalent histories with different tokenizations, reasoning echo, trailing whitespace, reordered tool-schema object keys, tool_choice: none, late system messages and changed tool-call arguments.
Test full-prefix repeated requests, cancellation/resend and restarted/truncated MTP history. Match restore offsets to the actual history represented by the snapshot/lease.
Separate a safe cache miss or unsupported warm-reuse optimization from corrupted state or a demonstrated violation of the existing reuse contract.
D. Multimodal sparse QSA and position boundaries
Starting evidence: MTPLX 03a0a0c, 18087e7, 8172196 and the 2.12.1 content-keyed image restore work.
Our active attention explicitly clears sparse selection when vision is present. Reproduce whether the resulting dense route violates the corrected model behavior; compare indexer selections, rotated Q/K, attention output and logits with latest MTPLX.
QSA queries and pooled block-start keys must use the correct image-aware M-RoPE positions, not just the text attention projections.
A chunk that starts inside the image position table and ends outside it appears to select plain RoPE for every row. Reproduce wholly-inside, wholly-outside and straddling chunks; verify each row's rule against an independent position oracle.
Exercise multiple images, text edits before/between/after images, identical image-pad IDs with different pixels, unchanged images, changed pixels and cancel/resend.
A restore must never land inside an image span. Determine whether the present whole-image identity policy is safe but coarse or actually incorrect.
Keep sparse-QSA behavior and the straddling-position defect in separate bug issues if confirmed.
E. Verify numerical parity: gates and float32 operands
Starting evidence: MTPLX 55de58a, be258bc, e6f560f, a075e78, 04f5299, 404084b, and its eager/compiled verify exactness tests.
Inspect attention, PLE, hyper-connection and shared-expert gates; attention.rs, rope.rs, fused.rs, scalar constant folding and graph compilation.
Reproduce BF16 sigmoid differences around gate -6.84375 for PLE, attention output, hyper inject and shared expert. Cover the applicable dtype, phase and eager/compiled routes rather than globally changing all sigmoid operations.
Our compiler preserves the old seven-significant-digit scalar formatting. Trace whether the QSA divisor, YaRN amplitude and rows-gather softmax scale become those constants in actual graphs.
Build near-cutoff QSA score ties at the real 128-dimensional/512-block geometry; compare selected block IDs and ordering, not just approximate scores.
Test ordinary RoPE and static YaRN factor 4, including vision. Require bit-level agreement where the latest oracle requires it and record any justified tolerance separately.
Group child issues by the actual root cause and affected sites. Do not hide distinct numerical errors in one general parity ticket.
Inspect moe.rs, linear.rs, kernel dispatch, shape/weight binding and MTP loading.
Our sorted gathers use the same 0.32.2 NAX runtime shader family without the new pad rule. The default 2,048-token chunk is below the reported limit; verify larger supported user settings as well.
Reproduce ten-routes-per-token widths around 3,277, 3,404 and 4,095 tokens and aligned controls such as 4,096. Use initialized output sentinels to detect unwritten rows, compare every routed row, and cover gate/up and down projections.
Confirm the exact hardware, alignment and function-constant predicates. Distinguish a present configurable-width bug from a future problem that would only be introduced by increasing defaults.
Inspect whether fused MTP expert weights in our pinned Bare-Speed artifact are all correctly bound. Do not apply an unrelated model family's loader fix solely because its title mentions Qwen.
G. Cache capacity, copy verification and long-context ownership
Starting evidence: MTPLX's context-copy growth-edge repairs, 57fa917, capacity-bucket/final-capture fixes, 09c82f7, and layerwise growth/recovery changes in 2.12.1.
Trace our actual verifier, copy path, KV/QSA backing growth, COW/donation and snapshot ownership. Existing compiled primitives do not prove that the entire fixed-M4 model verifier is installed.
Reproduce accepted and rejected context-copy blocks crossing a capacity boundary, final pending writes at the boundary and transitions between verifier widths.
Verify reserve-before-write, no dropped rows, coherent offset/capacity updates and unchanged retained snapshots.
Analyse large-context verify attention's score intermediates and head-chunking, including 64K/128K/near-native-maximum contexts where feasible.
Measure whole-cache copies and growth peaks. Study 8,192-token buckets, one-copy ownership and per-layer growth as optimization candidates; do not file a bug merely because our representation differs.
If allocation/growth is refused, test no partially mutated reusable state, retention of already emitted tokens and correct next-turn recovery.
H. Lazy graph evaluation, event/resource lifetime and memory admission
Starting evidence: MTPLX 73628ba, 011f23f, related #544 history scheduling fixes, and the 2.12.1 memory-guard work.
Check whether cache writes and history appends are actually evaluated even when no later draft consumes them. Track pending graphs, shared events, input/temporary retention and cleanup across long non-streamed and streamed requests.
Reproduce repeated turns and interrupted requests with active/cached/peak bytes and live event/resource counts. A short smoke test does not establish absence of long-run growth.
Trace existing DS4Server memory/admission policy before proposing additions. Include restore copies, prefill intermediates, verifier growth, snapshots, PLE staging and pending persistence owners in peak accounting where applicable.
Investigate whole-Mac free/reclaimable memory and compressor growth, protecting active sessions while reclaiming idle state. Record whether a failure is corruption, a reproducible leak, an incorrect refusal, or an optional policy improvement.
Test failed admission followed by a valid retry; failed work must not retain allocations that make every retry fail.
I. Tokenizer contract
Starting evidence: MTPLX 54b1063a and its Qwen3 combining-mark regression tests.
DS4Server uses a native tokenizer, so the transformers/Qwen2 loader bug is not automatically applicable.
Compare exact token IDs for Hindi, Thai, vowelled Arabic, mixed Unicode punctuation/symbols, Latin, code and Chinese with corrected Qwen3 pretokenization and our exact artifact's vocabulary/merges.
Check combining marks and letter/category boundaries rather than inferring correctness from a few English/code prompts.
File a tokenizer bug only for a demonstrated mismatch in our native path.
J. Prefill and decode optimization candidates
Read the promoted paths and exclusions in MTPLX 2.11.3, 2.12.0 and 2.12.1; inspect actual defaults and runtime engagement, not only flag names.
Candidates include:
M5 4,096-token prefill chunks and sparse attention from 16K, after the row-limit investigation.
Cache-only MTP history append, fewer draft readbacks/synchronizations and PLE read-ahead ordering/overlap.
Full fixed-M4 replay, verify-width HC/QSA/pooled-row fusion and routed-down reduction.
In-forward recurrent boundary capture and better restore-point retention.
First prove output/state compatibility, then measure each relevant candidate separately using the same Bare-Speed artifacts and workload. Report TTFT, prefill/decode rates, accepted tokens per depth, synchronization/dispatch counts, memory peaks, thermal state and variance. Run paired comparisons serially without competing GPU workloads.
A missing optional optimization is not a bug. Create a bug issue only when measurement establishes a concrete regression or violation of an existing supported performance contract. Keep other opportunities in the analysis report for a separately authorized optimization task.
Stage 3: Convert evidence into independent bug issues
For each confirmed DS4Server defect, create one focused bug issue with tea, after checking existing tickets. Reuse/update an existing issue when it already owns the same root cause. Cross-link every resulting issue to #103 and maintain the output list here.
Every bug issue must be self-contained and include:
A concrete title naming the broken behavior, affected route and user-visible impact.
Exact affected DS4Server revision, files/functions/callers and prerequisite settings/hardware/artifacts.
Root cause with evidence distinguishing an actual local defect from the upstream defect or a hypothesis.
Minimal reproduction command/input and durable analysis-code/test paths; actual and expected IDs, distributions, offsets, hashes or resource measurements.
Immutable MTPLX source/test links and the relevant behavioral contract. Name any remaining uncertainty explicitly.
A focused native Rust/Metal implementation scope, affected sibling paths, error/state-cleanup behavior and non-goals.
The reproduction test to retain as the regression test, plus end-to-end warm/cold or next-turn checks necessary to establish the fix.
Concrete acceptance criteria and appropriate validation commands. Before any future commit, apply the repository gates: cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, make bundle, cargo test --all-features.
Severity supported by observed impact, dependencies only where technically necessary, and a note that implementation awaits the user's independent-agent review.
Analysis snippets and tests can be shared by multiple child issues where justified; assign ownership and stable references so no bug depends on discarded scratch files. Do not copy an entire upstream test suite or production implementation. If a repro cannot run on available hardware, retain it as an unresolved candidate with the missing evidence; do not present it as a confirmed local bug.
Independent review handoff
Deliver the evidence inventory and linked bug issues so the user's other agent can reproduce them without this chat. That review should verify active-route applicability, the reproduction on the affected revision, oracle/test correctness, the proposed root cause, implementation boundaries and acceptance criteria. Do not spawn or message another agent under this issue unless the user separately asks for that action.
No production-fix implementation belongs to this stage. Review findings should update the relevant bug issue or return an uncertain item to the analysis inventory before implementation is requested.
The comparison pins our implementation, the old MTPLX reference and the latest MTPLX revision inspected, including its runtime and artifact contracts.
Every area above has an explicit disposition and supporting evidence. Source suspicion, reproduced local failure and verified non-applicability are clearly distinct.
Every confirmed local bug has a focused, linked, independently reviewable bug issue; no unrelated optimization is mislabeled as a bug.
Every retained analysis program or reproduction test has durable instructions and an owning child issue or a documented disposition.
The final report lists confirmed bugs, already-covered/non-applicable changes, optimization opportunities and unresolved blockers separately, with links to the evidence and generated issues.
The user has the complete review handoff. This analysis issue may then be closed with the resulting issue list; implementation and final resolution of those bugs remain in their own issues.
Output tracking
Analysis on 2026-10-03 refreshed MTPLX to 2.12.2 / 9882703f3105363ddc37eca9f97aa09a1d387112, keeping old e652d55e2652137a4abcf1312357abbf3eb9d692, MLX0.32.2 and unchanged Bare-Speed revision74559cdf. All33 artifact sidecar hashes were recomputed and match (106,336,742,717 bytes checked). Eight issues were filed; after independent review, seven retain confirmed active-defect or parity-difference status and #108 is a confirmed module mismatch with unproven product reachability:
Durable local evidence commit: dcc4c516f0e030566c67472f8f1be0baa21ab629, report docs/issue103-analysis/README.md; full hashes/options, JSONL receipts, source diagnostics and isolated oracle probe are committed. No production fixes or tuning changes. Publishing this commit to origin/main is pending explicit user approval after automatic approval review refused the default-branch push. Repository file links in child issues become available after publication; meanwhile the commit and /Users/gb/Projects/DS4Server/docs/issue103-analysis/README.md are available in the user's local workspace.
Evidence strongest for first-primary MTP limit/stop exits (depth 1/2/3, greedy/.6): key4 vs trunk3/MTP2, with247,643 next-turn logits differing; expert rows above32767 (unaligned sorted M5 dispatch), actual vision Attention sparse and straddling outputs, active BF16 attention gate, sampler support/invalid rows, and exact Unicode numeral tokenization. Six ignored native diagnostics passed; existing finite KV/QSA/event/fence/checkpoint/eval controls passed; targeted official upstream tests passed with one unrelated installed27B tokenizer asset skip, plus12 gate-site tests.
All areas A–J have explicit dispositions in the report. Tiny-gap hybrid trimming and the Qwen3.5/3.6 fused-MTP loader fix were not found applicable to this active owner/artifact. Generic seven-digit scalar compilation is a reproduced dormant hazard, not a confirmed active QSA/YaRN failure; full fixed-M4 QSA/PLE owner is not installed. Optional kernels/chunk defaults/capacity buckets/history scheduling remain optimization candidates without paired performance evidence.
Remaining verification keeps #103 open: full MTP exit/cancel/capture matrix; history edits/tool/render identity matrix; real-geometry long-image QSA and image restore spans; PLE/hyper/shared-gate siblings and real cutoff/YaRN comparisons; full gate/up/down expert geometries; copy/growth/OOM boundaries and64K/128K/near-maximum contexts; long streamed/non-streamed event/resource/admission/persistence tests; matched Bare-Speed performance. These are unmeasured obligations, not additional confirmed bugs. Child acceptance owns the related fix verification; no production implementation is authorized by this handoff.
Commit checks: fmt, Clippy and bundle passed. First parallel full test run had two unrelated existing timing/random-fixture failures; serial rerun passed307 main tests (210 ignored) plus8 tool tests, without unrelated fixes. Both logs are retained.
Review corrections and renewed verification (2026-10-03)
The child descriptions were corrected after independent review and the user's clarification: DSv4 Flash/GLM 5.3 Flash must preserve DS4 parity; Qwen 3.8 Flash Next must follow pinned MTPLX2.12.2. The blanket AGENTS.md wording needs model-specific interpretation; no global DS4 behavioral change is authorized. This tracker update supersedes conflicting preliminary scopes and report claims; raw receipts remain historical evidence.
#104: Replaced the instruction to keep GPU/lazy AR with the current MTPLX classic-AR/removal contract, including db6fe0fb and 5a9fe002.; Added the second root cause: PCG64 token 0 followed by keyed device RNG.; Corrected seeded acceptance: same-implementation token-for-token comparison is required; latest no longer contains the obsolete pipeline test.; Restricted sampler/RNG changes to Qwen and added DSv4/GLM regression controls.
#105: Removed retention of an allegedly valid prefill prefix.; Required eviction of all reusable state belonging to the affected Qwen session, including disk/bootstrap selection and delayed persistence ownership.; Aligned lazy scope with #104: only actual remaining primitive callers after the decode lane is retired.; Added unaffected-session/DS4-family controls and retained the legal -inf mask distinction.
#106: Narrowed the production fix to re-arming the pending primary before the round-head exit; retained existing final commit.; Removed speculative final_arrays/capture_session redesign from mandatory scope.; Added explicit invalidation/detection of already-persisted bad QWMTKV01 state and stale live reuse.; Added pre-fix checkpoint → post-fix rejection/recovery acceptance, alongside new-checkpoint roundtrip.
#107: Added invalidation/versioning of pre-fix dense-vision live and persisted cache state.; Explained why image digest/span equality alone cannot identify the attention-policy version.; Required row-wise indexer position handling from the start and kept #108 as a separate latent attention issue.; Added old-cache rejection and corrected new-cache restore acceptance, scoped to Qwen.
#108: Reclassified the result as a confirmed module defect with unproven product reachability; withdrew High severity.; Corrected the upstream reference to 18087e7446cb182591839ab69a67f965523adc27.; Removed the ineffective ordinary MTP table-end reproduction claim and explained full-prompt table/text-tail safety.; Required genuinely unequal image positions in the synthetic boundary case and a real caller/trigger before promoting product impact.
#109: Replaced the wrong F32-sigmoid/cast description with the exact BF16 round-trip compiled expression.; Explained fast-exp fused JIT versus precise-exp standalone lowering and eager/fixed-M4 parity motivation.; Restricted the fix to Qwen verify-phase attention, with F32 coverage only if reachable; prefill/decode/FP16 follow their reference contracts.; Explicitly prohibited global sigmoid/fast-math changes and speculative changes to other gate sites.
#110: Specified the concrete native host padding rule:64-row tiles, zero activations, repeat last sorted expert, slice back.; Made the M5/NAX/predicate/runtime gate explicit and protected unaffected hardware/decode/verify routes.; Recorded that the shader is inside the pinned metallib and upgrading to0.32.3 is a non-goal.; Retained actual gate/up/down geometry and allocation/cleanup acceptance without extending DS4-family kernels.
#111: Corrected the mechanism:① is punctuation;Ⅷ/²/𝟘 take the letter fallback.; Restricted the correction explicitly to ModelFamily::Qwen at both plain and rendered tokenizer entry points.; Protected shared GLM/DS4 Unicode tables, GLM digit grouping and DeepSeek JoyAI behavior with exact-ID regression controls.; Added existing-regex/lookahead guidance without normalization/new asset dependency, and documented safe old-prefix misses/no checkpoint bump for this issue alone.
Seven issues retain confirmed defect/parity status. #108 is latent/Medium at module level; ordinary MTP end-of-table windows are not a demonstrated numerical/product trigger. Review that classification before returning it to the confirmed product-bug inventory.
Each child received a comment listing its changes and concrete renewed-verification questions. No production code, AGENTS.md, assets/defaults/runtime or committed report/receipts were changed, and no new tests/performance measurements were claimed. The evidence commit is still local/unpublished; the existing publication warning remains. Parent #103 and all children remain open. The other agent should verify current descriptions, not implement the old report's superseded scopes.
## Purpose and required output
Keep this issue as an analysis task for Qwen 3.8 Flash Next. Compare our native Rust/Metal implementation with the relevant changes in the latest MTPLX, reproduce actual defects in our active execution paths, and create independently reviewable bug issues for the defects that are confirmed.
The user will review those bug issues with another agent before implementation. Closing this analysis issue does not authorize implementing its child issues. Production fixes belong to the resulting bug issues and begin only after that review and the user's instruction to implement them.
Analysis code, diagnostic instrumentation, small oracle probes and reproduction/regression tests may be created under this issue. Every retained reproduction must be linked or carried into the bug issue that owns its eventual fix. Do not leave analysis-only work as an untracked implementation obligation.
## Reference baseline and reproducibility
Initial inspection on 2026-10-03 found:
- DS4Server checkout: `cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983`.
- Our pinned MTPLX reference: `e652d55e2652137a4abcf1312357abbf3eb9d692`, MTPLX 2.10.2, dated 2026-09-01. The later vision work used the same reference.
- Latest inspected MTPLX: release 2.12.2 and HEAD `9882703f3105363ddc37eca9f97aa09a1d387112`.
- MTPLX itself requires `mlx==0.32.2` on Apple Silicon. Its commit `0d56bc6d1940906884fc1ec40f97c180a48ed43b` pins that version because 0.32.3 produces different Flash-Next output and its release checks used 0.32.2. Our shader runtime is also 0.32.2.
- Primary artifact: `Youssofal/Qwen 3.8-Flash-Next-MTPLX-Bare-Speed`, revision `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`.
At analysis start, refresh MTPLX HEAD and release tags, record the immutable commit actually used, and include any newer relevant changes. Keep both the old reference and the refreshed reference available for before/after comparisons. Do not silently change model weights, quantization, tokenizer assets or MLX version to obtain a passing comparison.
Record checkout SHA, model/config/tokenizer hashes, runtime/compiler versions, hardware, OS, effective options, prompt/image digests, sampling parameters, seed, context, chunk size and cold/warm state for every reproduction. Compare identical artifacts and workloads. The primary performance target remains the same-machine Bare-Speed/M5 Max contract; published Optimized-Speed figures are candidate evidence, not our acceptance result.
## Scope and boundaries
- Analyse the active product path from model loading and request rendering through Rust graph execution, Metal kernels, sampling, state capture, session restore and response publication. Trace callers and effective routing before declaring a defect applicable.
- MTPLX Python and its tests may run in an isolated reference environment as an oracle. Production application, loading, inference, cache/session and orchestration code remains Rust; GPU work remains Metal with the existing Objective-C bridge. Do not introduce MTPLX/MLX Python or C/C++ host execution into DS4Server.
- Preserve DS4 behavioral parity for DSv4 Flash and GLM 5.3 Flash. Qwen 3.8 Flash Next follows the pinned MTPLX oracle instead. Keep shared helpers model-scoped where those contracts differ, and require exact DS4 regression controls for DS4-backed callers; do not impose MTPLX semantics globally. This is the user's explicit model-specific clarification of the blanket DS4 wording in AGENTS.md.
- Under this issue, do not implement or enable production fixes, change default tuning, replace shipped shader artifacts, change managed model revisions, or mark an untested candidate as fixed.
- Temporary experiments must stay isolated and reversible. Explain any instrumentation effect on synchronization, memory, scheduling or timing. Keep receipts needed for another agent to reproduce the result in durable repository evidence; `/tmp` paths alone are insufficient for child issues.
- Use direct git and `tea` from this project for delivery and tracker operations. Use ordinary bullets in issue bodies, not task checkboxes.
## Stage 1: Build an applicability and evidence inventory
Read relevant MTPLX commits, actual source diffs, release notes and regression tests between our pinned reference and the refreshed latest reference. Search existing DS4Server issues before creating duplicates.
For each candidate record:
- Upstream change/test and immutable source reference.
- Root cause, trigger and affected upstream version/route.
- Corresponding DS4Server code, full caller chain, active/inactive status, effective settings and hardware/dtype constraints.
- Preliminary classification: confirmed defect, suspected defect awaiting reproduction, already covered, not applicable, optimization candidate, or blocked by missing evidence.
- Required reproduction and exact expected result.
The observations below are starting hypotheses from source inspection, not completed runtime reproductions. Re-evaluate them against the exact analysis checkout.
## Stage 2: Analyse and reproduce the following areas
### A. Sampling law and invalid logits
Starting evidence: MTPLX `a2d0b4a`; `tests/test_non_finite_logits_and_lazy_sampler_law.py` and the follow-up allowing legitimate grammar-masked negative infinity.
Inspect `src/engine/metal/qwen_mtplx/sampling.rs`, `ar.rs`, `speculative.rs` and API/session error handling.
- The active AR pipeline appears to compute top-p from the top-k slice's softmax; ordinary sampling already has a full-vocabulary path. Compare the actual support and normalized probabilities across pipeline AR, ordinary AR, target and draft sampling.
- Reproduce with small synthetic rows where full-vocabulary top-p and top-k-renormalized top-p retain different tokens. Cover the official sampler and representative temperatures/top-k/top-p combinations. Compare distributions rather than expecting different RNG implementations to emit the same seeded token.
- Exercise NaN, positive infinity, all-negative-infinity and valid finite-winner rows with negative-infinity masks, in greedy, pipeline and sampled paths.
- Verify explicit error reporting, no fabricated token, invalid session-state removal and the ability to serve a subsequent valid request.
- Keep sampler-law and invalid-logit failures as separate bug issues if both are confirmed: their root causes and acceptance criteria differ.
### B. MTP exits and complete final state
Starting evidence: MTPLX `1a977b65`; `tests/test_mtp_final_state_covers_committed_tokens.py`.
Inspect `speculative.rs`, `session_cache.rs`, `request.rs`, `owner.rs` and MTP history append/restore.
- A final pending-token commit exists, but the round-head EOS/max-token exit appears not to re-arm the emitted primary as pending.
- Reproduce first-primary stop, fresh-primary stop, deferred greedy correction, bonus stop, accepted-draft stop, sampled correction and max-token exits at depths 1, 2 and 3, with temperature zero and the native sampler where applicable.
- Check emitted IDs against trunk KV offset, GDN/conv state, QSA state, PLE state, MTP history and the banked token key.
- Compare the next warm turn with a correctly teacher-forced/replayed state. A snapshot flagged safe must cover every token represented by its key.
- Cover cancellation and final-capture failure so neither publishes a falsely complete state.
### C. Recurrent restore boundaries and committed history identity
Starting evidence: MTPLX `1a977b65`, the 2.11.2/2.11.3 warm-cache repairs, and 2.12.1 restore fixes.
- Our active `LiveSession` currently selects complete stored token prefixes, so the upstream tiny-gap tolerance defect is not automatically ours. Verify outer checkpoint/bootstrap/session selection as well.
- Exercise edits or re-rendering 1, 4, 8, 9 and 40 tokens before a saved end. A partial hybrid restore must use a real GDN/conv boundary and replay its suffix; attention-only trimming cannot justify retaining a later recurrent state.
- Compare visible-text-equivalent histories with different tokenizations, reasoning echo, trailing whitespace, reordered tool-schema object keys, `tool_choice: none`, late system messages and changed tool-call arguments.
- Test full-prefix repeated requests, cancellation/resend and restarted/truncated MTP history. Match restore offsets to the actual history represented by the snapshot/lease.
- Separate a safe cache miss or unsupported warm-reuse optimization from corrupted state or a demonstrated violation of the existing reuse contract.
### D. Multimodal sparse QSA and position boundaries
Starting evidence: MTPLX `03a0a0c`, `18087e7`, `8172196` and the 2.12.1 content-keyed image restore work.
Inspect `qsa_attention.rs`, `qsa_indexer.rs`, `qsa_prepare.rs`, `rope.rs`, `vision.rs`, request/session identity and checkpoint code.
- Our active attention explicitly clears sparse selection when vision is present. Reproduce whether the resulting dense route violates the corrected model behavior; compare indexer selections, rotated Q/K, attention output and logits with latest MTPLX.
- QSA queries and pooled block-start keys must use the correct image-aware M-RoPE positions, not just the text attention projections.
- A chunk that starts inside the image position table and ends outside it appears to select plain RoPE for every row. Reproduce wholly-inside, wholly-outside and straddling chunks; verify each row's rule against an independent position oracle.
- Exercise multiple images, text edits before/between/after images, identical image-pad IDs with different pixels, unchanged images, changed pixels and cancel/resend.
- A restore must never land inside an image span. Determine whether the present whole-image identity policy is safe but coarse or actually incorrect.
- Keep sparse-QSA behavior and the straddling-position defect in separate bug issues if confirmed.
### E. Verify numerical parity: gates and float32 operands
Starting evidence: MTPLX `55de58a`, `be258bc`, `e6f560f`, `a075e78`, `04f5299`, `404084b`, and its eager/compiled verify exactness tests.
Inspect attention, PLE, hyper-connection and shared-expert gates; `attention.rs`, `rope.rs`, `fused.rs`, scalar constant folding and graph compilation.
- Reproduce BF16 sigmoid differences around gate `-6.84375` for PLE, attention output, hyper inject and shared expert. Cover the applicable dtype, phase and eager/compiled routes rather than globally changing all sigmoid operations.
- Our compiler preserves the old seven-significant-digit scalar formatting. Trace whether the QSA divisor, YaRN amplitude and rows-gather softmax scale become those constants in actual graphs.
- Build near-cutoff QSA score ties at the real 128-dimensional/512-block geometry; compare selected block IDs and ordering, not just approximate scores.
- Test ordinary RoPE and static YaRN factor 4, including vision. Require bit-level agreement where the latest oracle requires it and record any justified tolerance separately.
- Group child issues by the actual root cause and affected sites. Do not hide distinct numerical errors in one general parity ticket.
### F. Expert-gather row limits and artifact binding
Starting evidence: MTPLX `b36b106`; `tests/test_moe_sorted_gather.py`, `tests/test_qwen4_moe_sorted_rows.py`; fused MTP expert-loading fix `13128e2`/#574.
Inspect `moe.rs`, `linear.rs`, kernel dispatch, shape/weight binding and MTP loading.
- Our sorted gathers use the same 0.32.2 NAX runtime shader family without the new pad rule. The default 2,048-token chunk is below the reported limit; verify larger supported user settings as well.
- Reproduce ten-routes-per-token widths around 3,277, 3,404 and 4,095 tokens and aligned controls such as 4,096. Use initialized output sentinels to detect unwritten rows, compare every routed row, and cover gate/up and down projections.
- Confirm the exact hardware, alignment and function-constant predicates. Distinguish a present configurable-width bug from a future problem that would only be introduced by increasing defaults.
- Inspect whether fused MTP expert weights in our pinned Bare-Speed artifact are all correctly bound. Do not apply an unrelated model family's loader fix solely because its title mentions Qwen.
### G. Cache capacity, copy verification and long-context ownership
Starting evidence: MTPLX's context-copy growth-edge repairs, `57fa917`, capacity-bucket/final-capture fixes, `09c82f7`, and layerwise growth/recovery changes in 2.12.1.
- Trace our actual verifier, copy path, KV/QSA backing growth, COW/donation and snapshot ownership. Existing compiled primitives do not prove that the entire fixed-M4 model verifier is installed.
- Reproduce accepted and rejected context-copy blocks crossing a capacity boundary, final pending writes at the boundary and transitions between verifier widths.
- Verify reserve-before-write, no dropped rows, coherent offset/capacity updates and unchanged retained snapshots.
- Analyse large-context verify attention's score intermediates and head-chunking, including 64K/128K/near-native-maximum contexts where feasible.
- Measure whole-cache copies and growth peaks. Study 8,192-token buckets, one-copy ownership and per-layer growth as optimization candidates; do not file a bug merely because our representation differs.
- If allocation/growth is refused, test no partially mutated reusable state, retention of already emitted tokens and correct next-turn recovery.
### H. Lazy graph evaluation, event/resource lifetime and memory admission
Starting evidence: MTPLX `73628ba`, `011f23f`, related #544 history scheduling fixes, and the 2.12.1 memory-guard work.
- Check whether cache writes and history appends are actually evaluated even when no later draft consumes them. Track pending graphs, shared events, input/temporary retention and cleanup across long non-streamed and streamed requests.
- Reproduce repeated turns and interrupted requests with active/cached/peak bytes and live event/resource counts. A short smoke test does not establish absence of long-run growth.
- Trace existing DS4Server memory/admission policy before proposing additions. Include restore copies, prefill intermediates, verifier growth, snapshots, PLE staging and pending persistence owners in peak accounting where applicable.
- Investigate whole-Mac free/reclaimable memory and compressor growth, protecting active sessions while reclaiming idle state. Record whether a failure is corruption, a reproducible leak, an incorrect refusal, or an optional policy improvement.
- Test failed admission followed by a valid retry; failed work must not retain allocations that make every retry fail.
### I. Tokenizer contract
Starting evidence: MTPLX `54b1063a` and its Qwen3 combining-mark regression tests.
- DS4Server uses a native tokenizer, so the transformers/Qwen2 loader bug is not automatically applicable.
- Compare exact token IDs for Hindi, Thai, vowelled Arabic, mixed Unicode punctuation/symbols, Latin, code and Chinese with corrected Qwen3 pretokenization and our exact artifact's vocabulary/merges.
- Check combining marks and letter/category boundaries rather than inferring correctness from a few English/code prompts.
- File a tokenizer bug only for a demonstrated mismatch in our native path.
### J. Prefill and decode optimization candidates
Read the promoted paths and exclusions in MTPLX 2.11.3, 2.12.0 and 2.12.1; inspect actual defaults and runtime engagement, not only flag names.
Candidates include:
- M5 4,096-token prefill chunks and sparse attention from 16K, after the row-limit investigation.
- Cache-direct sparse prefill, simdgroup-per-row block selection, runtime block counts and reduced shader recompilation.
- Fused hyper read/write, GDN prework/gated norm, MoE combine and M5 dequantize-plus-dense-GEMM projections.
- Cache-only MTP history append, fewer draft readbacks/synchronizations and PLE read-ahead ordering/overlap.
- Full fixed-M4 replay, verify-width HC/QSA/pooled-row fusion and routed-down reduction.
- In-forward recurrent boundary capture and better restore-point retention.
First prove output/state compatibility, then measure each relevant candidate separately using the same Bare-Speed artifacts and workload. Report TTFT, prefill/decode rates, accepted tokens per depth, synchronization/dispatch counts, memory peaks, thermal state and variance. Run paired comparisons serially without competing GPU workloads.
A missing optional optimization is not a bug. Create a bug issue only when measurement establishes a concrete regression or violation of an existing supported performance contract. Keep other opportunities in the analysis report for a separately authorized optimization task.
## Stage 3: Convert evidence into independent bug issues
For each confirmed DS4Server defect, create one focused `bug` issue with `tea`, after checking existing tickets. Reuse/update an existing issue when it already owns the same root cause. Cross-link every resulting issue to #103 and maintain the output list here.
Every bug issue must be self-contained and include:
- A concrete title naming the broken behavior, affected route and user-visible impact.
- Exact affected DS4Server revision, files/functions/callers and prerequisite settings/hardware/artifacts.
- Root cause with evidence distinguishing an actual local defect from the upstream defect or a hypothesis.
- Minimal reproduction command/input and durable analysis-code/test paths; actual and expected IDs, distributions, offsets, hashes or resource measurements.
- Immutable MTPLX source/test links and the relevant behavioral contract. Name any remaining uncertainty explicitly.
- A focused native Rust/Metal implementation scope, affected sibling paths, error/state-cleanup behavior and non-goals.
- The reproduction test to retain as the regression test, plus end-to-end warm/cold or next-turn checks necessary to establish the fix.
- Concrete acceptance criteria and appropriate validation commands. Before any future commit, apply the repository gates: `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `make bundle`, `cargo test --all-features`.
- Severity supported by observed impact, dependencies only where technically necessary, and a note that implementation awaits the user's independent-agent review.
Analysis snippets and tests can be shared by multiple child issues where justified; assign ownership and stable references so no bug depends on discarded scratch files. Do not copy an entire upstream test suite or production implementation. If a repro cannot run on available hardware, retain it as an unresolved candidate with the missing evidence; do not present it as a confirmed local bug.
## Independent review handoff
Deliver the evidence inventory and linked bug issues so the user's other agent can reproduce them without this chat. That review should verify active-route applicability, the reproduction on the affected revision, oracle/test correctness, the proposed root cause, implementation boundaries and acceptance criteria. Do not spawn or message another agent under this issue unless the user separately asks for that action.
No production-fix implementation belongs to this stage. Review findings should update the relevant bug issue or return an uncertain item to the analysis inventory before implementation is requested.
## Completion criteria for #103
- The comparison pins our implementation, the old MTPLX reference and the latest MTPLX revision inspected, including its runtime and artifact contracts.
- Every area above has an explicit disposition and supporting evidence. Source suspicion, reproduced local failure and verified non-applicability are clearly distinct.
- Every confirmed local bug has a focused, linked, independently reviewable bug issue; no unrelated optimization is mislabeled as a bug.
- Every retained analysis program or reproduction test has durable instructions and an owning child issue or a documented disposition.
- The final report lists confirmed bugs, already-covered/non-applicable changes, optimization opportunities and unresolved blockers separately, with links to the evidence and generated issues.
- The user has the complete review handoff. This analysis issue may then be closed with the resulting issue list; implementation and final resolution of those bugs remain in their own issues.
## Output tracking
Analysis on 2026-10-03 refreshed MTPLX to 2.12.2 / `9882703f3105363ddc37eca9f97aa09a1d387112`, keeping old `e652d55e2652137a4abcf1312357abbf3eb9d692`, MLX0.32.2 and unchanged Bare-Speed revision74559cdf. All33 artifact sidecar hashes were recomputed and match (106,336,742,717 bytes checked). Eight issues were filed; after independent review, seven retain confirmed active-defect or parity-difference status and #108 is a confirmed module mismatch with unproven product reachability:
- [104: Use full-vocabulary top-p law in Qwen AR pipeline](https://git.rfc1437.de/hugo/DS4Server/issues/104) — area A1; owned diagnostic `issue103_sampling_receipts`; awaiting the user's independent-agent review.
- [105: Reject invalid Qwen logits instead of emitting fabricated tokens](https://git.rfc1437.de/hugo/DS4Server/issues/105) — area A2; owned diagnostic `issue103_sampling_receipts`; awaiting the user's independent-agent review.
- [106: Commit the final MTP primary before banking warm session state](https://git.rfc1437.de/hugo/DS4Server/issues/106) — area B; owned diagnostic `issue103_mtp_final_receipts`; awaiting the user's independent-agent review.
- [107: Preserve sparse QSA and image-aware indexer positions for vision turns](https://git.rfc1437.de/hugo/DS4Server/issues/107) — area D1; owned diagnostic `issue103_vision_position_receipts`; awaiting the user's independent-agent review.
- [108: Apply M-RoPE row by row across the vision position-table boundary](https://git.rfc1437.de/hugo/DS4Server/issues/108) — area D2; owned diagnostic `issue103_vision_position_receipts`; awaiting the user's independent-agent review.
- [109: Preserve compiled BF16 attention-gate math in MTP verification](https://git.rfc1437.de/hugo/DS4Server/issues/109) — area E; owned diagnostic `issue103_vision_position_receipts`; awaiting the user's independent-agent review.
- [110: Prevent uninitialized expert outputs for large unaligned M5 prefill chunks](https://git.rfc1437.de/hugo/DS4Server/issues/110) — area F; owned diagnostic `issue103_sorted_gather_receipts`; awaiting the user's independent-agent review.
- [111: Tokenize all Unicode number categories with the Qwen3 contract](https://git.rfc1437.de/hugo/DS4Server/issues/111) — area I; owned diagnostic `issue103_tokenizer_receipts`; awaiting the user's independent-agent review.
Durable local evidence commit: `dcc4c516f0e030566c67472f8f1be0baa21ab629`, report `docs/issue103-analysis/README.md`; full hashes/options, JSONL receipts, source diagnostics and isolated oracle probe are committed. No production fixes or tuning changes. Publishing this commit to origin/main is pending explicit user approval after automatic approval review refused the default-branch push. Repository file links in child issues become available after publication; meanwhile the commit and `/Users/gb/Projects/DS4Server/docs/issue103-analysis/README.md` are available in the user's local workspace.
Evidence strongest for first-primary MTP limit/stop exits (depth 1/2/3, greedy/.6): key4 vs trunk3/MTP2, with247,643 next-turn logits differing; expert rows above32767 (unaligned sorted M5 dispatch), actual vision Attention sparse and straddling outputs, active BF16 attention gate, sampler support/invalid rows, and exact Unicode numeral tokenization. Six ignored native diagnostics passed; existing finite KV/QSA/event/fence/checkpoint/eval controls passed; targeted official upstream tests passed with one unrelated installed27B tokenizer asset skip, plus12 gate-site tests.
All areas A–J have explicit dispositions in the report. Tiny-gap hybrid trimming and the Qwen3.5/3.6 fused-MTP loader fix were not found applicable to this active owner/artifact. Generic seven-digit scalar compilation is a reproduced dormant hazard, not a confirmed active QSA/YaRN failure; full fixed-M4 QSA/PLE owner is not installed. Optional kernels/chunk defaults/capacity buckets/history scheduling remain optimization candidates without paired performance evidence.
Remaining verification keeps #103 open: full MTP exit/cancel/capture matrix; history edits/tool/render identity matrix; real-geometry long-image QSA and image restore spans; PLE/hyper/shared-gate siblings and real cutoff/YaRN comparisons; full gate/up/down expert geometries; copy/growth/OOM boundaries and64K/128K/near-maximum contexts; long streamed/non-streamed event/resource/admission/persistence tests; matched Bare-Speed performance. These are unmeasured obligations, not additional confirmed bugs. Child acceptance owns the related fix verification; no production implementation is authorized by this handoff.
Commit checks: fmt, Clippy and bundle passed. First parallel full test run had two unrelated existing timing/random-fixture failures; serial rerun passed307 main tests (210 ignored) plus8 tool tests, without unrelated fixes. Both logs are retained.
## Review corrections and renewed verification (2026-10-03)
The child descriptions were corrected after independent review and the user's clarification: DSv4 Flash/GLM 5.3 Flash must preserve DS4 parity; Qwen 3.8 Flash Next must follow pinned MTPLX2.12.2. The blanket AGENTS.md wording needs model-specific interpretation; no global DS4 behavioral change is authorized. This tracker update supersedes conflicting preliminary scopes and report claims; raw receipts remain historical evidence.
- [#104](https://git.rfc1437.de/hugo/DS4Server/issues/104): Replaced the instruction to keep GPU/lazy AR with the current MTPLX classic-AR/removal contract, including db6fe0fb and 5a9fe002.; Added the second root cause: PCG64 token 0 followed by keyed device RNG.; Corrected seeded acceptance: same-implementation token-for-token comparison is required; latest no longer contains the obsolete pipeline test.; Restricted sampler/RNG changes to Qwen and added DSv4/GLM regression controls.
- [#105](https://git.rfc1437.de/hugo/DS4Server/issues/105): Removed retention of an allegedly valid prefill prefix.; Required eviction of all reusable state belonging to the affected Qwen session, including disk/bootstrap selection and delayed persistence ownership.; Aligned lazy scope with #104: only actual remaining primitive callers after the decode lane is retired.; Added unaffected-session/DS4-family controls and retained the legal -inf mask distinction.
- [#106](https://git.rfc1437.de/hugo/DS4Server/issues/106): Narrowed the production fix to re-arming the pending primary before the round-head exit; retained existing final commit.; Removed speculative final_arrays/capture_session redesign from mandatory scope.; Added explicit invalidation/detection of already-persisted bad QWMTKV01 state and stale live reuse.; Added pre-fix checkpoint → post-fix rejection/recovery acceptance, alongside new-checkpoint roundtrip.
- [#107](https://git.rfc1437.de/hugo/DS4Server/issues/107): Added invalidation/versioning of pre-fix dense-vision live and persisted cache state.; Explained why image digest/span equality alone cannot identify the attention-policy version.; Required row-wise indexer position handling from the start and kept #108 as a separate latent attention issue.; Added old-cache rejection and corrected new-cache restore acceptance, scoped to Qwen.
- [#108](https://git.rfc1437.de/hugo/DS4Server/issues/108): Reclassified the result as a confirmed module defect with unproven product reachability; withdrew High severity.; Corrected the upstream reference to 18087e7446cb182591839ab69a67f965523adc27.; Removed the ineffective ordinary MTP table-end reproduction claim and explained full-prompt table/text-tail safety.; Required genuinely unequal image positions in the synthetic boundary case and a real caller/trigger before promoting product impact.
- [#109](https://git.rfc1437.de/hugo/DS4Server/issues/109): Replaced the wrong F32-sigmoid/cast description with the exact BF16 round-trip compiled expression.; Explained fast-exp fused JIT versus precise-exp standalone lowering and eager/fixed-M4 parity motivation.; Restricted the fix to Qwen verify-phase attention, with F32 coverage only if reachable; prefill/decode/FP16 follow their reference contracts.; Explicitly prohibited global sigmoid/fast-math changes and speculative changes to other gate sites.
- [#110](https://git.rfc1437.de/hugo/DS4Server/issues/110): Specified the concrete native host padding rule:64-row tiles, zero activations, repeat last sorted expert, slice back.; Made the M5/NAX/predicate/runtime gate explicit and protected unaffected hardware/decode/verify routes.; Recorded that the shader is inside the pinned metallib and upgrading to0.32.3 is a non-goal.; Retained actual gate/up/down geometry and allocation/cleanup acceptance without extending DS4-family kernels.
- [#111](https://git.rfc1437.de/hugo/DS4Server/issues/111): Corrected the mechanism:① is punctuation;Ⅷ/²/𝟘 take the letter fallback.; Restricted the correction explicitly to ModelFamily::Qwen at both plain and rendered tokenizer entry points.; Protected shared GLM/DS4 Unicode tables, GLM digit grouping and DeepSeek JoyAI behavior with exact-ID regression controls.; Added existing-regex/lookahead guidance without normalization/new asset dependency, and documented safe old-prefix misses/no checkpoint bump for this issue alone.
Seven issues retain confirmed defect/parity status. #108 is latent/Medium at module level; ordinary MTP end-of-table windows are not a demonstrated numerical/product trigger. Review that classification before returning it to the confirmed product-bug inventory.
Each child received a comment listing its changes and concrete renewed-verification questions. No production code, AGENTS.md, assets/defaults/runtime or committed report/receipts were changed, and no new tests/performance measurements were claimed. The evidence commit is still local/unpublished; the existing publication warning remains. Parent #103 and all children remain open. The other agent should verify current descriptions, not implement the old report's superseded scopes.
hugo
changed title from upate the qwen code to the newest version of mtplx to Analyse Qwen3.8 Flash Next changes in MTPLX and create confirmed bug issues2026-10-03 04:51:58 +00:00
Child issue descriptions corrected after independent review (2026-10-03)
Updated #104–#111 and added one change/renewed-verification comment to each. The descriptions now explicitly separate Qwen/MTPLX parity from DSv4 Flash and GLM 5.3 Flash/DS4 parity.
#104 follows latest classic AR/lane removal and addresses RNG divergence as well as top-p.
#105 drops all reusable state of the affected Qwen session; no retained poisoned prefix.
#106 uses the minimal pending-token repair and adds rejection of old corrupt checkpoints.
#107 invalidates old dense-vision state and requires row-wise indexer positions.
#108 is latent/Medium with unproven product reachability; correct upstream commit and effective test trigger.
#109 specifies exact fused BF16/F32 verify lowering, with no global sigmoid/fast-math change.
#110 specifies predicate-gated host padding and no runtime/shader upgrade.
#111 is strictly Qwen-only, with corrected punctuation/letter mechanism and unchanged GLM/DeepSeek DS4 token-ID controls.
The original report/diagnostics are unchanged historical evidence where superseded by these descriptions. Evidence commit remains local/unpublished. No production implementation, new test result, AGENTS.md edit or push occurred. All issues remain open for renewed independent verification before implementation.
## Child issue descriptions corrected after independent review (2026-10-03)
Updated #104–#111 and added one change/renewed-verification comment to each. The descriptions now explicitly separate Qwen/MTPLX parity from DSv4 Flash and GLM 5.3 Flash/DS4 parity.
- #104 follows latest classic AR/lane removal and addresses RNG divergence as well as top-p.
- #105 drops all reusable state of the affected Qwen session; no retained poisoned prefix.
- #106 uses the minimal pending-token repair and adds rejection of old corrupt checkpoints.
- #107 invalidates old dense-vision state and requires row-wise indexer positions.
- #108 is latent/Medium with unproven product reachability; correct upstream commit and effective test trigger.
- #109 specifies exact fused BF16/F32 verify lowering, with no global sigmoid/fast-math change.
- #110 specifies predicate-gated host padding and no runtime/shader upgrade.
- #111 is strictly Qwen-only, with corrected punctuation/letter mechanism and unchanged GLM/DeepSeek DS4 token-ID controls.
The original report/diagnostics are unchanged historical evidence where superseded by these descriptions. Evidence commit remains local/unpublished. No production implementation, new test result, AGENTS.md edit or push occurred. All issues remain open for renewed independent verification before implementation.
Re-verification of revised child issues (2026-10-03)
Oracle assignment confirmed by the user: DS4 is the parity reference for DeepSeek and GLM; MTPLX (pinned 9882703f3105363ddc37eca9f97aa09a1d387112) is the parity reference for Qwen 3.8 Flash Next. For Qwen, our Rust/Metal code must implement the current MTPLX state, not only patch individual defects. AGENTS.md still states DS4 as the blanket oracle and should be updated to say this explicitly.
Re-verification of the revised #104–#111 bodies against code and upstream: all review corrections are incorporated correctly, and all cited upstream SHAs and tests exist at 9882703. #105, #106, #107, #108, #109 and #111 have no remaining gaps.
Remaining gaps (details in the respective issue comments):
#104: The lane only engages with a resident n-gram table (>= 160 GiB RAM or MTPLX_NGRAM_RESIDENT=1; ngram.rs:14, model.rs:308). On the 128 GiB reference machine default AR is already classic, so the trigger and severity wording must be corrected. The removal scope must cover every engagement site, including Model::set_ar_pipeline_mode with its PLE staging toggle and the harness read at execution.rs:1188, as in upstream 5a9fe002.
#110: The scope covers only the MLX row guard. Current MTPLX Flash-Next MoE prefill uses the row-map sort plus the fused SwiGLU NAX kernel for gate/up (aa68928d and follow-ups; >= 4,096 routed rows on MLX 0.32.2, canary + fallback) and the guarded stock kernel only for the down projection. DS4Server still has the old copy-gather-split-SiLU chain. Either extend #110's scope or add a linked Qwen issue for the kernel port.
Still open: evidence commit dcc4c51 is unpublished, so the commit-pinned links resolve only after it is pushed.
## Re-verification of revised child issues (2026-10-03)
Oracle assignment confirmed by the user: **DS4 is the parity reference for DeepSeek and GLM; MTPLX (pinned `9882703f3105363ddc37eca9f97aa09a1d387112`) is the parity reference for Qwen 3.8 Flash Next.** For Qwen, our Rust/Metal code must implement the current MTPLX state, not only patch individual defects. AGENTS.md still states DS4 as the blanket oracle and should be updated to say this explicitly.
Re-verification of the revised #104–#111 bodies against code and upstream: all review corrections are incorporated correctly, and all cited upstream SHAs and tests exist at `9882703`. #105, #106, #107, #108, #109 and #111 have no remaining gaps.
Remaining gaps (details in the respective issue comments):
- **#104:** The lane only engages with a resident n-gram table (>= 160 GiB RAM or `MTPLX_NGRAM_RESIDENT=1`; ngram.rs:14, model.rs:308). On the 128 GiB reference machine default AR is already classic, so the trigger and severity wording must be corrected. The removal scope must cover every engagement site, including `Model::set_ar_pipeline_mode` with its PLE staging toggle and the harness read at execution.rs:1188, as in upstream `5a9fe002`.
- **#110:** The scope covers only the MLX row guard. Current MTPLX Flash-Next MoE prefill uses the row-map sort plus the fused SwiGLU NAX kernel for gate/up (`aa68928d` and follow-ups; >= 4,096 routed rows on MLX 0.32.2, canary + fallback) and the guarded stock kernel only for the down projection. DS4Server still has the old copy-gather-split-SiLU chain. Either extend #110's scope or add a linked Qwen issue for the kernel port.
Still open: evidence commit `dcc4c51` is unpublished, so the commit-pinned links resolve only after it is pushed.
Output list addition: #112 Port MTPLX row-map SwiGLU expert gather for Qwen prefill MoE (required route-parity port per the Qwen/MTPLX oracle assignment; bit-identical to the guarded stock chain upstream, so not a numerical bug; depends on #110's guard). With this, the #110 gap from the re-verification comment is owned. Re-verification of the revised #104 and #110 bodies: both correct, no remaining gaps.
Output list addition: #112 Port MTPLX row-map SwiGLU expert gather for Qwen prefill MoE (required route-parity port per the Qwen/MTPLX oracle assignment; bit-identical to the guarded stock chain upstream, so not a numerical bug; depends on #110's guard). With this, the #110 gap from the re-verification comment is owned. Re-verification of the revised #104 and #110 bodies: both correct, no remaining gaps.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Purpose and required output
Keep this issue as an analysis task for Qwen 3.8 Flash Next. Compare our native Rust/Metal implementation with the relevant changes in the latest MTPLX, reproduce actual defects in our active execution paths, and create independently reviewable bug issues for the defects that are confirmed.
The user will review those bug issues with another agent before implementation. Closing this analysis issue does not authorize implementing its child issues. Production fixes belong to the resulting bug issues and begin only after that review and the user's instruction to implement them.
Analysis code, diagnostic instrumentation, small oracle probes and reproduction/regression tests may be created under this issue. Every retained reproduction must be linked or carried into the bug issue that owns its eventual fix. Do not leave analysis-only work as an untracked implementation obligation.
Reference baseline and reproducibility
Initial inspection on 2026-10-03 found:
cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983.e652d55e2652137a4abcf1312357abbf3eb9d692, MTPLX 2.10.2, dated 2026-09-01. The later vision work used the same reference.9882703f3105363ddc37eca9f97aa09a1d387112.mlx==0.32.2on Apple Silicon. Its commit0d56bc6d1940906884fc1ec40f97c180a48ed43bpins that version because 0.32.3 produces different Flash-Next output and its release checks used 0.32.2. Our shader runtime is also 0.32.2.Youssofal/Qwen 3.8-Flash-Next-MTPLX-Bare-Speed, revision74559cdf34fbfc0b593de72d17e93f37fd4f9ea7.At analysis start, refresh MTPLX HEAD and release tags, record the immutable commit actually used, and include any newer relevant changes. Keep both the old reference and the refreshed reference available for before/after comparisons. Do not silently change model weights, quantization, tokenizer assets or MLX version to obtain a passing comparison.
Record checkout SHA, model/config/tokenizer hashes, runtime/compiler versions, hardware, OS, effective options, prompt/image digests, sampling parameters, seed, context, chunk size and cold/warm state for every reproduction. Compare identical artifacts and workloads. The primary performance target remains the same-machine Bare-Speed/M5 Max contract; published Optimized-Speed figures are candidate evidence, not our acceptance result.
Scope and boundaries
/tmppaths alone are insufficient for child issues.teafrom this project for delivery and tracker operations. Use ordinary bullets in issue bodies, not task checkboxes.Stage 1: Build an applicability and evidence inventory
Read relevant MTPLX commits, actual source diffs, release notes and regression tests between our pinned reference and the refreshed latest reference. Search existing DS4Server issues before creating duplicates.
For each candidate record:
The observations below are starting hypotheses from source inspection, not completed runtime reproductions. Re-evaluate them against the exact analysis checkout.
Stage 2: Analyse and reproduce the following areas
A. Sampling law and invalid logits
Starting evidence: MTPLX
a2d0b4a;tests/test_non_finite_logits_and_lazy_sampler_law.pyand the follow-up allowing legitimate grammar-masked negative infinity.Inspect
src/engine/metal/qwen_mtplx/sampling.rs,ar.rs,speculative.rsand API/session error handling.B. MTP exits and complete final state
Starting evidence: MTPLX
1a977b65;tests/test_mtp_final_state_covers_committed_tokens.py.Inspect
speculative.rs,session_cache.rs,request.rs,owner.rsand MTP history append/restore.C. Recurrent restore boundaries and committed history identity
Starting evidence: MTPLX
1a977b65, the 2.11.2/2.11.3 warm-cache repairs, and 2.12.1 restore fixes.LiveSessioncurrently selects complete stored token prefixes, so the upstream tiny-gap tolerance defect is not automatically ours. Verify outer checkpoint/bootstrap/session selection as well.tool_choice: none, late system messages and changed tool-call arguments.D. Multimodal sparse QSA and position boundaries
Starting evidence: MTPLX
03a0a0c,18087e7,8172196and the 2.12.1 content-keyed image restore work.Inspect
qsa_attention.rs,qsa_indexer.rs,qsa_prepare.rs,rope.rs,vision.rs, request/session identity and checkpoint code.E. Verify numerical parity: gates and float32 operands
Starting evidence: MTPLX
55de58a,be258bc,e6f560f,a075e78,04f5299,404084b, and its eager/compiled verify exactness tests.Inspect attention, PLE, hyper-connection and shared-expert gates;
attention.rs,rope.rs,fused.rs, scalar constant folding and graph compilation.-6.84375for PLE, attention output, hyper inject and shared expert. Cover the applicable dtype, phase and eager/compiled routes rather than globally changing all sigmoid operations.F. Expert-gather row limits and artifact binding
Starting evidence: MTPLX
b36b106;tests/test_moe_sorted_gather.py,tests/test_qwen4_moe_sorted_rows.py; fused MTP expert-loading fix13128e2/#574.Inspect
moe.rs,linear.rs, kernel dispatch, shape/weight binding and MTP loading.G. Cache capacity, copy verification and long-context ownership
Starting evidence: MTPLX's context-copy growth-edge repairs,
57fa917, capacity-bucket/final-capture fixes,09c82f7, and layerwise growth/recovery changes in 2.12.1.H. Lazy graph evaluation, event/resource lifetime and memory admission
Starting evidence: MTPLX
73628ba,011f23f, related #544 history scheduling fixes, and the 2.12.1 memory-guard work.I. Tokenizer contract
Starting evidence: MTPLX
54b1063aand its Qwen3 combining-mark regression tests.J. Prefill and decode optimization candidates
Read the promoted paths and exclusions in MTPLX 2.11.3, 2.12.0 and 2.12.1; inspect actual defaults and runtime engagement, not only flag names.
Candidates include:
First prove output/state compatibility, then measure each relevant candidate separately using the same Bare-Speed artifacts and workload. Report TTFT, prefill/decode rates, accepted tokens per depth, synchronization/dispatch counts, memory peaks, thermal state and variance. Run paired comparisons serially without competing GPU workloads.
A missing optional optimization is not a bug. Create a bug issue only when measurement establishes a concrete regression or violation of an existing supported performance contract. Keep other opportunities in the analysis report for a separately authorized optimization task.
Stage 3: Convert evidence into independent bug issues
For each confirmed DS4Server defect, create one focused
bugissue withtea, after checking existing tickets. Reuse/update an existing issue when it already owns the same root cause. Cross-link every resulting issue to #103 and maintain the output list here.Every bug issue must be self-contained and include:
cargo fmt --all -- --check,cargo clippy --all-targets --all-features -- -D warnings,make bundle,cargo test --all-features.Analysis snippets and tests can be shared by multiple child issues where justified; assign ownership and stable references so no bug depends on discarded scratch files. Do not copy an entire upstream test suite or production implementation. If a repro cannot run on available hardware, retain it as an unresolved candidate with the missing evidence; do not present it as a confirmed local bug.
Independent review handoff
Deliver the evidence inventory and linked bug issues so the user's other agent can reproduce them without this chat. That review should verify active-route applicability, the reproduction on the affected revision, oracle/test correctness, the proposed root cause, implementation boundaries and acceptance criteria. Do not spawn or message another agent under this issue unless the user separately asks for that action.
No production-fix implementation belongs to this stage. Review findings should update the relevant bug issue or return an uncertain item to the analysis inventory before implementation is requested.
Completion criteria for #103
Output tracking
Analysis on 2026-10-03 refreshed MTPLX to 2.12.2 /
9882703f3105363ddc37eca9f97aa09a1d387112, keeping olde652d55e2652137a4abcf1312357abbf3eb9d692, MLX0.32.2 and unchanged Bare-Speed revision74559cdf. All33 artifact sidecar hashes were recomputed and match (106,336,742,717 bytes checked). Eight issues were filed; after independent review, seven retain confirmed active-defect or parity-difference status and #108 is a confirmed module mismatch with unproven product reachability:issue103_sampling_receipts; awaiting the user's independent-agent review.issue103_sampling_receipts; awaiting the user's independent-agent review.issue103_mtp_final_receipts; awaiting the user's independent-agent review.issue103_vision_position_receipts; awaiting the user's independent-agent review.issue103_vision_position_receipts; awaiting the user's independent-agent review.issue103_vision_position_receipts; awaiting the user's independent-agent review.issue103_sorted_gather_receipts; awaiting the user's independent-agent review.issue103_tokenizer_receipts; awaiting the user's independent-agent review.Durable local evidence commit:
dcc4c516f0e030566c67472f8f1be0baa21ab629, reportdocs/issue103-analysis/README.md; full hashes/options, JSONL receipts, source diagnostics and isolated oracle probe are committed. No production fixes or tuning changes. Publishing this commit to origin/main is pending explicit user approval after automatic approval review refused the default-branch push. Repository file links in child issues become available after publication; meanwhile the commit and/Users/gb/Projects/DS4Server/docs/issue103-analysis/README.mdare available in the user's local workspace.Evidence strongest for first-primary MTP limit/stop exits (depth 1/2/3, greedy/.6): key4 vs trunk3/MTP2, with247,643 next-turn logits differing; expert rows above32767 (unaligned sorted M5 dispatch), actual vision Attention sparse and straddling outputs, active BF16 attention gate, sampler support/invalid rows, and exact Unicode numeral tokenization. Six ignored native diagnostics passed; existing finite KV/QSA/event/fence/checkpoint/eval controls passed; targeted official upstream tests passed with one unrelated installed27B tokenizer asset skip, plus12 gate-site tests.
All areas A–J have explicit dispositions in the report. Tiny-gap hybrid trimming and the Qwen3.5/3.6 fused-MTP loader fix were not found applicable to this active owner/artifact. Generic seven-digit scalar compilation is a reproduced dormant hazard, not a confirmed active QSA/YaRN failure; full fixed-M4 QSA/PLE owner is not installed. Optional kernels/chunk defaults/capacity buckets/history scheduling remain optimization candidates without paired performance evidence.
Remaining verification keeps #103 open: full MTP exit/cancel/capture matrix; history edits/tool/render identity matrix; real-geometry long-image QSA and image restore spans; PLE/hyper/shared-gate siblings and real cutoff/YaRN comparisons; full gate/up/down expert geometries; copy/growth/OOM boundaries and64K/128K/near-maximum contexts; long streamed/non-streamed event/resource/admission/persistence tests; matched Bare-Speed performance. These are unmeasured obligations, not additional confirmed bugs. Child acceptance owns the related fix verification; no production implementation is authorized by this handoff.
Commit checks: fmt, Clippy and bundle passed. First parallel full test run had two unrelated existing timing/random-fixture failures; serial rerun passed307 main tests (210 ignored) plus8 tool tests, without unrelated fixes. Both logs are retained.
Review corrections and renewed verification (2026-10-03)
The child descriptions were corrected after independent review and the user's clarification: DSv4 Flash/GLM 5.3 Flash must preserve DS4 parity; Qwen 3.8 Flash Next must follow pinned MTPLX2.12.2. The blanket AGENTS.md wording needs model-specific interpretation; no global DS4 behavioral change is authorized. This tracker update supersedes conflicting preliminary scopes and report claims; raw receipts remain historical evidence.
Seven issues retain confirmed defect/parity status. #108 is latent/Medium at module level; ordinary MTP end-of-table windows are not a demonstrated numerical/product trigger. Review that classification before returning it to the confirmed product-bug inventory.
Each child received a comment listing its changes and concrete renewed-verification questions. No production code, AGENTS.md, assets/defaults/runtime or committed report/receipts were changed, and no new tests/performance measurements were claimed. The evidence commit is still local/unpublished; the existing publication warning remains. Parent #103 and all children remain open. The other agent should verify current descriptions, not implement the old report's superseded scopes.
upate the qwen code to the newest version of mtplxto Analyse Qwen3.8 Flash Next changes in MTPLX and create confirmed bug issuesChild issue descriptions corrected after independent review (2026-10-03)
Updated #104–#111 and added one change/renewed-verification comment to each. The descriptions now explicitly separate Qwen/MTPLX parity from DSv4 Flash and GLM 5.3 Flash/DS4 parity.
The original report/diagnostics are unchanged historical evidence where superseded by these descriptions. Evidence commit remains local/unpublished. No production implementation, new test result, AGENTS.md edit or push occurred. All issues remain open for renewed independent verification before implementation.
Re-verification of revised child issues (2026-10-03)
Oracle assignment confirmed by the user: DS4 is the parity reference for DeepSeek and GLM; MTPLX (pinned
9882703f3105363ddc37eca9f97aa09a1d387112) is the parity reference for Qwen 3.8 Flash Next. For Qwen, our Rust/Metal code must implement the current MTPLX state, not only patch individual defects. AGENTS.md still states DS4 as the blanket oracle and should be updated to say this explicitly.Re-verification of the revised #104–#111 bodies against code and upstream: all review corrections are incorporated correctly, and all cited upstream SHAs and tests exist at
9882703. #105, #106, #107, #108, #109 and #111 have no remaining gaps.Remaining gaps (details in the respective issue comments):
MTPLX_NGRAM_RESIDENT=1; ngram.rs:14, model.rs:308). On the 128 GiB reference machine default AR is already classic, so the trigger and severity wording must be corrected. The removal scope must cover every engagement site, includingModel::set_ar_pipeline_modewith its PLE staging toggle and the harness read at execution.rs:1188, as in upstream5a9fe002.aa68928dand follow-ups; >= 4,096 routed rows on MLX 0.32.2, canary + fallback) and the guarded stock kernel only for the down projection. DS4Server still has the old copy-gather-split-SiLU chain. Either extend #110's scope or add a linked Qwen issue for the kernel port.Still open: evidence commit
dcc4c51is unpublished, so the commit-pinned links resolve only after it is pushed.Output list addition: #112 Port MTPLX row-map SwiGLU expert gather for Qwen prefill MoE (required route-parity port per the Qwen/MTPLX oracle assignment; bit-identical to the guarded stock chain upstream, so not a numerical bug; depends on #110's guard). With this, the #110 gap from the re-verification comment is owned. Re-verification of the revised #104 and #110 bodies: both correct, no remaining gaps.
bug issues created, this analysis task is done.