DS4Server/docs/issue103-analysis/README.md

MTPLX 2.12.2 review handoff for #103

Analysis only, 2026-10-03. Eight independently reproduced native defects were filed as bugs. No production behavior, model revision, runtime shader, default tuning or tokenizer asset was changed. Implementation requires the user's review and subsequent instruction. Some requested stress/branch matrices remain unmeasured; the inventory distinguishes them from confirmed failures.

Immutable comparison and inputs

• Affected DS4Server: cea8fc6f7c7aa7aeb97ebc303594ae4088ef6983.
• Previous oracle: MTPLX 2.10.2 e652d55e2652137a4abcf1312357abbf3eb9d692.
• Refreshed HEAD and peeled v2.12.2: 9882703f3105363ddc37eca9f97aa09a1d387112. The range contains 1,077 commits, including merges; applicability was narrowed using source, tests and release notes rather than treating every change as relevant to this backend.
• Immutable release notes, runtime pin.
• Unchanged Youssofal/Qwen3.8-Flash-Next-MTPLX-Bare-Speed, revision 74559cdf34fbfc0b593de72d17e93f37fd4f9ea7, MLX exactly 0.32.2. Model/config/tokenizer/template/vision/MTP/PLE files were streamed through SHA256 and every checked file matches its stored digest. environment.json contains actual hashes, sizes, runtime/compiler, hardware/OS, prompt digest and effective model-test options.
• M5 Max/Mac17,7/128GiB, macOS26.6.2, Metal32023.883. Reference environment: mlx-lm0.31.3, transformers5.8.0, tokenizers0.22.2. Both oracle checkouts were retained separately; Python is used only for isolated reference probes.
• Fresh model execution with SSD PLE; each model case restores the same initial prompt snapshot. This is not a claim of cold SSD pages. Tests add synchronization/readbacks and a watchdog; their timings cannot establish product speed. Synthetic cases use explicit dtypes/shapes/values in the test, no image/template or API context-admission setting.

Confirmed bugs and diagnostic ownership

Area	Issue	Observed failure	Owned receipt cases
A1	#104	Pipeline top-p normalizes only top-k. Minimal expected support {0,1} becomes {0}; official .6/.95/k20 also loses valid candidates.	sampling_law, sampling_matrix in native-sampling.jsonl
A2	#105	NaN/+inf/all-inf rows fabricate token IDs; finite-winner -inf masks remain legitimate.	invalid_logits, invalid_logits_lazy, same file
B	#106	First-primary stop/limit banks a four-token key over three-token trunk state; next-turn 247,643 logits differ.	All native-model.jsonl cases
D1	#107	Vision presence discards sparse QSA selection, changing actual Attention output with identical positions. Image-aware indexer threading is missing in source.	vision_sparse_qsa in native-vision.jsonl
D2	#108	A straddling chunk applies plain RoPE to its table-covered image rows; two wrong rows, inside/outside controls exact.	vision_positions, same file
E	#109	Actual BF16 attention gate differs from latest compiled verify law at -6.84375.	active_attention_gate, same file; bf16_gate in native-numeric.jsonl
F	#110	M5 sorted expert gather leaves 32/1,280/8,192 rows unwritten at 3,277/3,404/4,095 tokens; 3,276 and 4,096 controls pass.	native-gather.jsonl
I	#111	Qwen native pretokenizer misclassifies Unicode number categories, changing exact IDs for circled/Roman/superscript/mathematical numerals.	native-tokenizer.jsonl

Each issue contains caller/trigger, actual vs expected, immutable upstream links, severity, native scope, limitations, regression ownership and product acceptance. Existing issues were searched in all states; #93–99 own the completed integrations, not these subsequently reproduced regressions. #100/#102 and optional paging #77 have different roots. No duplicate was created for an existing root cause.

Applicability and evidence inventory A–J

Area	Disposition and active-route evidence	What is still unmeasured
A	Two confirmed bugs above. policy::server_overrides → ar::decode enables pipeline AR when eligible; ordinary target/draft support uses full vocabulary. Latest a2d0b4a and mask tests distinguish invalid from legal masked rows. p=1 control preserves sampling law.	HTTP error injection, invalid-session cleanup and valid retry; MTP invalid target/draft end-to-end. These are explicit #105 acceptance, not observed cleanup failures.
B	Confirmed first-primary stop and max-token exits, depths1/2/3, greedy and native .6 sampler, real model. pending.take/early break bypasses final commit; final_arrays/capture trusts completeness. Trunk offset3, MTP2 versus bank key4. Warm teacher-force comparison independently confirms missing commit.	Fresh-primary/correction/bonus/accepted-draft exits, cancellation and final-capture failure. GDN/PLE remain at prefill because no final forward runs; individual tensor hashes for each site were not measured. #106 owns the full matrix.
C	Source-verified non-applicability of upstream tiny-gap tolerance: request::LiveSession requires token-exact complete starts_with plus content identity; owner::align_prompt resets mismatching prompts. restore_session rejects non-prefix before replacing caches. Outer checkpoint requires exact model/context/vocabulary/MTP identity; interrupted/incompatible/corrupt restore tests pass. No attention-only trimming of a later recurrent state was found. B can still make a token-exact bank lie about its state.	Product edits1/4/8/9/40 tokens before end; alternate tokenization/reasoning echo, whitespace, tool-schema ordering/choice/argument/system-message matrices; restart/truncated history. A safe miss is not a confirmed bug. Do not infer all warm reuse correct from checkpoint tests.
D	Two confirmed active Attention defects. qsa_indexer and qsa_prepare use plain query/block-start positions; latest 03a0a0c/18087e7/8172196 establishes corrected vision contract. Native table builder/rendering already exists. request::LiveSession gates on qwen_vision::same_prefix content/whole-image identity; whole-image policy is coarse, with no demonstrated incorrect restore.	Real dim128/topk512 image indexer selection and full long-image logits; multiple-image edit/content/cancel matrices and proof of all restore-span boundaries. Source indexer omission is within #107 scope but has no independent real-image receipt yet.
E	BF16 attention verification mismatch is confirmed. PLE/hyper/shared-expert gate lowering must be reviewed by phase; their independent active-site failures have not been asserted. A separate generic compiler probe demonstrates seven-digit constants changing sqrt128 and YaRN factor4 bits. No scalar-format bug issue: qsa_core uses qsa_compile::Body.trace/Replay::trace with exact captured operands; QSA prepare/select passes scales separately; whole model runs compiles only linear GDN spans without PLE. Full QSA/PLE fixed-M4 compiler is not installed. qsa_graph is test-only.	Real 128-dim/512-block near-cutoff IDs/order under actual routes; static YaRN factor4 ordinary/vision; PLE/hyper/shared gate site reproduction and full-model verify/logit comparison. qsa_divisor/yarn_amplitude receipts are a dormant compiler hazard owned by this report, not proof of active model divergence.
F	Confirmed same-dispatcher row defect under M5/0.32.2/large supported chunks; default2048 is below trigger. Unaligned >32767 sorted routes + BM64 produces unwritten sentinel rows. Upstream b36b106 provides pad/slice contract. Pinned MTP header has58 tensors with explicit switch_mlp gate/up/down and scales/biases. binding::Bank.load includes mtp.* and binds decoder/feed-forward/linear strictly; full native MTP model load succeeds. mtp-binding.json records header shapes/dtypes. Qwen3.5/3.6 fused naming fix13128e2 is not automatically applicable to these keys.	Full model larger-chunk text output and both gate/up/down real geometries. #110 owns those tests; no loader bug was observed.
G	Source-verified dynamic reserve-before-update in cache::KvCache/QsaCache; existing native KV/QSA growth/trim/snapshot controls pass. copy_round feeds ordinary model forward, not the complete upstream fixed-M4/paged replay owner. Presence of compiled primitives does not establish that lane. No new cache-capacity defect has been demonstrated outside B.	Accepted/rejected copy windows/final pending writes on full-model growth boundaries; verifier width transitions; OOM atomicity/recovery; 64K/128K/near-native-max memory/head chunking and snapshot copy peaks. Existing finite cache controls do not discharge these stress obligations.
H	Normal policy keeps lazy_history=false; append_committed evaluates MTP history even without later draft use. eval::evaluate detaches evaluated graph nodes/events; native eval failure/retry, donation, tracer topology, siblings/continuations, graph-event/fence and checkpoint ownership tests pass. Allocator already tracks active/cached/peak/resources, reclaims cached buffers and limits memory to min(1.5×recommended,.95×RAM). Product owner drops snapshots before context and publishes checkpoint only after successful validation.	Long streamed/non-streamed event marathon, many-session pending persistence, interrupted request memory/admission retry and whole-Mac reclaimable/compressor peaks. Short tests and one ~73.9GB model peak do not prove absence of a leak. Latest admission accounting is a policy/measurement candidate until a concrete local defect is reproduced. 2.12.2 dense-model/small-Mac repair is not evidence of a Flash-Next128GiB failure.
I	Confirmed native Unicode category bug; Hindi/Thai/vowelled Arabic combining marks and English/code/Chinese/mixed punctuation match exact trained IDs. Upstream transformers loader repair is not our implementation, but its corrected regex/oracle exposes local category error. Both old and refreshed resilient tokenizer agree on those IDs.	Wider Unicode category coverage and rendered API warm/context accounting belong to #111 acceptance.
J	Optimization candidates only. New MTPLX kernel/default engagement reviewed against native model/MoE/QSA/history routes. No optional optimization was filed as a bug without measured supported-contract violation.	Paired serial Bare-Speed workloads for TTFT, prefill/decode, accepted tokens/depth, dispatch/readback, memory/thermal/variance. No new product performance regression or 60tok/s/5%/75.9coding acceptance result was measured.

Important upstream follow-ups: long-context verify head chunking, lazy write lifetime, history scheduling. Their upstream failures alone are not local bug proof.

Optimization opportunities retained for separate authorization

• M5 4096 prefill/sparse-from16K needs #110 resolved and exact output/state validation before tuning.
• Cache-direct sparse prefill, simdgroup row selectors, runtime block counts and less recompilation.
• Fused hyper read/write, GDN prework/norm, expert combine and M5 dequantize/GEMM; actual current compiled spans exclude QSA/PLE.
• Cache-only MTP history, reduced readbacks, PLE read-ahead/overlap.
• Full fixed-M4 model replay, verify-width pooled/QSA/HC fusion and routed-down reduction; installing that owner makes presently dormant capacity/scalar obligations active.
• 8192 capacity buckets, one-copy/per-layer growth, recurrent boundary retention and content-exact image reuse.

Validate semantics first and then paired performance on unchanged Bare-Speed/M5 Max. Published Optimized-Speed numbers and kernel-only measurements do not satisfy this repository's same-machine contract.

Reproduce without this chat

issue103_analysis.rs is cfg(test) and ignored normally. It deliberately pins/prints broken behavior on the affected revision; a green diagnostic means confirmed reproduction. Future fixes must turn owned assertions into correct-contract regression tests, not weaken the oracle. Six tests share minimal primitives and actual native Attention/model paths. No upstream production code/test suite was copied into production.

cargo test --all-features --bin ds4-server issue103_ --no-run
cargo build --bin test-supervisor
export DS4_QWEN38_ARTIFACTS='/path/to/the/unchanged/qwen3.8-flash-next'
export DS4_ISSUE103_ORACLE="$PWD/docs/issue103-analysis/latest-oracle.jsonl"
# Use the ds4_server test executable printed by cargo (not cargo itself).
# Full model requires enough RAM for SSD-PLE Bare-Speed; limit is 98GiB.
target/debug/test-supervisor 100352 60 120 --command /path/to/test-binary issue103_ --ignored --nocapture --test-threads=1

Model-free sampling/numeric/vision/gather cases can be selected individually under a4096MiB watchdog. Tokenizer additionally needs the artifact and oracle environment variables. issue103_mtp_final_receipts is the only full-model test. On a different Metal device, don't treat failure to select the M5 NAX kernel as reproducing #110.

For the independent Python oracle, install each exact MTPLX revision in a separate checkout. Use MLX0.32.2 and the recorded tokenizer library versions; in the existing analysis environment the command was:

PYTHONPATH=/path/to/mtplx-9882703f /path/to/reference-python tools/issue103-reference.py "$DS4_QWEN38_ARTIFACTS"
# Repeat with the e652d55e checkout; do not change tokenizer assets.

The probe reads only tokenizer/config/template/merges; no model tensors. latest-oracle.jsonl and old-oracle.jsonl are preserved even if the external scratch environments disappear. bf16_verify_gate is unavailable on the old reference and recorded explicitly. Bad-row oracle exceptions are deliberate; warnings are not harness failures.

Targeted upstream test files: non_finite_logits_and_lazy_sampler_law, mtp_final_state_covers_committed_tokens, qwen4_vision_mrope, float32_operand, qwen4_yarn_amplitude, moe_sorted_gather, qwen4_moe_sorted_rows, tokenizer_pretokenizer_repair; additionally attention_gate_exactness and qwen4_verify_gate_sites. See upstream-tests.log and upstream-gates.log. One installed27B asset-dependent tokenizer case was skipped; native exact Bare-Speed tokenizer was separately exercised.

Existing native controls were run serially with their own watchdogs: mtplx_kv_cache_graph_matches_reference, mtplx_qsa_cache_graph_matches_reference, graph-event/fence tests and qwen_checkpoint_roundtrip_and_failed_restore_preserve_live_state. native-controls.log records exact filters/results; native-eval-controls.log covers four eval/lifetime/retry tests. These test their finite scenarios, not unrestricted long-run resource behavior.

Delivery checks and unresolved verification

The durable evidence commit contains only cfg(test) diagnostics, an isolated reference script and these reports/receipts. All confirmed failures have owning issues; the generic scalar-format experiment belongs to this report with the dormant-route disposition above.

The broad branch/stress cases in the inventory still need evidence before claiming those paths correct. They are explicit review/acceptance or follow-up investigation obligations, not additional confirmed bugs and not authorization for production changes. #103 remains open to keep that unfinished verification visible.

Validation completed:

• cargo fmt --all -- --check: passed.
• cargo clippy --all-targets --all-features -- -D warnings: passed (log).
• make bundle: passed including app packaging/ad-hoc signature (log).
• cargo test --all-features: first parallel run had two unrelated existing test failures (log): a generated seven-digit all-numeric git prefix was parsed as a YAML integer; process-wide CPU accounting exceeded its5ms concurrent tolerance. No unrelated code was changed. Justified serial rerun cargo test --all-features -- --test-threads=1: passed307 main tests,210 ignored, plus8 tool tests (log).
• Explicit ignored native diagnostics: six passed; final sampler matrix and tokenizer assertions separately passed after their additions. Existing ignored native controls all passed. Targeted official upstream gates:12 passed; other selected upstream tests passed with one installed27B tokenizer asset case skipped. Ignored full-model tests outside this analysis were not broadly executed.

Review the remaining inventory obligations before claiming exhaustive branch, long-context, resource or performance coverage.