Optimize Qwen3.8 inference on Apple silicon

docs/QWEN38_PERFORMANCE.md

@@ -0,0 +1,146 @@
+# Qwen3.8 Flash Next performance and parity receipt
+
+Issue #99 optimizes the pinned `Youssofal/Qwen3.8-Flash-Next-MTPLX-Bare-Speed`
+text model without changing its artifact precision, sampler, or token behavior.
+All application and inference work remains Rust; native additions are confined to
+the existing Objective-C Metal bridge and Qwen `.metal` kernels. Integrated MTP
+is the Qwen production default; an explicitly stored user opt-out remains
+authoritative.
+
+## Result
+
+The acceptance B1 median is **60.825779 tok/s MTP**. A later B2 confirmation,
+after adding complete repair telemetry and under the then-current machine load,
+measured **58.019436 tok/s**. Both sequences produced the same 64-token SHA-256
+`46ce63601c44d80b5ef8297eaa49a636f6680040b916fe30cf7a154c05daef72`, with 44
+accepted drafts from 60 proposals in 20 verifier passes. The fresh matched MTPLX
+median was 59.751170 tok/s, putting B1 1.8% above and B2 2.9% below the oracle;
+both are inside the issue's 5% parity band. Raw per-run data is in
+`docs/qwen38-issue-99.csv`.
+
+Plain AR reached a 50.671077 tok/s median with stable token hash
+`15373689bcb1c667123accfcfbe49736466f28ce7beb89d2cbe5e4120416056d`.
+The published 47 tok/s MTPLX figure is contextual rather than a fresh matched
+AR baseline: after the valid MTP oracle run, the installed MTPLX package and
+checked-out source drifted apart. The current CLI rejects this Flash Next pack
+as unsupported and the source path fails at the missing
+`DecoderLayer.input_layernorm`, so no invalid number is presented as a fresh AR
+comparison.
+
+The published 75.9 tok/s MTP result is from a different MTPLX runtime/workload.
+With identical pinned artifacts, 4,096 context, prompt, sampling, seed, and
+64-token measurement window, the current usable oracle result is 59.751170
+tok/s. The remaining gap is therefore not explained by quantization drift or a
+different output distribution.
+
+Maximum fan control could not be established because neither ThermalForge nor
+TG Pro is installed; MTPLX reports its fan boost as unavailable. Both compared
+engines ran under the same no-boost constraint. No system utility was installed
+or substituted for the missing control.
+
+## Fixed workload
+
+- Hardware: MacBook Pro `Mac17,7`, Apple M5 Max, 18 CPU cores, 128 GiB unified
+  memory.
+- OS: macOS 26.6.2, build 25G83, arm64.
+- Model revision: `74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`.
+- Model manifest contract SHA-256:
+  `6f1172de47fa30b9602e13fc7ad14e578a813bb5b1fab8f0ef320041aac6c19a`.
+- Tensor inventory SHA-256:
+  `b5731e6febcf865d276a0e7b144da02375f2d7e3129594f271ee9c6c351f4c8f`.
+- `config.json` SHA-256:
+  `826302af9ff3293cfc9975c9a5a96546fa0aa9f0808505adfc30a62fef806965`.
+- `mtplx_runtime.json` SHA-256:
+  `381c7666872f878542494a48a874bb0784c72a955d18cc077324d3c9e74b8cdb`.
+- Prompt: `Write a concise Rust implementation of binary search and explain its invariants.`
+- Prompt tokens: 26; context: 4,096; generated tokens: 64.
+- Sampling: temperature 1.0, top-p 0.95, top-k 20, min-p 0, seed 12,345.
+- One process, three runs: first run cold, following runs warm.
+
+The exact DS4Server commands were:
+
+```sh
+DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
+DS4SERVER_QWEN38_MEASURED=64 DS4SERVER_QWEN38_RUNS=3 \
+cargo test --release qwen_m5_performance_gate -- --ignored --nocapture
+
+DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
+DS4SERVER_QWEN38_MTP=1 DS4SERVER_QWEN38_MEASURED=64 \
+DS4SERVER_QWEN38_RUNS=3 \
+cargo test --release qwen_m5_performance_gate -- --ignored --nocapture
+
+DS4SERVER_QWEN38_SOURCE=/private/tmp/qwen38-bare-source \
+cargo test --release qwen_mtp_matches_target_and_restores_its_cache \
+  -- --ignored --nocapture
+```
+
+The MTPLX comparison used its normal `serve` path with the same local Bare Speed
+directory, generation mode, prompt, context, sampler, seed, and single request
+stream. A later retry is not a valid benchmark because the local MTPLX
+source/package compatibility gate fails as described above.
+
+## MTPLX logic review
+
+The production path now follows the MTPLX/MLX graph rather than retaining the
+temporary Qwen experiments used while earlier issues established correctness:
+
+- PLE uses the artifact's 4-bit, group-32 rows, streams them with `pread`, keeps
+  a bounded hot cache, and publishes hit, miss, eviction, byte, and read-time
+  counters. The warm MTP runs had 2,080 hits, no misses, and no reads.
+- GDN uses fused input projection, MTPLX-equivalent conv+SiLU+L2 normalization,
+  BF16 rounding points, decay/delta recurrence, gated norm, and direct verifier
+  capture buffers. Rejected rows restore or advance authoritative recurrence
+  rather than inventing state.
+- QSA uses the MTPLX `qsa_flash_skip` row ownership, online-softmax merge, and
+  native 256-thread/eight-SIMD layout. Long-context state remains linear and
+  admitted through 262,144 tokens without a gather-copy fallback.
+- MoE uses the exact router ownership and merged gate/up projection. The custom
+  verifier MoE experiment and NAX path were removed because they are not the
+  oracle's promoted path.
+- MTP uses the same checkpoint, history, proposal, verifier, rejection-repair,
+  full-accept bonus, and NumPy-compatible PCG64 sampling rules. The exact cache
+  restoration test passes after the final telemetry change.
+- Sparse top-k sampling stays on device for the supported official sampler and
+  uses the existing exact CPU path only when the requested sampler is outside
+  that kernel's contract. Full production logits are not copied to the CPU.
+- Capability fallbacks remain exact implementations, not alternate Qwen
+  behavior. Experiments that changed tokens (normal Metal math, wider Tensor
+  QMM, and recurrence-only repair) were removed rather than left dormant.
+
+The 4-bit/group-64 core, declared 8-bit router/QSA matrices, BF16 recurrent and
+normalization tensors, 4-bit/group-32 PLE, and published 2/4/8-bit MTP tensors
+are consumed as stored. No blanket up-conversion or replacement quantization is
+used. Peak RSS in the final MTP sequence was 52.61 GB, down from the original
+102.65 GB path; the 105.44 GB artifact size includes a 32.00 GB file-backed PLE
+table that is not resident wholesale.
+
+## A/B evidence
+
+The optimization work used exact hashes or tensor comparisons before promotion:
+
+| Area | Before | After | Decision |
+| --- | ---: | ---: | --- |
+| Resident memory | 102.65 GB RSS | 52.61 GB RSS | Reused staging and streamed PLE promoted |
+| Router | 49.7 ms GPU | 18.7 ms GPU, 23.4 ms wall | Row-owned reduced router promoted |
+| 16K QSA AR | 2.6 tok/s | 45.7 tok/s | MTPLX flash-skip rewrite promoted |
+| Shared verifier gate/up | 54.7 tok/s | 58.3 tok/s | Exact fused projection promoted |
+| Direct GDN verifier capture | 58.5/58.7 tok/s | 60.8 tok/s | Direct capture buffers promoted |
+
+The direct-capture B1 acceptance was followed by the B2 confirmation in the CSV;
+both retained the same token hash. Normal Metal math, Tensor QMM at verifier
+width, and shortened recurrence repair failed exact-oracle checks and were
+removed. Custom MoE verify, fused GDN output projection, and async AR were not
+promoted because the MTPLX evidence did not show a win and DS4Server profiles did
+not justify overriding that result.
+
+## Context and verification
+
+The long-context gate exercised 2,048, 16,384, 65,536, 131,072, and the native
+262,144-token maximum. AR remained 43.488 tok/s at the maximum, and live state
+grew from 58,195,968 bytes at 2K to 7,449,083,904 bytes at 262K without collapse
+or over-admission. The CSV records every context result and state size.
+
+Final issue verification comprises the exact MTP cache/token oracle, affine
+QMV/QMM MLX reference comparisons, focused Qwen tests, the long-context state
+gate, the release performance gate, formatting, Clippy with warnings denied, the
+macOS application bundle, and the full all-features Rust test suite.