Reject invalid Qwen logits and discard poisoned sessions

docs/issue105-invalid-logits/README.md

@@ -0,0 +1,95 @@
+# Issue 105: reject invalid Qwen logits and invalidate the failed session
+
+Qwen now reports numerical failure instead of emitting fabricated IDs for NaN,
+positive infinity or all-masked logits. Negative-infinity masks with a finite
+winner remain legal. Failed sessions lose their live, resident and persisted
+state, while unrelated healthy conversations remain reusable.
+
+## Oracle and approved exception
+
+The oracle is MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2.
+The unchanged installed Bare-Speed model and environment were verified for
+`../issue104-ar/environment.json` immediately before this issue.
+
+Ordinary greedy uses the upstream `any(isnan(row)) | isinf(max(row))` graph,
+evaluated with argmax. Only failures read the full row for the NaN/infinity
+census. Sampled support retains its existing GPU route and host-softmax fallback;
+invalid mass now raises the same numerical error instead of selecting token 0.
+SparseDistribution no longer invents an ID when positive mass is absent.
+
+The pinned oracle remains inconsistent in specialized greedy MTP paths:
+`generation.py:7577,13295,15371` still use unchecked argmax. The user explicitly
+approved extending rejection throughout MTP on 2026-10-04. This deliberate
+invalid-row exception is not claimed as upstream parity. `oracle.jsonl` records
+the ordinary error, lazy sentinel and specialized-MTP token 0 side by side.
+Batched verification validates every row, so a healthy row cannot conceal an
+all-masked neighbor. Valid choices and RNG draws are unchanged.
+
+The old lazy sampler is only a historical #104/RNG test diagnostic. Its finite
+law remains historical and its invalid result is now the checked -1 sentinel.
+No product path calls it. The unrelated diagnostic prefix sampler retains its
+upstream contract; it is not a production target/draft sampling route.
+
+## Ownership and family boundary review
+
+| Requirement | Implementation and evidence |
+| --- | --- |
+| Reject before consuming a fabricated ID | Ordinary AR, MTP primary/target/draft and context-copy target all reach checked sampling; invalid prefill emits no text. |
+| Clear a prefix captured before failure | LiveSession discards current and prefix; QwenExecutor resets trunk/head/boundary caches. Installed faults occur after real prefill and prefix capture. |
+| Clear selected, resident and disk state | Generator invalidates the failed conversation's checkpoints, resident aliases, indexes and staging files, and clears selection/store frontier. |
+| No stale bootstrap or later persistence | First local fault creates then removes a real system bootstrap. A later explicit save fails without a committed snapshot. All failed requests retry cold. |
+| Preserve other sessions | A real healthy checkpoint remains byte-identical and resident throughout both product matrices, then restores successfully. |
+| Final capture ownership | A test-only fault consumes an invalid row through the real numerical guard during final capture; the resulting error crosses the actual snapshot/request/Generator owners and prevents persistence. Normal capture adds no sampling pass. |
+| Streaming and non-streaming | Actual Generator events pass through the production HTTP response adapters; explicit SSE errors/HTTP 500 are checked, followed by valid requests using the same Generator. |
+| DS4/GLM unchanged | Shared cleanup requires both QwenMtplx and the numerical-error discriminator. DS4 `ds4.c:39410-39460` retains its greedy/no-finite-mass fallback; shared RNG, samplers and checkpoint formats are untouched. Default tests include exact DS4 token/RNG controls. |
+
+DS4Server uses content-addressed API conversations rather than MTPLX session IDs.
+Its cleanup therefore invalidates every ancestor of the failed API request and
+its selected source. Local chats invalidate their file and system bootstrap.
+Shared bootstrap prefixes belonging to the failed ancestry are invalidated,
+while other conversations' complete checkpoints remain intact. This is the
+DS4Server-specific persistence integration of MTPLX's session-bank eviction.
+
+Checkpoint writes are synchronous on the serialized model worker. Errors return
+before final persistence; the cleared owner cannot later save or be banked by an
+outgoing resident swap. Filesystem cleanup failures are reported explicitly.
+Production remains Rust/Metal; Python is used solely for isolated oracle receipts.
+
+## Validation
+
+- Fresh ordinary-entry fixtures: 1,080 cases, three consecutive draws each,
+  including evaluation boundaries, valid tokens/probabilities and next RNG.
+- Sparse fallback: 150 cases; batched fallback: 653; mixed distributions: 100;
+  historical keyed/lazy diagnostics: 387. Mixed and distribution/RNG tests run
+  in the ordinary cargo suite, not with `--ignored`.
+- Four synthetic rows at temperatures 0/.6/1, with exact oracle error census,
+  legal tokens and invalid lazy sentinel: `native-invalid-rows.jsonl`.
+- Batched greedy: both row orders, widths 64/4097, all four rows, including a
+  healthy neighbor beside an all-masked row.
+- Installed AR: 24 cases (prefill, later forward, final capture; four rows;
+  both HTTP modes), 18 invalid/cold-retry cases and six legal-mask successes.
+- Installed MTP: 32 cases, additionally exercising draft-head faults; 24
+  invalid/cold-retry cases and eight legal-mask successes. A real supported
+  penalty disables context-copy for draft tests and half the target tests,
+  ensuring both target routes and the actual draft head are exercised.
+- All four commit gates: formatting, Clippy with warnings denied, signed app
+  bundle, and all-feature cargo tests. Exact outcomes are in `checks.json`.
+
+These are correctness and cleanup tests, not throughput measurements. Test-only
+injection fields, output substitution and HTTP harness are absent from release
+code. The initial test runs deliberately failed if the requested fault boundary
+was bypassed; the final matrices reach every specified boundary.
+
+## Reproduction
+
+Build with `cargo test --all-features --bin ds4-server --no-run` and
+`cargo build --bin test-supervisor`. Resolve the executable printed by Cargo.
+Run model-free ignored tests serially under `test-supervisor 4096 60 120
+--command <test-executable> <test-name> --ignored --nocapture --test-threads=1`.
+
+For `issue105_product_ar_cleanup_and_http_retry` and
+`issue105_product_mtp_cleanup_and_http_retry`, use memory limit 114688 MiB,
+`DS4_QWEN38_ARTIFACTS` pointing to the installed unchanged model, and a fresh
+`DS4_ISSUE105_DIR` per run. Keep model processes serial. For the oracle, run
+`tools/issue105-reference.py` in the isolated MLX 0.32.2 environment with
+`PYTHONPATH` pointing at the pinned MTPLX checkout.