Files
DS4Server/docs/issue105-invalid-logits

Issue 105: reject invalid Qwen logits and invalidate the failed session

Qwen now reports numerical failure instead of emitting fabricated IDs for NaN, positive infinity or all-masked logits. Negative-infinity masks with a finite winner remain legal. Failed sessions lose their live, resident and persisted state, while unrelated healthy conversations remain reusable.

Oracle and approved exception

The oracle is MTPLX 9882703f3105363ddc37eca9f97aa09a1d387112, MLX 0.32.2. The unchanged installed Bare-Speed model and environment were verified for ../issue104-ar/environment.json immediately before this issue.

Ordinary greedy uses the upstream any(isnan(row)) | isinf(max(row)) graph, evaluated with argmax. Only failures read the full row for the NaN/infinity census. Sampled support retains its existing GPU route and host-softmax fallback; invalid mass now raises the same numerical error instead of selecting token 0. SparseDistribution no longer invents an ID when positive mass is absent.

The pinned oracle remains inconsistent in specialized greedy MTP paths: generation.py:7577,13295,15371 still use unchecked argmax. The user explicitly approved extending rejection throughout MTP on 2026-10-04. This deliberate invalid-row exception is not claimed as upstream parity. oracle.jsonl records the ordinary error, lazy sentinel and specialized-MTP token 0 side by side. Batched verification validates every row, so a healthy row cannot conceal an all-masked neighbor. Valid choices and RNG draws are unchanged.

The old lazy sampler is only a historical #104/RNG test diagnostic. Its finite law remains historical and its invalid result is now the checked -1 sentinel. No product path calls it. The unrelated diagnostic prefix sampler retains its upstream contract; it is not a production target/draft sampling route.

Ownership and family boundary review

Requirement Implementation and evidence
Reject before consuming a fabricated ID Ordinary AR, MTP primary/target/draft and context-copy target all reach checked sampling; invalid prefill emits no text.
Clear a prefix captured before failure LiveSession discards current and prefix; QwenExecutor resets trunk/head/boundary caches. Installed faults occur after real prefill and prefix capture.
Clear selected, resident and disk state Generator invalidates the failed conversation's checkpoints, resident aliases, indexes and staging files, and clears selection/store frontier.
No stale bootstrap or later persistence First local fault creates then removes a real system bootstrap. A later explicit save fails without a committed snapshot. All failed requests retry cold.
Preserve other sessions A real healthy checkpoint remains byte-identical and resident throughout both product matrices, then restores successfully.
Final capture ownership A test-only fault consumes an invalid row through the real numerical guard during final capture; the resulting error crosses the actual snapshot/request/Generator owners and prevents persistence. Normal capture adds no sampling pass.
Streaming and non-streaming Actual Generator events pass through the production HTTP response adapters; explicit SSE errors/HTTP 500 are checked, followed by valid requests using the same Generator.
DS4/GLM unchanged Shared cleanup requires both QwenMtplx and the numerical-error discriminator. DS4 ds4.c:39410-39460 retains its greedy/no-finite-mass fallback; shared RNG, samplers and checkpoint formats are untouched. Default tests include exact DS4 token/RNG controls.

DS4Server uses content-addressed API conversations rather than MTPLX session IDs. Its cleanup therefore invalidates every ancestor of the failed API request and its selected source. Local chats invalidate their file and system bootstrap. Shared bootstrap prefixes belonging to the failed ancestry are invalidated, while other conversations' complete checkpoints remain intact. This is the DS4Server-specific persistence integration of MTPLX's session-bank eviction.

Checkpoint writes are synchronous on the serialized model worker. Errors return before final persistence; the cleared owner cannot later save or be banked by an outgoing resident swap. Filesystem cleanup failures are reported explicitly. Production remains Rust/Metal; Python is used solely for isolated oracle receipts.

Validation

  • Fresh ordinary-entry fixtures: 1,080 cases, three consecutive draws each, including evaluation boundaries, valid tokens/probabilities and next RNG.
  • Sparse fallback: 150 cases; batched fallback: 653; mixed distributions: 100; historical keyed/lazy diagnostics: 387. Mixed and distribution/RNG tests run in the ordinary cargo suite, not with --ignored.
  • Four synthetic rows at temperatures 0/.6/1, with exact oracle error census, legal tokens and invalid lazy sentinel: native-invalid-rows.jsonl.
  • Batched greedy: both row orders, widths 64/4097, all four rows, including a healthy neighbor beside an all-masked row.
  • Installed AR: 24 cases (prefill, later forward, final capture; four rows; both HTTP modes), 18 invalid/cold-retry cases and six legal-mask successes.
  • Installed MTP: 32 cases, additionally exercising draft-head faults; 24 invalid/cold-retry cases and eight legal-mask successes. A real supported penalty disables context-copy for draft tests and half the target tests, ensuring both target routes and the actual draft head are exercised.
  • All four commit gates: formatting, Clippy with warnings denied, signed app bundle, and all-feature cargo tests. Exact outcomes are in checks.json.

These are correctness and cleanup tests, not throughput measurements. Test-only injection fields, output substitution and HTTP harness are absent from release code. The initial test runs deliberately failed if the requested fault boundary was bypassed; the final matrices reach every specified boundary.

Reproduction

Build with cargo test --all-features --bin ds4-server --no-run and cargo build --bin test-supervisor. Resolve the executable printed by Cargo. Run model-free ignored tests serially under test-supervisor 4096 60 120 --command <test-executable> <test-name> --ignored --nocapture --test-threads=1.

For issue105_product_ar_cleanup_and_http_retry and issue105_product_mtp_cleanup_and_http_retry, use memory limit 114688 MiB, DS4_QWEN38_ARTIFACTS pointing to the installed unchanged model, and a fresh DS4_ISSUE105_DIR per run. Keep model processes serial. For the oracle, run tools/issue105-reference.py in the isolated MLX 0.32.2 environment with PYTHONPATH pointing at the pinned MTPLX checkout.