96 lines
6.5 KiB
Markdown
96 lines
6.5 KiB
Markdown
# Issue 105: reject invalid Qwen logits and invalidate the failed session
|
|
|
|
Qwen now reports numerical failure instead of emitting fabricated IDs for NaN,
|
|
positive infinity or all-masked logits. Negative-infinity masks with a finite
|
|
winner remain legal. Failed sessions lose their live, resident and persisted
|
|
state, while unrelated healthy conversations remain reusable.
|
|
|
|
## Oracle and approved exception
|
|
|
|
The oracle is MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`, MLX 0.32.2.
|
|
The unchanged installed Bare-Speed model and environment were verified for
|
|
`../issue104-ar/environment.json` immediately before this issue.
|
|
|
|
Ordinary greedy uses the upstream `any(isnan(row)) | isinf(max(row))` graph,
|
|
evaluated with argmax. Only failures read the full row for the NaN/infinity
|
|
census. Sampled support retains its existing GPU route and host-softmax fallback;
|
|
invalid mass now raises the same numerical error instead of selecting token 0.
|
|
SparseDistribution no longer invents an ID when positive mass is absent.
|
|
|
|
The pinned oracle remains inconsistent in specialized greedy MTP paths:
|
|
`generation.py:7577,13295,15371` still use unchecked argmax. The user explicitly
|
|
approved extending rejection throughout MTP on 2026-10-04. This deliberate
|
|
invalid-row exception is not claimed as upstream parity. `oracle.jsonl` records
|
|
the ordinary error, lazy sentinel and specialized-MTP token 0 side by side.
|
|
Batched verification validates every row, so a healthy row cannot conceal an
|
|
all-masked neighbor. Valid choices and RNG draws are unchanged.
|
|
|
|
The old lazy sampler is only a historical #104/RNG test diagnostic. Its finite
|
|
law remains historical and its invalid result is now the checked -1 sentinel.
|
|
No product path calls it. The unrelated diagnostic prefix sampler retains its
|
|
upstream contract; it is not a production target/draft sampling route.
|
|
|
|
## Ownership and family boundary review
|
|
|
|
| Requirement | Implementation and evidence |
|
|
| --- | --- |
|
|
| Reject before consuming a fabricated ID | Ordinary AR, MTP primary/target/draft and context-copy target all reach checked sampling; invalid prefill emits no text. |
|
|
| Clear a prefix captured before failure | LiveSession discards current and prefix; QwenExecutor resets trunk/head/boundary caches. Installed faults occur after real prefill and prefix capture. |
|
|
| Clear selected, resident and disk state | Generator invalidates the failed conversation's checkpoints, resident aliases, indexes and staging files, and clears selection/store frontier. |
|
|
| No stale bootstrap or later persistence | First local fault creates then removes a real system bootstrap. A later explicit save fails without a committed snapshot. All failed requests retry cold. |
|
|
| Preserve other sessions | A real healthy checkpoint remains byte-identical and resident throughout both product matrices, then restores successfully. |
|
|
| Final capture ownership | A test-only fault consumes an invalid row through the real numerical guard during final capture; the resulting error crosses the actual snapshot/request/Generator owners and prevents persistence. Normal capture adds no sampling pass. |
|
|
| Streaming and non-streaming | Actual Generator events pass through the production HTTP response adapters; explicit SSE errors/HTTP 500 are checked, followed by valid requests using the same Generator. |
|
|
| DS4/GLM unchanged | Shared cleanup requires both QwenMtplx and the numerical-error discriminator. DS4 `ds4.c:39410-39460` retains its greedy/no-finite-mass fallback; shared RNG, samplers and checkpoint formats are untouched. Default tests include exact DS4 token/RNG controls. |
|
|
|
|
DS4Server uses content-addressed API conversations rather than MTPLX session IDs.
|
|
Its cleanup therefore invalidates every ancestor of the failed API request and
|
|
its selected source. Local chats invalidate their file and system bootstrap.
|
|
Shared bootstrap prefixes belonging to the failed ancestry are invalidated,
|
|
while other conversations' complete checkpoints remain intact. This is the
|
|
DS4Server-specific persistence integration of MTPLX's session-bank eviction.
|
|
|
|
Checkpoint writes are synchronous on the serialized model worker. Errors return
|
|
before final persistence; the cleared owner cannot later save or be banked by an
|
|
outgoing resident swap. Filesystem cleanup failures are reported explicitly.
|
|
Production remains Rust/Metal; Python is used solely for isolated oracle receipts.
|
|
|
|
## Validation
|
|
|
|
- Fresh ordinary-entry fixtures: 1,080 cases, three consecutive draws each,
|
|
including evaluation boundaries, valid tokens/probabilities and next RNG.
|
|
- Sparse fallback: 150 cases; batched fallback: 653; mixed distributions: 100;
|
|
historical keyed/lazy diagnostics: 387. Mixed and distribution/RNG tests run
|
|
in the ordinary cargo suite, not with `--ignored`.
|
|
- Four synthetic rows at temperatures 0/.6/1, with exact oracle error census,
|
|
legal tokens and invalid lazy sentinel: `native-invalid-rows.jsonl`.
|
|
- Batched greedy: both row orders, widths 64/4097, all four rows, including a
|
|
healthy neighbor beside an all-masked row.
|
|
- Installed AR: 24 cases (prefill, later forward, final capture; four rows;
|
|
both HTTP modes), 18 invalid/cold-retry cases and six legal-mask successes.
|
|
- Installed MTP: 32 cases, additionally exercising draft-head faults; 24
|
|
invalid/cold-retry cases and eight legal-mask successes. A real supported
|
|
penalty disables context-copy for draft tests and half the target tests,
|
|
ensuring both target routes and the actual draft head are exercised.
|
|
- All four commit gates: formatting, Clippy with warnings denied, signed app
|
|
bundle, and all-feature cargo tests. Exact outcomes are in `checks.json`.
|
|
|
|
These are correctness and cleanup tests, not throughput measurements. Test-only
|
|
injection fields, output substitution and HTTP harness are absent from release
|
|
code. The initial test runs deliberately failed if the requested fault boundary
|
|
was bypassed; the final matrices reach every specified boundary.
|
|
|
|
## Reproduction
|
|
|
|
Build with `cargo test --all-features --bin ds4-server --no-run` and
|
|
`cargo build --bin test-supervisor`. Resolve the executable printed by Cargo.
|
|
Run model-free ignored tests serially under `test-supervisor 4096 60 120
|
|
--command <test-executable> <test-name> --ignored --nocapture --test-threads=1`.
|
|
|
|
For `issue105_product_ar_cleanup_and_http_retry` and
|
|
`issue105_product_mtp_cleanup_and_http_retry`, use memory limit 114688 MiB,
|
|
`DS4_QWEN38_ARTIFACTS` pointing to the installed unchanged model, and a fresh
|
|
`DS4_ISSUE105_DIR` per run. Keep model processes serial. For the oracle, run
|
|
`tools/issue105-reference.py` in the isolated MLX 0.32.2 environment with
|
|
`PYTHONPATH` pointing at the pinned MTPLX checkout.
|