Retire the Qwen AR pipeline

docs/issue104-ar/README.md

@@ -0,0 +1,120 @@
+# Issue 104: retire the Qwen AR pipeline
+
+The Qwen AR driver now samples every token through `sample_from_logits` and
+the request's PCG64 stream, following MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`.
+The obsolete pipeline option, admission, keyed decode loop, cached speculative
+tail, model mode and compile-policy lane are removed. Classic tail capture
+forwards the final sampled token before banking its state.
+
+The server no longer supplies or reads `MTPLX_AR_PIPELINE`. The installed-model
+reference harness cannot enable it either. Generic PLE staging/resident-table
+fields remain for their independently tested callers. Their classic defaults
+are preserved. The historical lazy sampling primitive and its scalar keyed
+categorical wrapper are test-only, retained to reproduce the original sampling
+law receipt and exercise the existing RNG fixtures.
+
+## Baseline and oracle
+
+Pre-fix production revision: `5773942c8ffe41352104d673ff0eef949df10c71`, with only
+the installed-model diagnostic added. Model: unchanged Bare-Speed revision
+`74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`; MLX 0.32.2; Apple M5 Max / 128 GiB.
+Each model process ran serially under `test-supervisor 114688 60 120`.
+
+| Configuration | Actual old lane | Temperature .6, p .95, k 20, seed 0, eight tokens |
+| --- | --- | --- |
+| SSD, pipeline disabled | Classic | 314, 279, 198, 2, 20732, 198, 2, 198 |
+| SSD, pipeline enabled | Classic | Same tokens and every final-state hash |
+| Full resident table, pipeline disabled | Classic | Same tokens and every final-state hash |
+| Full resident table, pipeline enabled | Keyed pipeline | 314, 279, 35067, 82, 13, 220, 19, 15 |
+
+The temperature-1 control also diverged at token 3 only with the resident
+pipeline. Its classic tokens were `314,279,198,2,20732,20086,198,2053`; pipeline
+tokens were `314,279,695,47,4920,82,5212,2502`. Full resident attachment was asserted,
+and the old driver's actual `Decoded::pipeline` was recorded, rather than
+inferring engagement from an environment flag.
+
+`tools/issue104-reference.py` runs the pinned original `generate_ar` and
+SessionBank against the installed artifacts. The checked-in
+`tests/fixtures/issue104-ar-model.json` contains cold and warm token IDs, final
+logits and every trunk-state array's shape/hash. Cold receipts equal both
+pre-fix native classic baselines in full. The warm turn restores all 11 committed
+tokens and adds suffix `[1075]` before decoding eight more tokens.
+
+Native regression `issue104_installed_ar_stream_and_state` checks those complete
+receipts with synchronous and asynchronous evaluation. It also cancels a warm
+turn after three callbacks, checks that the saved state is intact, and retries
+from that state. Every completed turn asserts that PLE staging remains enabled
+and resident preference remains in classic mode. Run with `MTPLX_AR_PIPELINE=1`
+to exercise an explicit obsolete flag without recreating the removed lane.
+
+## Scope review
+
+The checked route is request/session ownership → `Execution::decode_ar` →
+`ar::decode` → classic sampling → final-token capture. No production AR caller
+can reach the lazy primitive. Every former admission and mode call in policy,
+model, decoder, binding and installed reference tests was removed. GDN compilation
+still obeys its two existing environment gates, including explicit-off precedence.
+Normal prefill and MTP retain their compilation/staging paths. DS4/GLM sampling,
+RNG, inference and Metal kernels have no changes.
+
+The 1,120 fresh MTPLX orchestration fixtures cover both obsolete flag values,
+sync/async evaluation, official .6/.95/k20, 1/.95/k20, .5/k2, p=0/1, greedy,
+presence penalty, zero/one/two/three/twelve-token limits, stops and final capture.
+Existing sampler fixtures additionally exercise tied logits, frequency penalties
+and exact PCG64 continuation. The historical primitive-law receipt remains in
+`docs/issue103-analysis/native-sampling-final.log`.
+
+The server-policy oracle's `--server-only` mode compares all existing native
+defaults and verifies pipeline removal across 268 environment/MTP cases.
+Current upstream additionally supplies defaults for other, unported features and
+has replaced its old boundary-count diagnostic API; this scoped check does not
+claim those unrelated policy extensions are implemented.
+
+## Timing interpretation
+
+`receipts.jsonl` records measured decode seconds separately from state equality
+and process memory peaks. These are bounded eight-token correctness probes,
+including first-use and memory-pressure effects, not steady-state throughput
+claims. Old resident classic/pipeline times at .6 were 7.576/17.518 seconds;
+at 1.0 they were 1.332/5.452 seconds. SSD classic repeated at .6 took
+0.649/1.120 seconds and at 1.0 took 1.280/1.273 seconds. No overlap is retained
+to preserve these old timings.
+
+After removal, synchronous SSD decode took 0.642/1.309 seconds at temperatures
+.6/1.0; synchronous resident decode took 0.813/1.427 seconds. Async controls took
+1.182/1.298 seconds (SSD) and 1.435/1.426 seconds (resident). All produce the
+classic stream and state. Peak process physical footprint was 72.5 GB for the
+SSD regression and 104.5 GB for the resident regression.
+
+## Completed validation
+
+- Both installed-model regressions passed with the obsolete flag explicitly on,
+  including warm state, cancellation/retry, sync/async and pre-fix equality.
+- All 1,120 AR orchestration fixtures, 1,080 sampler-entry fixtures, the penalty
+  and distribution/RNG fixtures, the original sampling-law diagnostic, and the
+  connected decoder/cache control passed under the supervisor.
+- The scoped 268-case current server-default comparison passed. The ordinary
+  suite includes DS4's 64 exact token/RNG fixtures and compile-policy controls.
+- `cargo fmt --all -- --check`, warnings-denied Clippy, `make bundle` including
+  signing, and `cargo test --all-features` passed (308 application tests plus
+  two tool suites of four tests each; 211 opt-in application tests ignored).
+- Every artifact SHA256 was re-read and matched the unchanged revision;
+  see `environment.json`. Compact run results are in `checks.json` and
+  `receipts.jsonl`. Source review found no DS4/GLM changes and no remaining
+  production AR-pipeline admission or mode path.
+
+## Reproduction
+
+Build the test executable with `cargo test --all-features --bin ds4-server
+--no-run`; pass that executable, not Cargo, to the supervisor. Set
+`DS4_QWEN38_ARTIFACTS` to the installed artifact directory, and set
+`MTPLX_COMPILED_GDN=1 MTPLX_QWEN4EXP_COMPILE=1`. Run
+`issue104_installed_ar_stream_and_state --ignored --nocapture --test-threads=1`
+once with default SSD policy and once with `MTPLX_NGRAM_RESIDENT=1` and adequate
+RAM. Set `MTPLX_AR_PIPELINE=1` on both runs. Optional `DS4_ISSUE104_BASELINE`
+compares the cold records to a separately saved pre-fix classic receipt.
+
+For the isolated Python oracle, point `PYTHONPATH` at pinned MTPLX, use the
+MLX-0.32.2 environment, the same compilation flags, and
+`MTPLX_SUSTAINED_PREFILL=1 MTPLX_PREFILL_CHUNK_SIZE=2048`. Run
+`tools/issue104-reference.py MODEL_DIRECTORY` under the same supervisor.