Save inference parity implementation and evaluation harness

docs/qwen-reference-responsiveness-20260905.md

@@ -0,0 +1,100 @@
+# Qwen: MTPLX versus DS4Server GPU responsiveness
+
+Measured on the same M5 Max on 2026-09-05. This is a responsiveness diagnosis,
+not acceptance of the goal to match reference throughput within 2%.
+
+## Method
+
+- Installed Qwen3.8 Flash Next Bare Speed artifacts only; identical model path
+  for both implementations. No downloads, offline Hugging Face mode for MTPLX.
+- Context capacity 131072, prefill chunks 2048, reasoning low, temperature 1,
+  top-p 0.95, top-k 20, seed 12345. DS4Server power 100%; MTPLX has no imposed
+  throttling, fan mode default, turbo profile.
+- The exact raw system prompt and user prompts were exported from DS4Server.
+  Warmup is separate; README summary, story and Python generation form one chat.
+  Each response runs to EOS, not a fixed-token performance sample.
+- Same native `ds4_gpu_canary_probe`: a 4096-byte Metal blit on an independent
+  queue, in the model process, nominally every 100 ms. Native commit-to-scheduled
+  and commit-to-completed times exclude Python scheduling delay. The Python
+  wrapper separately records host scheduling lag.
+- MTPLX CLI `exec` handoff is intercepted only to invoke its ordinary server
+  entry point in the instrumented process; inference code is unchanged.
+- Separate supervisor: 120 seconds without startup/inference progress or a
+  physical footprint above 116823110451 bytes terminates the child. Canary
+  activity alone does not renew inference progress. No overall runtime timeout.
+- Runs are sequential. Both valid MTPLX AR runs and the MTP run completed all
+  three responses normally. All probes reported successful completion.
+
+Reference: MTPLX commit `e652d55e2652137a4abcf1312357abbf3eb9d692`, MLX 0.32.2.
+DS4Server: dirty working tree based on `b99ce2a`; no scheduling fix applied for
+these measurements. The exported JSON records the complete effective settings.
+
+## Results
+
+Maximum native GPU-canary completion latency in milliseconds:
+
+| Run | Prefill | Decode | Warmup |
+| --- | ---: | ---: | ---: |
+| DS4Server AR, exported-input control | 4090.027 | 17.903 | see raw JSON |
+| MTPLX AR, first valid run | 7.610 | 0.527 | 0.894 |
+| MTPLX AR, explicit session repeat | 2.974 | 0.713 | 2.854 |
+| MTPLX MTP, explicit session | 0.550 | 0.696 | 1.460 |
+
+Thus the multi-second GPU queue delay is reproduced in DS4Server but was **not
+observed in MTPLX**, including a complete MTP conversation. This does not prove
+that MTPLX can never stall or replace measurement of an actual GUI event loop.
+
+## Source-level differences and comparison caveats
+
+MLX does not put an entire lazy evaluation into one command buffer. Its
+[`CommandEncoder::needs_commit`](https://github.com/ml-explore/mlx/blob/v0.32.2/mlx/backend/metal/device.cpp)
+checks operations and referenced buffer bytes; the Max defaults are 50 operations
+and 50 MiB, with environment overrides. The
+[`eval` path](https://github.com/ml-explore/mlx/blob/v0.32.2/mlx/backend/metal/eval.cpp)
+commits asynchronously when those thresholds are exceeded. These are batching
+thresholds, not a guarantee of a maximum GPU duration.
+
+DS4Server Qwen `eval_prefill_batch` currently finishes only after all layers at
+power 100 unless layer profiling is enabled. This differs from MLX's automatic
+submission boundaries. No smaller prefill chunk is needed to investigate this
+specific scheduling difference.
+
+The raw inputs are identical, but rendered tokens are not yet identical:
+DS4Server summary has 7490 prompt tokens versus MTPLX's 7460. The Qwen
+`encode_continuation` path calls `encode_qwen_messages` with an empty system
+string, which still emits the low-reasoning system instruction. That inserts
+another system message when extending a checkpoint. Full-conversation rendering
+does not repeat that instruction for every user turn. This must be fixed and
+tested independently before claiming exact input parity.
+
+Update after these measurements: the continuation renderer has been corrected
+locally and the focused token-only regression test passes. These historical
+measurements still describe the pre-fix binary; they are not post-fix evidence.
+The complete execution-chain inventory and follow-up work are tracked in the
+[reference-parity audit](/Users/gb/Projects/DS4Server/docs/inference-reference-parity-audit-20260905.md).
+
+MTPLX AR reused 5120 cached tokens on every turn, even with an explicit session
+header. MTP reused 5120, 7840 and 8861. Both sent complete ongoing history, but
+their cache reuse is different. Native MTPLX `timings.prompt_per_second` divides
+the whole prompt by prefill time; for fair suffix throughput use
+`(prompt_tokens - cached_tokens) / prefill_seconds`. Neither this number nor raw
+total chat time is currently a matched-work 2% acceptance result.
+
+## Local evidence
+
+All artifacts are in `/private/tmp/ds4-ongoing-verified.16vzVw/`:
+
+- `qwen-export.jsonl`: DS4Server inputs, all outputs, per-turn metrics, canary.
+- `mtplx-ar-v4.jsonl`: first successful MTPLX AR conversation.
+- `mtplx-ar-session.jsonl`: repeated AR with explicit session ID.
+- `mtplx-mtp.jsonl`: MTP with explicit session ID.
+- `mtplx_probe.py`: supervised reference runner; invoke with `ar` or `mtp`
+  using the reference checkout's `.venv/bin/python`. Each starts and stops its
+  own server on localhost port 18091.
+- `libds4_probe.dylib`: unchanged DS4Server Metal bridge object, linked with
+  dead stripping and only `ds4_gpu_canary_probe` exported. No DS4 C inference
+  implementation is linked or invoked.
+
+Earlier `mtplx-ar.jsonl`, `mtplx-ar-v2.jsonl`, and `mtplx-ar-v3.jsonl` are failed
+instrumentation attempts (linking, CLI exec losing threads, Python module
+identity respectively), **not benchmark evidence**.