Save inference parity implementation and evaluation harness

README.md

@@ -94,6 +94,90 @@ pie-natural`, `--case composed-basics`, `--attempts 3`, `--model MODEL_ID`, or
 `--endpoint URL` to narrow or repeat runs. This slow, model-dependent suite is
 intentionally separate from `cargo test`.
 
+### Headless model evaluation
+
+Run a bounded, instrumented generation through the production runtime without
+opening the UI:
+
+```sh
+cargo run --release -- model-eval \
+  --model qwen3.8-flash-next \
+  --reasoning low \
+  --acceleration off \
+  --power-percent 100
+```
+
+The evaluator first performs a short unmeasured warmup through the same system-prompt
+builder and production runtime as the UI, then resets its GPU counters and starts
+the measured request. `warmup_result.elapsed_ms` includes initial model loading;
+its `timing` object separates `model_load_ms` and `non_loading_elapsed_ms` (the
+latter still includes setup, inference, and checkpoint work). Do not add the
+reported load time to the warmup total. A `model_loaded` event reports the same
+load duration as the shared runtime metrics. Canary samples retain their start
+`phase` and record `completion_phase`, actual completion `elapsed_ms`, and host
+observation `observed_elapsed_ms`; per-phase summaries group by probe start.
+`result.timing.decode_ms` measures the shared engine decode loop, including
+sampling and streamed output callbacks but excluding subsequent pending-token
+finalization and checkpoint work. Stats shows the same measurement as "Decode
+loop (without checkpoint)". The older host-observed interval through request
+completion is retained as `decode_to_request_end_ms`; it is not pure decode time.
+`decode_loop_tokens_per_second` uses completion tokens divided by the loop duration;
+`decode_tokens_per_second` remains the UI's last progress rate. `checkpoint_io`
+reports request operation counts and the last read/write duration, not summed I/O
+time. A missing decode duration is `null`, not zero.
+The default test
+prompt identifies a prose-generation test and asks the model to choose
+its own genre, setting, and characters, then output only a complete short story
+without questions, choices, limitations, or coding discussion. JSON Lines report
+runtime phases, prefill and decode speed, Metal busy time,
+command-buffer duration, CPU, physical memory, disk I/O, and the complete reasoning
+and answer. `--input-file PATH` appends a UTF-8 file to the supplied prompt for
+realistic long-prefill tests. Repeat `--prompt` to run multiple turns in one
+ongoing chat: every later turn includes all preceding user prompts and complete
+assistant responses and continues from the same session checkpoint. An independent
+Metal canary runs inside the same
+process as the production runtime through
+loading, prefill, decode, and finishing, reporting per-phase scheduling and
+completion latency plus stalls that could make a UI unresponsive. Use
+`--acceleration on|off` to compare DSpark on DeepSeek or integrated MTP on
+GLM/Qwen. The canary itself submits GPU work, so use `--canary off` for clean
+throughput measurements and `--canary on` for responsiveness diagnostics. The UI
+canary is off by default and can be enabled in Stats. A supervisor terminates
+the worker only when startup or token progress stalls, or when the memory limit
+is exceeded; there is no whole-run timeout. See `model-eval --help` for all
+overrides. The evaluator only opens already installed model artifacts and never
+downloads missing models.
+
+Evaluation defaults to `--plain-chat on`: the shared UI prompt builder omits
+agent/tool schemas, skill instructions and the built-in coding-agent system
+prompt; an explicitly customized system prompt is preserved. The corresponding
+UI configuration is `plain_chat: true` in its YAML configuration. In that mode
+the UI also skips workspace/date injections, extension hooks, A2UI handling,
+tool execution, automatic response retries and automatic first-turn titles.
+Use a **new chat** in both frontends: existing history is never silently removed.
+The normal UI default remains agent mode (`plain_chat: false`). `--plain-chat off`
+restores agent prompt construction for diagnostics, but does not make the harness
+an agent/tool executor. The effective mode and complete system prompt are reported
+in the start receipt; do not compare runs with different modes.
+
+For a summary, story, and Python one-shot in a single chat, pass three prompts:
+
+```sh
+target/release/ds4-server model-eval \
+  --model qwen3.8-flash-next --reasoning low \
+  --power-percent 100 --acceleration off --canary on \
+  --input-file README.md \
+  --prompt 'Give a concise summary of the following project README. Return the summary directly without asking questions.' \
+  --prompt 'This is a prose-generation test. Respond only with a complete short fictional story in ordinary prose. Choose the genre, setting, and characters yourself and begin the story immediately. Do not ask questions, offer choices, explain limitations, or discuss coding.' \
+  --prompt 'This is an offline code-generation benchmark. Do not invoke tools or emit tool-call markup. Return exactly one Python code block defining has_close_elements(numbers: list[float], threshold: float) -> bool. It must return True if any two distinct numbers are closer than threshold. Include no explanation outside the code block.'
+```
+
+The input file is appended only to the first prompt. Each turn starts after the
+previous response finishes; `measurement_start` records the turn number and the
+number of history messages (1, 3, 5), and each `result` reports its own timings,
+cached tokens, reasoning, and answer. The warmup uses a separate session and is
+not included in this history.
+
 ## Build and run
 
 DS4Server targets Apple silicon with macOS 15 or newer and requires Rust 1.97 plus