DS4Server/docs/qwen-reference-responsiveness-20260905.md

Qwen: MTPLX versus DS4Server GPU responsiveness

Measured on the same M5 Max on 2026-09-05. This is a responsiveness diagnosis, not acceptance of the goal to match reference throughput within 2%.

Method

• Installed Qwen3.8 Flash Next Bare Speed artifacts only; identical model path for both implementations. No downloads, offline Hugging Face mode for MTPLX.
• Context capacity 131072, prefill chunks 2048, reasoning low, temperature 1, top-p 0.95, top-k 20, seed 12345. DS4Server power 100%; MTPLX has no imposed throttling, fan mode default, turbo profile.
• The exact raw system prompt and user prompts were exported from DS4Server. Warmup is separate; README summary, story and Python generation form one chat. Each response runs to EOS, not a fixed-token performance sample.
• Same native ds4_gpu_canary_probe: a 4096-byte Metal blit on an independent queue, in the model process, nominally every 100 ms. Native commit-to-scheduled and commit-to-completed times exclude Python scheduling delay. The Python wrapper separately records host scheduling lag.
• MTPLX CLI exec handoff is intercepted only to invoke its ordinary server entry point in the instrumented process; inference code is unchanged.
• Separate supervisor: 120 seconds without startup/inference progress or a physical footprint above 116823110451 bytes terminates the child. Canary activity alone does not renew inference progress. No overall runtime timeout.
• Runs are sequential. Both valid MTPLX AR runs and the MTP run completed all three responses normally. All probes reported successful completion.

Reference: MTPLX commit e652d55e2652137a4abcf1312357abbf3eb9d692, MLX 0.32.2. DS4Server: dirty working tree based on b99ce2a; no scheduling fix applied for these measurements. The exported JSON records the complete effective settings.

Results

Maximum native GPU-canary completion latency in milliseconds:

Run	Prefill	Decode	Warmup
DS4Server AR, exported-input control	4090.027	17.903	see raw JSON
MTPLX AR, first valid run	7.610	0.527	0.894
MTPLX AR, explicit session repeat	2.974	0.713	2.854
MTPLX MTP, explicit session	0.550	0.696	1.460

Thus the multi-second GPU queue delay is reproduced in DS4Server but was not observed in MTPLX, including a complete MTP conversation. This does not prove that MTPLX can never stall or replace measurement of an actual GUI event loop.

Source-level differences and comparison caveats

MLX does not put an entire lazy evaluation into one command buffer. Its CommandEncoder::needs_commit checks operations and referenced buffer bytes; the Max defaults are 50 operations and 50 MiB, with environment overrides. The eval path commits asynchronously when those thresholds are exceeded. These are batching thresholds, not a guarantee of a maximum GPU duration.

DS4Server Qwen eval_prefill_batch currently finishes only after all layers at power 100 unless layer profiling is enabled. This differs from MLX's automatic submission boundaries. No smaller prefill chunk is needed to investigate this specific scheduling difference.

The raw inputs are identical, but rendered tokens are not yet identical: DS4Server summary has 7490 prompt tokens versus MTPLX's 7460. The Qwen encode_continuation path calls encode_qwen_messages with an empty system string, which still emits the low-reasoning system instruction. That inserts another system message when extending a checkpoint. Full-conversation rendering does not repeat that instruction for every user turn. This must be fixed and tested independently before claiming exact input parity.

Update after these measurements: the continuation renderer has been corrected locally and the focused token-only regression test passes. These historical measurements still describe the pre-fix binary; they are not post-fix evidence. The complete execution-chain inventory and follow-up work are tracked in the reference-parity audit.

MTPLX AR reused 5120 cached tokens on every turn, even with an explicit session header. MTP reused 5120, 7840 and 8861. Both sent complete ongoing history, but their cache reuse is different. Native MTPLX timings.prompt_per_second divides the whole prompt by prefill time; for fair suffix throughput use (prompt_tokens - cached_tokens) / prefill_seconds. Neither this number nor raw total chat time is currently a matched-work 2% acceptance result.

Local evidence

All artifacts are in /private/tmp/ds4-ongoing-verified.16vzVw/:

• qwen-export.jsonl: DS4Server inputs, all outputs, per-turn metrics, canary.
• mtplx-ar-v4.jsonl: first successful MTPLX AR conversation.
• mtplx-ar-session.jsonl: repeated AR with explicit session ID.
• mtplx-mtp.jsonl: MTP with explicit session ID.
• mtplx_probe.py: supervised reference runner; invoke with ar or mtp using the reference checkout's .venv/bin/python. Each starts and stops its own server on localhost port 18091.
• libds4_probe.dylib: unchanged DS4Server Metal bridge object, linked with dead stripping and only ds4_gpu_canary_probe exported. No DS4 C inference implementation is linked or invoked.

Earlier mtplx-ar.jsonl, mtplx-ar-v2.jsonl, and mtplx-ar-v3.jsonl are failed instrumentation attempts (linking, CLI exec losing threads, Python module identity respectively), not benchmark evidence.