101 lines
5.5 KiB
Markdown
101 lines
5.5 KiB
Markdown
# Qwen: MTPLX versus DS4Server GPU responsiveness
|
|
|
|
Measured on the same M5 Max on 2026-09-05. This is a responsiveness diagnosis,
|
|
not acceptance of the goal to match reference throughput within 2%.
|
|
|
|
## Method
|
|
|
|
- Installed Qwen3.8 Flash Next Bare Speed artifacts only; identical model path
|
|
for both implementations. No downloads, offline Hugging Face mode for MTPLX.
|
|
- Context capacity 131072, prefill chunks 2048, reasoning low, temperature 1,
|
|
top-p 0.95, top-k 20, seed 12345. DS4Server power 100%; MTPLX has no imposed
|
|
throttling, fan mode default, turbo profile.
|
|
- The exact raw system prompt and user prompts were exported from DS4Server.
|
|
Warmup is separate; README summary, story and Python generation form one chat.
|
|
Each response runs to EOS, not a fixed-token performance sample.
|
|
- Same native `ds4_gpu_canary_probe`: a 4096-byte Metal blit on an independent
|
|
queue, in the model process, nominally every 100 ms. Native commit-to-scheduled
|
|
and commit-to-completed times exclude Python scheduling delay. The Python
|
|
wrapper separately records host scheduling lag.
|
|
- MTPLX CLI `exec` handoff is intercepted only to invoke its ordinary server
|
|
entry point in the instrumented process; inference code is unchanged.
|
|
- Separate supervisor: 120 seconds without startup/inference progress or a
|
|
physical footprint above 116823110451 bytes terminates the child. Canary
|
|
activity alone does not renew inference progress. No overall runtime timeout.
|
|
- Runs are sequential. Both valid MTPLX AR runs and the MTP run completed all
|
|
three responses normally. All probes reported successful completion.
|
|
|
|
Reference: MTPLX commit `e652d55e2652137a4abcf1312357abbf3eb9d692`, MLX 0.32.2.
|
|
DS4Server: dirty working tree based on `b99ce2a`; no scheduling fix applied for
|
|
these measurements. The exported JSON records the complete effective settings.
|
|
|
|
## Results
|
|
|
|
Maximum native GPU-canary completion latency in milliseconds:
|
|
|
|
| Run | Prefill | Decode | Warmup |
|
|
| --- | ---: | ---: | ---: |
|
|
| DS4Server AR, exported-input control | 4090.027 | 17.903 | see raw JSON |
|
|
| MTPLX AR, first valid run | 7.610 | 0.527 | 0.894 |
|
|
| MTPLX AR, explicit session repeat | 2.974 | 0.713 | 2.854 |
|
|
| MTPLX MTP, explicit session | 0.550 | 0.696 | 1.460 |
|
|
|
|
Thus the multi-second GPU queue delay is reproduced in DS4Server but was **not
|
|
observed in MTPLX**, including a complete MTP conversation. This does not prove
|
|
that MTPLX can never stall or replace measurement of an actual GUI event loop.
|
|
|
|
## Source-level differences and comparison caveats
|
|
|
|
MLX does not put an entire lazy evaluation into one command buffer. Its
|
|
[`CommandEncoder::needs_commit`](https://github.com/ml-explore/mlx/blob/v0.32.2/mlx/backend/metal/device.cpp)
|
|
checks operations and referenced buffer bytes; the Max defaults are 50 operations
|
|
and 50 MiB, with environment overrides. The
|
|
[`eval` path](https://github.com/ml-explore/mlx/blob/v0.32.2/mlx/backend/metal/eval.cpp)
|
|
commits asynchronously when those thresholds are exceeded. These are batching
|
|
thresholds, not a guarantee of a maximum GPU duration.
|
|
|
|
DS4Server Qwen `eval_prefill_batch` currently finishes only after all layers at
|
|
power 100 unless layer profiling is enabled. This differs from MLX's automatic
|
|
submission boundaries. No smaller prefill chunk is needed to investigate this
|
|
specific scheduling difference.
|
|
|
|
The raw inputs are identical, but rendered tokens are not yet identical:
|
|
DS4Server summary has 7490 prompt tokens versus MTPLX's 7460. The Qwen
|
|
`encode_continuation` path calls `encode_qwen_messages` with an empty system
|
|
string, which still emits the low-reasoning system instruction. That inserts
|
|
another system message when extending a checkpoint. Full-conversation rendering
|
|
does not repeat that instruction for every user turn. This must be fixed and
|
|
tested independently before claiming exact input parity.
|
|
|
|
Update after these measurements: the continuation renderer has been corrected
|
|
locally and the focused token-only regression test passes. These historical
|
|
measurements still describe the pre-fix binary; they are not post-fix evidence.
|
|
The complete execution-chain inventory and follow-up work are tracked in the
|
|
[reference-parity audit](/Users/gb/Projects/DS4Server/docs/inference-reference-parity-audit-20260905.md).
|
|
|
|
MTPLX AR reused 5120 cached tokens on every turn, even with an explicit session
|
|
header. MTP reused 5120, 7840 and 8861. Both sent complete ongoing history, but
|
|
their cache reuse is different. Native MTPLX `timings.prompt_per_second` divides
|
|
the whole prompt by prefill time; for fair suffix throughput use
|
|
`(prompt_tokens - cached_tokens) / prefill_seconds`. Neither this number nor raw
|
|
total chat time is currently a matched-work 2% acceptance result.
|
|
|
|
## Local evidence
|
|
|
|
All artifacts are in `/private/tmp/ds4-ongoing-verified.16vzVw/`:
|
|
|
|
- `qwen-export.jsonl`: DS4Server inputs, all outputs, per-turn metrics, canary.
|
|
- `mtplx-ar-v4.jsonl`: first successful MTPLX AR conversation.
|
|
- `mtplx-ar-session.jsonl`: repeated AR with explicit session ID.
|
|
- `mtplx-mtp.jsonl`: MTP with explicit session ID.
|
|
- `mtplx_probe.py`: supervised reference runner; invoke with `ar` or `mtp`
|
|
using the reference checkout's `.venv/bin/python`. Each starts and stops its
|
|
own server on localhost port 18091.
|
|
- `libds4_probe.dylib`: unchanged DS4Server Metal bridge object, linked with
|
|
dead stripping and only `ds4_gpu_canary_probe` exported. No DS4 C inference
|
|
implementation is linked or invoked.
|
|
|
|
Earlier `mtplx-ar.jsonl`, `mtplx-ar-v2.jsonl`, and `mtplx-ar-v3.jsonl` are failed
|
|
instrumentation attempts (linking, CLI exec losing threads, Python module
|
|
identity respectively), **not benchmark evidence**.
|