Align DeepSeek and GLM execution with DS4
This commit is contained in:
Georg Bauer
@@ -0,0 +1,519 @@
|
||||
# DeepSeek standalone-reference follow-up — 2026-09-11
|
||||
|
||||
The original antirez/ds4 process is used only under the user's explicit
|
||||
reference-benchmark authorization. No original C engine is linked into DS4Server.
|
||||
Evidence: `local-eval-results/deepseek-paired-20260911.nyUCsL/`.
|
||||
|
||||
**Measurement qualification (user clarification, 2026-09-11):** The user was
|
||||
watching videos in parallel, using part of the GPU. The exact overlap with each
|
||||
earlier run is not recorded. The current sequential comparisons are therefore
|
||||
not controlled idle-device performance acceptance or causal before/after proof.
|
||||
Their outputs and functional checks remain useful; timing receipts are retained,
|
||||
not discarded. Do not attribute every gap to video playback or claim thermal
|
||||
throttling from this information. Further throughput acceptance runs are deferred
|
||||
until competing GPU activity can be controlled; code work and functional tests
|
||||
can continue. This does not waive the2% requirement.
|
||||
|
||||
## Initial pair is not performance-parity evidence
|
||||
|
||||
Both AR processes finished the full Summary README → lighthouse Story → Python
|
||||
`is_prime` conversation after a separate32-token OK warmup. Installed0731 GGUF
|
||||
only, power100, Low, temperature0.6/top-p0.95/min-p0/top-k0/seed42, context32768,
|
||||
quality/SSD/canary/DSpark off. Startup30s/progress45s,112GiB process-memory guards,
|
||||
no total timeout. No builds or other model tests overlapped the GPU runs.
|
||||
|
||||
Two mismatches invalidate this initial pair (`ar-{reference,rust}.*`):
|
||||
|
||||
- The standalone driver requested2048 prefill rows, whereas the UI/harness
|
||||
requested automatic (0), which the DeepSeek executor resolves to4096 at this
|
||||
context. Original DS4's automatic setting also resolves to4096. The reference
|
||||
driver now requests0 and records the public session prefill-cap value instead
|
||||
of silently assuming the same chunk for all models. GLM's automatic request
|
||||
remains unchanged. No product chunk reduction or special harness path.
|
||||
- The same first prompt contains2741 tokens in the original but2742 in the
|
||||
production runtime. The direct Rust tokenizer and all three original
|
||||
continuation fixtures agree exactly; the mismatch occurs later, in the shared
|
||||
cached-prompt renderer. A valid system-bootstrap tag with empty chat history
|
||||
took the continuation branch and prepended an EOS before the first user turn.
|
||||
Both UI and headless generation use this function.
|
||||
|
||||
## Root-cause correction and regression
|
||||
|
||||
`render_text_prompt` now requires nonempty history for the cached continuation
|
||||
branch. A first user turn uses canonical full rendering, while `align_prompt`
|
||||
still reuses its prepared system prefix. Real ongoing histories retain the
|
||||
existing append-only behavior. No artificial wait, power change, kernel change
|
||||
or KV-persistence rewrite was introduced.
|
||||
|
||||
The CPU-only `ds4_chat_matches_original_session_tokens` regression loads only
|
||||
GGUF metadata/tokenizer and original recorded token IDs. It first checks the
|
||||
three-turn tokenizer contract, then invokes the same shared renderer with the
|
||||
actual system-bootstrap frontier and tag. Before the fix it fails with:
|
||||
|
||||
```text
|
||||
shared bootstrap renderer differs: 2742/2741 tokens;
|
||||
starts [0, 1, 128803, 45585]/[0, 128803, 45585, 260]
|
||||
```
|
||||
|
||||
`token-before.*` is the passing standalone-tokenizer diagnostic;
|
||||
`bootstrap-before.*` is the deliberately failing shared-runtime regression.
|
||||
Neither file is overwritten or treated as a successful full parity run.
|
||||
The test also accepts the saved GLM fixture to cover the other shared consumer.
|
||||
|
||||
Reproduction after building release tests:
|
||||
|
||||
```sh
|
||||
DS4SERVER_CHAT_REFERENCE=local-eval-results/deepseek-paired-20260911.nyUCsL/chat-token-fixture.json \
|
||||
target/release/test-supervisor 2048 30 30 \
|
||||
target/release/deps/ds4_server-6141856e5c0fa6df \
|
||||
ds4_chat_matches_original_session_tokens
|
||||
```
|
||||
|
||||
The fixed renderer passes both saved DeepSeek and GLM three-turn fixtures
|
||||
(`{chat,glm-chat}-token-fixture-fixed.*`,0.04/0.05s). Release all-target/all-feature
|
||||
build, warnings-denied Clippy, rustfmt/diff checks,17 enabled sampling tests and
|
||||
two enabled tokenizer tests pass. The model-dependent tests remain explicitly
|
||||
ignored by default and were invoked with the recorded local fixtures above.
|
||||
CLI SHA256: `9fb13c7f9049c3b5059b4fe79b68bf997dbe4e1bfdc4c631c130595cab8c359a`.
|
||||
Standalone driver SHA256:
|
||||
`ad7a4b94a5918505c58130d336d026e2c5b9640aab717ef185808bebd7536a25`.
|
||||
|
||||
## Corrected complete AR pair
|
||||
|
||||
`fixed-ar-{reference,rust}.*` and `fixed-ar-comparison.json` contain the new
|
||||
original-then-Rust pair. The original session confirms prefill-cap4096 and
|
||||
engine power100. Every text/thinking/token/cache comparison passes; all turns
|
||||
end naturally. Counts are826/1444/199, prompts2741/3587/5057 and cached1/3567/5031.
|
||||
|
||||
| Turn | Rust / original engine-prefill ms | Rust / original decode t/s |
|
||||
| --- | ---: | ---: |
|
||||
| Summary | 6561.316 / 5425.137 | 36.916 / 37.813 |
|
||||
| Story | 286.826 / 290.380 | 32.712 / 34.246 |
|
||||
| Python | 334.486 / 369.018 | 32.020 / 32.734 |
|
||||
|
||||
This is functional agreement for the complete AR workload, **not** performance
|
||||
acceptance: all three decode ratios are below98% in this single pair, and the
|
||||
Summary prefill is slower. Different natural outputs in the invalid initial
|
||||
pair must not be used as before/after throughput evidence. No GUI/canary run was
|
||||
performed in this clean series.
|
||||
|
||||
## DSpark pair: functional comparison fails
|
||||
|
||||
Both full processes finished naturally and both watchdogs exited successfully.
|
||||
The final comparison exits1 deliberately (`dspark-comparison.json`); this is
|
||||
not an inference crash. Rust confirms opportunistic sampling, confidence0.6,
|
||||
strict/exact off; the original loads the installed three-stage/block5 support
|
||||
model and enables direct verifier commits.
|
||||
|
||||
The first prompt/cache counts agree (2741/1), but its output already diverges:
|
||||
647 Rust tokens versus1208 original tokens. Thus later speed ratios compare
|
||||
different histories and **cannot establish matched-work performance parity**.
|
||||
Counts in Story/Python are1675/195 versus1455/347. The raw rates are retained
|
||||
in the comparison file, not promoted to valid performance results.
|
||||
|
||||
An independent frontier defect is visible even within Rust's own history:
|
||||
after2741 prompt +647 output tokens, the next cache is3389 instead of3388;
|
||||
after3409 prompt +1675 output tokens, it is5085 instead of5084. A speculatively
|
||||
returned EOS remains committed. The shared consumer currently rewinds only
|
||||
GLM, while original `ds4_session_rewind` also truncates the DeepSeek logical
|
||||
frontier and invalidates DSpark capture. This is separate from the now-fixed
|
||||
first-prompt bootstrap EOS and from the first-answer decode divergence.
|
||||
|
||||
### Next cohesive DSpark work unit
|
||||
|
||||
1. Locate the first divergent cycle with identical recorded target history:
|
||||
compare proposal IDs/confidence, GPU row argmax, accepted prefixes and
|
||||
compression/DSpark capture state against original DS4. Cover full acceptance,
|
||||
partial acceptance and rejection before declaring the full implementation
|
||||
equivalent. Existing target-owned self-tests alone are not an oracle proof.
|
||||
2. Match the consumer's EOS frontier and capture invalidation using DS4's
|
||||
DeepSeek contract, including an accepted-EOS transition into the next user
|
||||
turn. Do not transplant GLM's KDA rollback or merely hide the extra count.
|
||||
3. Remove verified extra verifier work as part of that same comparison:
|
||||
Rust `eval_batch_inner(collect_tops)` reads/clones every row's full vocabulary
|
||||
and computes argmax on the CPU. Original opportunistic verification passes
|
||||
`row_logits=NULL`, obtains GPU row tops, then reads only the committed row
|
||||
(`ds4.c:69112–69124,69153–69159,69210–69216`). Preserve the separate exact-
|
||||
sampling requirements and verify row/stop decisions before timing the change.
|
||||
Also verify per-turn counter scope across warmup/resident switching before
|
||||
using cycle/acceptance ratios as exclusive stage measurements.
|
||||
|
||||
The four reviewed AR/DSpark Python answers (both implementations) each pass
|
||||
their five generated assertions and5011 independent cases in isolated Python
|
||||
with restricted builtins (`python-quality.jsonl`). No model-generated tools
|
||||
were invoked. This quality check does not waive DSpark's token/state mismatch.
|
||||
|
||||
The six-cell performance goal, repeated timing acceptance and actual UI
|
||||
responsiveness acceptance remain open. The user's working bundle is unchanged.
|
||||
All model/reference/canary processes from this follow-up have terminated.
|
||||
|
||||
## DSpark cycle/state audit (second work unit)
|
||||
|
||||
Evidence: `local-eval-results/ds4-cycle-audit-20260911.vfsUc8/`. The original
|
||||
source and installed artifacts remain the same. `DS4_SPEC_CYCLE_TRACE=1` is an
|
||||
opt-in diagnostic in the shared production consumer and standalone reference
|
||||
driver; it records the sampled first token, returned IDs, generated count and
|
||||
committed position. It is off by default. Original `DS4_DSPARK_SPEC_LOG=1`
|
||||
adds draft lengths, full/partial acceptance and scheduler decisions without
|
||||
changing the graph. These traced runs are **not clean throughput acceptance**.
|
||||
|
||||
The initial complete traced pair reproduces the failure. In the first Summary
|
||||
cycle, both targets sample671; Rust proposes12275 and emits only671, while DS4
|
||||
proposes3967 and emits `[671,3967]`. The original warmup first differs in the
|
||||
fifth returned cycle, after earlier partial accepts. This is not evidence of
|
||||
a context-size or power-setting cause.
|
||||
|
||||
### Consolidated findings and implementation tasks
|
||||
|
||||
This table scopes the inspected path to the installed0731 DSpark support,
|
||||
resident M5 Max execution and default opportunistic sampling. It does not claim
|
||||
completion for arbitrary SSD/strict/exact modes or the six-cell performance goal.
|
||||
Original line references below are for pinned `ds4.c` at `ec7642c`.
|
||||
|
||||
| Area | Original behavior / Rust discrepancy | Current action |
|
||||
| --- | --- | --- |
|
||||
| Initial support KV | `32627–32821`: target HC expansion, per-stage HC mixing and attention normalization precede KV projection. Rust projected stage-0 normalized hidden directly to KV. | Implemented the missing operations using existing Metal calls and batch scratch. First Summary proposal now matches. |
|
||||
| Single accepted draft | `36205` and `69112`: even one draft uses the target batch verifier. Rust used scalar decode, with different arithmetic and no captured suffix for the next support cache. | Removed the resident single-draft scalar branch. First two Summary cycles then match. |
|
||||
| Verifier capture | `27951–28053`: capture contains the preceding seed row plus all verified target rows. Rust retained only the suffix. | Added seed-row capture with the original slot stride and sufficient workspace. |
|
||||
| Cache timing and gaps | `66665–66732`: seed from the captured batch only when its end equals the next proposal position; otherwise crop/reset the absolute cache window. Ring maintenance does not create or bridge a window (`33329`, `27712`). Rust eagerly seeded each batch, extended windows after failed proposals and tracked only modulo positions. | Added deferred seed ownership and absolute window start; no false append after rejected proposals. Replaced batch workspaces are released before allocation of the next batch. |
|
||||
| Partial acceptance | `69221–69240`: ordinary partial commits invalidate row and batch captures, preserve compressor-prefix state, and do not claim support KV rows. Rust retained the captures and extended support counters; its old seed scratch also overwrote captured hidden rows. | Invalidate the deferred capture, preserve the original cache frontier, and stop overwriting hidden capture storage. Three and then32 recorded Summary cycles pass. |
|
||||
| Scheduler | `53500`, `65215`: reset per request. `53666`: a window pause replaces, rather than maximizes with, the cold no-draft pause. | Reset in shared prompt alignment; regression covers7→4 and clearing prior-request counters. |
|
||||
| EOS consumer | `73690`: logical DeepSeek rewind plus capture invalidation, not GLM KDA rollback. | Shared consumer now invokes the model-specific DeepSeek rewind; invalid forward rewind is rejected before mutation. Full next-turn verification still required. |
|
||||
| Q8 confidence arithmetic | `34350` calls `matvec_any`, which dispatches type8 to activation-quantized Q8 matvec (`8144–8170`). The installed confidence tensor is Q8_0 `[4352,1]`; Rust instead used the dequantized-weight × F32-input reference calculation. | Quantize activations for the confidence dot too, and match the two alternating four-lane ARM SDOT/FMA accumulators (`7510`) in both confidence and Markov scoring. CPU regression distinguishes the quantized result from the old F32-input dot. |
|
||||
| Confidence/output head | `66780–66835`: check first confidence before doing the vocabulary projection; read later rows only as required. Rust calculated all base logits and read all rows before the check. | Implemented the early gate and per-needed-row readback. Confidence-disabled execution keeps hidden/head submission fused. Full1755-cycle comparison remains exact; clean timing acceptance is separate. |
|
||||
| Verifier output | `36325–36431`, `69153`, `69210`: fuse head into the layer command sequence, obtain GPU row tops, read only the committed logits row. Rust drained before the head and cloned all vocabulary rows for CPU argmax. | Implemented fused submission, existing GPU argmax/top-k reductions and committed-row readback for the ordinary verifier. Exact sampling retains all target distributions. Full1755-cycle default-mode comparison remains exact. |
|
||||
| CPU Markov workers | `33889–33963`: persistent helper pool and fused Q8 argmax; `1912` defaults to min(online CPUs,12), including the caller. Rust created scoped OS threads on every dense argmax, using all18 logical CPUs here. | Implemented persistent Rust workers with the existing Q8 arithmetic, identical contiguous row partition and ordered first-tie reduction. The caller executes slot0; default12 total threads and positive integer `DS4_THREADS` overrides up to32. All1755 original cycles remain exact; timing acceptance remains separate. |
|
||||
| Verifier allocations | Original graph retains verifier scratch and GPU frontier/prefix buffers; Rust constructed `BatchScratch` and snapshot buffers for every verifier. | Reuse baseline and high-water prefix buffers, plus one verifier batch per existing padded row shape. Scratch is recycled only after delayed seeding or capture invalidation. Extended32-cycle oracle regression proves native-buffer identity reuse and byte-exact rollback; full-chat and timing evidence below. |
|
||||
| Acceptance telemetry | Rust's executor-wide cycle counts and resident support counters have different ownership scopes. | **Open:** reconcile request/session counters before deriving acceptance ratios or exclusive stage percentages. |
|
||||
|
||||
Excluded after checking the guards: seed-batch fusion defaults to the ROCm
|
||||
gfx1151 path, not this Mac (`53412`); the Markov GPU branch is under
|
||||
`#ifndef __APPLE__` (`34266`); adaptive extra decode splits are pre-M5-only.
|
||||
The output-head padding to8 rows is present in DS4's helper (`26086`) and is
|
||||
not an unmatched Rust optimization. None of these were blindly enabled/removed.
|
||||
|
||||
### Runnable regression evidence
|
||||
|
||||
`dspark_matches_original_summary_cycles` loads only the installed GGUFs and
|
||||
the recorded original JSONL, prepares the actual bootstrap boundary, samples
|
||||
with the shared DS4 sampler and verifies each cycle's IDs and position. It is
|
||||
ignored by default and explicitly supervised when run:
|
||||
|
||||
```sh
|
||||
DS4SERVER_DSPARK_REFERENCE=local-eval-results/ds4-cycle-audit-20260911.vfsUc8/before-reference.jsonl \
|
||||
target/release/test-supervisor 114688 30 45 \
|
||||
target/release/deps/ds4_server-6141856e5c0fa6df \
|
||||
dspark_matches_original_summary_cycles
|
||||
```
|
||||
|
||||
- `cycle-test-before.*`: fails at generated2, Rust `[10059]` versus original
|
||||
`[10059,260,13672,294,270]`.
|
||||
- `cycle-test-batch.*`: after single-row batch verification, fails at generated7,
|
||||
Rust `[4496,3051,943,30941,22]` versus original `[4496,3051,943,30941]`.
|
||||
- `cycle-test-capture.*`: all three unchanged expected cycles pass (7.28s).
|
||||
- `cycle-test-32.*`:32 original cycles, prior-request scheduler reset and
|
||||
safe logical rewind pass (9.32s). These are functional diagnostics, not
|
||||
stories truncated for a throughput comparison.
|
||||
|
||||
The original full pair, intermediate cache-only full run and failing receipts
|
||||
are retained. No original C implementation was added to the application; no
|
||||
model download, bundle replacement, commit or push was performed.
|
||||
|
||||
### Confidence follow-up from the full chat
|
||||
|
||||
The subsequent `fixed-rust.*` full run matches451 complete returned cycles
|
||||
(including warmup), then differs at Summary generated1018. The first divergent
|
||||
decision is one cycle earlier: at position3759, original DS4 proposes `[588,699]`
|
||||
and rejects the first draft, while Rust's confidence logit0.40281284 falls below
|
||||
the0.6 sigmoid threshold and suppresses drafting entirely. The different
|
||||
no-draft scheduler decision then changes the returned cycles. All turns still
|
||||
end naturally, with counts1194/1424/296 and **no extra cached EOS**; this is
|
||||
progress, not a matched-output performance pair (`fixed-cycle-comparison.json`).
|
||||
|
||||
Read-only inspection of the installed support GGUF confirms the confidence
|
||||
head type8/Q8_0,4352 inputs. Correcting the activation quantization yields
|
||||
confidence0.4063788 at that exact position and the original two draft IDs.
|
||||
The stable Rust SDOT intrinsic is unavailable in this toolchain; the two SDOT
|
||||
instructions are isolated in guarded Rust inline assembly, with stdlib NEON
|
||||
FMA/reduction and the existing scalar fallback. No C host code or dependency
|
||||
was added. Six focused DSpark CPU/layout tests pass, including the new
|
||||
quantized-confidence test. Release all-target/all-feature build, Clippy with
|
||||
warnings denied, rustfmt and diff checks pass after the correction.
|
||||
|
||||
Checkpoint-load invalidation now clears the new deferred capture and absolute
|
||||
cache metadata through the same `reset_cache` helper as session reset. This
|
||||
does not change the checkpoint format or disk KV policy.
|
||||
|
||||
### Completed functional comparison, performance still open
|
||||
|
||||
`quantized-rust.*` completes all three turns to natural EOS. All1755 returned
|
||||
cycles match the original recording exactly, including warmup, IDs and committed
|
||||
positions (`quantized-cycle-comparison.json`). All three text, thinking,
|
||||
completion, prompt and cached-token comparisons pass
|
||||
(`quantized-result-comparison.json`):
|
||||
|
||||
| Turn | Completion tokens | Prompt / cached | Rust / original prefill ms | Rust / original decode t/s | Decode gap |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Summary | 1208 | 2741 /1 | 5562.979 /5519.685 | 36.128 /36.580 | −1.24% |
|
||||
| Story | 1455 | 3969 /3949 | 261.651 /281.080 | 30.366 /31.321 | −3.05% |
|
||||
| Python | 347 | 5450 /5424 | 323.474 /344.868 | 33.936 /34.777 | −2.42% |
|
||||
|
||||
These are **diagnostic** timings with cycle logging, not repeat-median clean
|
||||
performance acceptance. Story and Python still miss even the single-pair2%
|
||||
threshold; the missing early-confidence gate, verifier submission/readback,
|
||||
allocation and worker-lifetime tasks above remain explicit work. No UI canary
|
||||
or real GUI event-loop acceptance is claimed for this series.
|
||||
|
||||
The Story has a coherent title, narrative and ending, not a clarification or
|
||||
broken output. The Python answer is byte-identical to the previous original
|
||||
DSpark answer (`python-reference-unchanged.json`), whose five generated asserts
|
||||
and5011 independent cases passed in `deepseek-paired-20260911.nyUCsL/python-quality.jsonl`.
|
||||
Six DSpark CPU/layout tests,17 shared sampling tests, the separate heap-fallback
|
||||
test and two tokenizer tests pass; model-dependent tests are not silently
|
||||
counted as run. The explicit32-cycle live regression passed before the final
|
||||
confidence correction, and the final full1755-cycle harness/oracle comparison
|
||||
validates the production path after it.
|
||||
|
||||
Final CLI SHA256:
|
||||
`2d15458d12a14457519a22232061ce76182123b24f23d1ae595df424c36963e5`.
|
||||
Traced standalone reference SHA256:
|
||||
`395cc58f8ea59afa0a6e22f8d46b3fb3f129bc367b8ee3ee1a0dd4bc0f01bd4d`.
|
||||
The working app bundle remains
|
||||
`ea4d555c2faf0940d9cbcf76d8638ca614a9cb2c6b034e3b2f80aeef86b0b339`.
|
||||
|
||||
### Early confidence and lazy verifier output
|
||||
|
||||
The shared Rust executor now follows the original early confidence gate before
|
||||
the draft vocabulary projection. Hidden and logits rows are read only when the
|
||||
proposal loop needs them; disabling confidence keeps hidden/head encoding fused.
|
||||
The ordinary verifier keeps its layer commands alive through the output head
|
||||
and the existing GPU argmax/top-k reduction. It reads only the committed
|
||||
distribution; exact stochastic sampling retains all row distributions.
|
||||
No prefill-cap, power, artificial pause, kernel math or KV persistence change.
|
||||
|
||||
`lazy-head-rust.*`, `lazy-head-cycle-comparison.json` and
|
||||
`lazy-head-result-comparison.json` establish the unchanged complete1755-cycle
|
||||
recording and all three answers/thinking/token/cache/stop boundaries. Seven
|
||||
focused CPU/layout tests pass (two installed-model tests remain ignored by
|
||||
default), including the committed-row boundary regression. Release all-target,
|
||||
all-feature build and warnings-denied Clippy pass. This full default-mode
|
||||
comparison is not a new exact-stochastic oracle claim.
|
||||
|
||||
CLI SHA256: `de10b537c2cf2bfbb680ebc350e810305def3ca2ba59f0bea89f1d8c980699cb`.
|
||||
The original driver and working app bundle hashes above are unchanged.
|
||||
`run-clean-pairs.sh` records serial three-turn timing pairs without cycle logging,
|
||||
original proposal logging or canary, with alternating reference-first/Rust-first
|
||||
order. Both workers retain startup/progress/memory supervision, no total timeout.
|
||||
The first postprocessing command used the wrong reference event name (`result`
|
||||
instead of `reference_result`) and failed after both workers had finished; the
|
||||
comparison was corrected against the unchanged receipts, with no inference rerun.
|
||||
|
||||
All six processes completed successfully, with natural EOS for all nine Rust
|
||||
answers and matching reference content, thinking, prompt, cache and completion
|
||||
counts (1208/1455/347 in every pair). `clean-{1,2,3}-comparison.json` and
|
||||
`clean-pairs-summary.json` retain all results; none were discarded:
|
||||
|
||||
| Pair / order | Turn | Rust / DS4 prefill ms | Rust / DS4 decode t/s | Decode delta |
|
||||
| --- | --- | ---: | ---: | ---: |
|
||||
| 1 DS4→Rust | Summary | 6396.150 /5435.701 | 34.878 /37.298 | −6.49% |
|
||||
| 1 | Story | 262.284 /289.008 | 28.623 /31.716 | −9.75% |
|
||||
| 1 | Python | 350.909 /332.242 | 30.372 /35.511 | −14.47% |
|
||||
| 2 Rust→DS4 | Summary | 7905.861 /7910.498 | 29.783 /28.784 | +3.47% |
|
||||
| 2 | Story | 307.278 /376.104 | 25.918 /24.574 | +5.47% |
|
||||
| 2 | Python | 363.522 /488.866 | 29.186 /27.959 | +4.39% |
|
||||
| 3 DS4→Rust | Summary | 8389.686 /8292.276 | 27.523 /27.801 | −1.00% |
|
||||
| 3 | Story | 301.627 /383.962 | 24.399 /24.151 | +1.02% |
|
||||
| 3 | Python | 379.914 /458.934 | 27.995 /27.801 | +0.70% |
|
||||
|
||||
Pair3 is within2% in decode, with Summary prefill1.17% slower and the two short
|
||||
continuation prefills faster. **It is not sufficient acceptance for this series:**
|
||||
DS4 itself slows from37.30 to27.80 Summary t/s (about25.5%), and pair order changes
|
||||
the sign of the relative gap. Median aggregation cannot establish a causal code
|
||||
speedup in this nonstationary series. Do not compare the clean first pair against
|
||||
the earlier logged pair as a before/after regression claim. Remaining worker and
|
||||
allocation discrepancies still need their reference-aligned implementation and
|
||||
an appropriately controlled follow-up.
|
||||
|
||||
Read-only system checks during pair2 reported no recorded thermal/performance
|
||||
warning and25% system memory free; our benchmark processes ran strictly serially.
|
||||
These observations do not prove stable GPU clocks or exclude other system GPU
|
||||
activity. No application was stopped or configuration changed in response.
|
||||
Rust supervisor totals were107.691/121.168/129.178s, maximum observed progress
|
||||
gaps6.396/7.905/8.389s in prefill, supervisor lag60/59/59ms and exit0 throughout.
|
||||
Canary was off: these are not GPU-canary or actual GUI event-loop measurements.
|
||||
Original supervisor totals were96.236/124.665/127.935s, all with `error:null`.
|
||||
|
||||
The existing ignored `flash_0731_runs_exact_sampled_dspark` initially failed its
|
||||
draft-count assertion: its four-token allowance takes the scheduler's existing
|
||||
`max_tokens <10` no-draft branch. The test now allows16 tokens and explicitly
|
||||
disables the confidence gate to exercise drafting independently of this short
|
||||
fixture. It retains the original assertions and additionally invokes two-row
|
||||
verification, checking both complete finite distributions, the GPU top ID
|
||||
against CPU argmax, and the retained last-row logits. The supervised rerun
|
||||
passes in1.60s (`lazy-head-exact-fixed.*`); the initial failure is retained in
|
||||
`lazy-head-exact-test.*`. This is a regression check, not a new exact-sampling
|
||||
reference/performance claim. No production setting was changed for this test.
|
||||
Final warnings-denied Clippy, rustfmt and diff checks pass after this test-only
|
||||
edit. No model or benchmark process remains from this work unit.
|
||||
|
||||
### Persistent CPU Markov workers
|
||||
|
||||
`src/engine/metal/markov.rs` replaces per-draft scoped thread creation with a
|
||||
model-owned Rust worker pool. Original source contracts are `ds4.c:1912–1947`
|
||||
(default min(online,12), caller plus helpers), `1971–2005` (contiguous partitions,
|
||||
serial execution below512 rows), and `33859–33963` (ordered first-tie Q8 argmax).
|
||||
Positive integer `DS4_THREADS` settings are supported up to the original32-thread
|
||||
limit. No CUDA/non-Apple Markov branch is enabled.
|
||||
|
||||
The GGUF mapping is shared through `Arc<Mmap>` without remapping or copying the
|
||||
weights. Worker inputs own their temporary data: the full logits `Vec` is moved
|
||||
into shared read-only ownership and returned after all dispatched jobs finish.
|
||||
There are no borrowed raw pointers between worker lifetimes. Workers release
|
||||
their input before signalling completion; results are drained on failure too,
|
||||
and pool destruction closes and joins every helper. Existing dot/quantization
|
||||
functions are reused without arithmetic changes. No dependency or native host
|
||||
code was added.
|
||||
|
||||
`markov-rust.*` completes the full warmup and ongoing three-turn chat. All1755
|
||||
original cycles and all answer/thinking/token/cache/EOS checks remain exact
|
||||
(`markov-cycle-comparison.json`, `markov-result-comparison.json`). Two focused
|
||||
Markov tests and seven GGUF tests pass, including thread and logits-buffer
|
||||
reuse, ordered ties, shape validation and mapping bounds. Release all-target/
|
||||
all-feature build and warnings-denied Clippy pass. The full diagnostic run is
|
||||
not throughput acceptance. CLI SHA256:
|
||||
`0e24c6e8102c2035cc89e93f77eede4235f5aa4611c36ce5fb225770641485f2`.
|
||||
|
||||
The remaining allocation task is concrete: `BatchScratch::allocate` reserves
|
||||
workspace based on both row count and context position; `snapshot_spec_frontier`
|
||||
allocates compressor/indexer/target snapshots every verifier cycle. Reuse must
|
||||
preserve delayed support seeding, partial acceptance, error rollback and padded
|
||||
output-head behavior. Merely retaining an arbitrary previous batch is not a
|
||||
safe implementation of the original persistent scratch contract.
|
||||
|
||||
Two clean full pairs (`clean-markov-{1,2}-*`) ran DS4→Rust→Rust→DS4, power100,
|
||||
same installed files/settings/warmup/ongoing chat, trace and canary off. All
|
||||
content/thinking/prompt/cache/token checks pass; both workers in both pairs exit
|
||||
successfully, without downloads or overlapping model processes:
|
||||
|
||||
| Pair | Turn | Rust / DS4 prefill ms | Rust / DS4 decode t/s | Decode delta |
|
||||
| --- | --- | ---: | ---: | ---: |
|
||||
| 1 | Summary | 6686.486 /5445.567 | 32.046 /36.968 | −13.32% |
|
||||
| 1 | Story | 296.457 /280.371 | 26.201 /31.162 | −15.92% |
|
||||
| 1 | Python | 376.741 /362.034 | 28.251 /34.122 | −17.21% |
|
||||
| 2 | Summary | 8407.889 /8535.688 | 26.901 /27.488 | −2.14% |
|
||||
| 2 | Story | 332.174 /383.535 | 23.301 /24.322 | −4.20% |
|
||||
| 2 | Python | 398.755 /466.153 | 26.975 /28.215 | −4.40% |
|
||||
|
||||
The second pair still misses decode parity. DS4 Summary throughput again falls
|
||||
by about25.6% over the series; no systemwide throttling cause is asserted. A
|
||||
single `sudo -n powermetrics` query failed immediately because a password was
|
||||
required (`markov-power-sample.txt`); it started no sampler. Rust supervisor
|
||||
totals117.095/132.871s versus DS4 totals98.054/127.837s include different
|
||||
frontend/finalization work and are not pure decode. Rust maximum progress gaps
|
||||
were6.686/8.407s and supervisor lag59/58ms. No canary/UI-responsiveness claim.
|
||||
|
||||
To separate dispatch cost from GPU drift, the ignored CPU-only
|
||||
`installed_markov_worker_dispatch` test uses installed Markov W1 row671 and the
|
||||
actual W2 mapping with a fixed logits row. It compares per-call18-thread spawning
|
||||
against persistent12 and18 workers, holding row arithmetic/input ownership
|
||||
constant. Four alternating orders,128 calls per mode per round, all1536 argmax
|
||||
results identical. Median times per128 calls:
|
||||
|
||||
| Dispatch | Median ms | ms/call |
|
||||
| --- | ---: | ---: |
|
||||
| Scoped18 | 40.356 | 0.3153 |
|
||||
| Persistent12 (production default) | 36.077 | 0.2819 |
|
||||
| Persistent18 (diagnostic only) | 30.496 | 0.2382 |
|
||||
|
||||
The default pool reduces this isolated dispatch/calculation time by10.6%, about
|
||||
0.034ms per call; it does not explain seconds of whole-chat difference. This
|
||||
diagnostic is neither a C Markov microbenchmark nor end-to-end parity evidence.
|
||||
Production retains the reference's default12-thread policy. The same runnable
|
||||
worker regression also closes one helper and checks that all other jobs drain,
|
||||
an error is returned and the caller's original logits buffer is preserved.
|
||||
Receipts: `markov-dispatch-test.*`, `markov-worker-failure-tests.txt`.
|
||||
|
||||
Final verification after the additional test-only coverage: seven enabled
|
||||
DSpark tests pass; the explicitly supervised exact-sampling test passes in1.35s;
|
||||
warnings-denied Clippy, rustfmt and diff checks pass. All processes from this
|
||||
work unit have finished. No app bundle replacement, commit or push was made.
|
||||
The full six-cell performance goal remains open.
|
||||
|
||||
### Persistent verifier buffers: functional checks pass, timing remains unaccepted
|
||||
|
||||
The next implementation reuses baseline compressor/indexer snapshots, high-water
|
||||
prefix snapshots and one verifier workspace per existing padded row shape.
|
||||
Deferred seed ownership determines when a batch can be recycled; full-prefill
|
||||
workspaces are not retained. Position-sensitive verifier storage reserves the
|
||||
session context. Existing output-head row padding, sampling, power and command
|
||||
submission boundaries are unchanged. Error rollback keeps its previous behavior;
|
||||
an error may drop scratch and require allocation on the next attempt.
|
||||
|
||||
Evidence remains in `local-eval-results/ds4-cycle-audit-20260911.vfsUc8/`.
|
||||
CLI SHA256: `4f3643bd854cad1d317fdd6bb93649327902b19cb3cd7b9cfff24a760d57e880`.
|
||||
The prior CLI is retained as `before-verifier-reuse-ds4-server` for a later
|
||||
controlled comparison; merely comparing older sequential runs is insufficient.
|
||||
|
||||
- `reuse-cycle-test.txt`: the32-cycle original oracle passes, now also checking
|
||||
native buffer identities for both padded verifier shapes and byte-exact
|
||||
compressor/indexer rollback after another target step.
|
||||
- `reuse-cycle-comparison.json` and `reuse-result-comparison.json`: the complete
|
||||
traced chat matches all1755 original cycles and all text/thinking/token/cache
|
||||
checks. Traced timing is diagnostic only.
|
||||
- `reuse-exact-test.txt`: supervised exact-sampling regression passes after the
|
||||
reuse changes, including complete target-distribution readback.
|
||||
- `reuse-build.txt`, `reuse-clippy.txt`, `reuse-dspark-tests.txt`: release build,
|
||||
warnings-denied Clippy and seven enabled DSpark tests pass.
|
||||
- GPU allocation at each measured turn end is95,039,750,144bytes, approximately
|
||||
73MiB more retained than the prior Summary endpoint. This is retained scratch,
|
||||
not proof of a throughput improvement.
|
||||
|
||||
Two complete pairs ran in DS4→Rust→Rust→DS4 order, with trace/canary off and no
|
||||
overlapping model processes or builds. The user subsequently confirmed concurrent
|
||||
video playback; "clean" in these artifact filenames means instrumentation off,
|
||||
**not** an idle GPU. Both pairs preserve all outputs and exit successfully.
|
||||
|
||||
| Pair | Turn | Rust / DS4 prefill ms | Rust / DS4 decode t/s | Decode delta |
|
||||
| --- | --- | ---: | ---: | ---: |
|
||||
| 1 | Summary | 6679.941 /5718.646 | 34.204 /38.273 | −10.63% |
|
||||
| 1 | Story | 259.070 /273.257 | 28.306 /33.200 | −14.74% |
|
||||
| 1 | Python | 355.567 /333.715 | 29.343 /36.744 | −20.14% |
|
||||
| 2 | Summary | 8968.416 /10283.116 | 26.678 /24.830 | +7.44% |
|
||||
| 2 | Story | 324.111 /445.256 | 21.926 /22.225 | −1.35% |
|
||||
| 2 | Python | 365.097 /560.790 | 23.092 /26.814 | −13.88% |
|
||||
|
||||
Receipts: `clean-reuse-{1,2}-{comparison.json,rust.jsonl,reference.jsonl}` and
|
||||
matching stderr. Rust supervisor totals110.153/140.256s versus original
|
||||
93.671/141.037s include frontend/finalization differences, not just decode.
|
||||
Rust progress gaps6.679/8.968s and supervisor lag45/45ms are not GPU-canary or
|
||||
GUI-eventloop delays. No new responsiveness acceptance, app bundle replacement,
|
||||
commit or push. Remaining work includes telemetry ownership reconciliation,
|
||||
reference code/cost audit and the controlled full six-cell comparison.
|
||||
|
||||
### Interim checkpoint requested by the user
|
||||
|
||||
The current work unit is being closed for commit/push, not declared full parity.
|
||||
The commit-gate test run exposed an already-committed stale Qwen source-inventory
|
||||
assertion:22 runtime units were expected although both HEAD's Metal export and
|
||||
the pinned generator contain26. The read-only command below verifies the entire
|
||||
export byte for byte against pinned sources (17 custom bodies,26 runtime units,
|
||||
dynamic QSA sources). Only the stale expected count is corrected; no kernel,
|
||||
hash, fixture or per-body assertion is changed or removed.
|
||||
|
||||
```sh
|
||||
python3 tools/mtplx-kernel-source.py local-eval-results/mtplx-reference-e652d55 --gated-delta-source local-eval-results/mtplx-reference-env-0.32.2/lib/python3.12/site-packages/mlx_lm/models/gated_delta.py --check
|
||||
```
|
||||
|
||||
The app bundle is rebuilt as required by the commit gates. This supersedes the
|
||||
earlier statements that this follow-up had not yet replaced the bundle; no GUI
|
||||
or new throughput series is launched. Unrelated `tools/__pycache__/` files are
|
||||
left untouched and excluded from the commit.
|
||||
|
||||
Commit verification: `cargo fmt --all -- --check`, warnings-denied all-target/
|
||||
all-feature Clippy, `make bundle`, and `cargo test --all-features` pass. The full
|
||||
suite reports303 main tests plus4 supervisor and4 integration tests passed,
|
||||
zero failures,202 explicitly ignored main tests. The separately supervised
|
||||
32-cycle and exact-sampling results above remain distinct from these normal
|
||||
gates. `codesign --verify --deep --strict` and the bundle's `model-eval --help`
|
||||
also pass without opening the GUI. Final suite receipt:
|
||||
`local-eval-results/ds4-cycle-audit-20260911.vfsUc8/interim-commit-tests.txt`.
|
||||
@@ -0,0 +1,585 @@
|
||||
# GLM execution and responsiveness follow-up — 2026-09-11
|
||||
|
||||
## User acceptance and scope
|
||||
|
||||
- Qwen is confirmed good in normal interactive use.
|
||||
- GLM decode is now also confirmed good interactively. GLM prefill remains
|
||||
usable, but feels less smooth than Qwen and affects other applications.
|
||||
This is not a claim of a complete freeze or a new confirmed beachball.
|
||||
- The user authorized resuming the outstanding work and explicitly authorized
|
||||
compiling/running antirez/ds4 as a standalone, supervised reference benchmark.
|
||||
No DS4 C objects are linked into DS4Server or its application bundle.
|
||||
- Qwen's golden master remains MTPLX; DeepSeek/GLM remain antirez/ds4.
|
||||
The accepted 2.6% Qwen Summary AR exception is not a general tolerance.
|
||||
|
||||
## Implemented execution changes
|
||||
|
||||
Both scalar GLM loops now flush periodically every four completed layers,
|
||||
excluding the final layer and SSD expert streaming. Previously they flushed
|
||||
only once at layer four. This follows the active indexed DS4 graph, including
|
||||
scalar MTP fallback/rejection replay. The reference's dynamic per-layer mapping
|
||||
fallback must not be confused with Rust's static non-expert decode mapping:
|
||||
`glm_streaming_model_spans` retains non-expert tensors, the configured resident
|
||||
expert prefix, and incompatible expert layouts; selected experts are loaded
|
||||
through the existing native cache. No new per-layer SSD waits were introduced.
|
||||
Low-memory dynamic mapping fallback parity is not established by this patch.
|
||||
|
||||
GLM 5.3 prefill progress now advances at existing completed GPU drains and after
|
||||
the final output evaluation, not after every submitted layer. Chunk selection,
|
||||
prefill flush/drain placement, Metal kernels, sampling and power policy are
|
||||
unchanged. This corrects progress accounting; it does not by itself fix the
|
||||
remaining prefill smoothness issue.
|
||||
|
||||
Targeted checks passed: periodic/final/SSD decode boundary test; existing
|
||||
prefill boundary test; live two-row verifier acceptance, rejection, rewind,
|
||||
scalar fallback, recurrent state, unused HC workspace guards and lifetime
|
||||
counters. The live test additionally verifies completed-prefill progress points.
|
||||
|
||||
## Standalone reference and instrumentation
|
||||
|
||||
`tools/ds4-session-reference.rs` is a separate Rust benchmark driver for the
|
||||
unchanged public DS4 engine/session interface. It is deliberately not a Cargo
|
||||
target and is never included in the app. Its build script verifies reference
|
||||
commit `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`, the pinned header hash,
|
||||
unchanged tracked engine sources and current reference objects. The arm64 ABI
|
||||
layout is checked against Clang's record layout (engine options 280 bytes,
|
||||
distributed offset 152, TP offset 216).
|
||||
|
||||
The public DS4 CLI/API maps `low` to `high`. The driver instead constructs the
|
||||
Low system prefix through the public chat API and passes tokens to the original
|
||||
session implementation. It also reproduces the UI's separate system-prefix
|
||||
prefill (9 tokens for GLM), and retains generated token history in one session.
|
||||
`glm53_reference_prompt_tokens_match_shared_runtime` checks every token of all
|
||||
four prompt streams, including continued turns, against the production Rust
|
||||
tokenizer. The first complete matched-bootstrap AR reference passed this check.
|
||||
|
||||
Build from the DS4Server checkout, with already-built reference objects:
|
||||
|
||||
```sh
|
||||
bash tools/build-ds4-session-reference.sh /Users/gb/Projects/ds4 /absolute/path/reference
|
||||
```
|
||||
|
||||
Run the resulting binary from the reference checkout under `test-supervisor`:
|
||||
|
||||
```text
|
||||
test-supervisor 114688 30 45 --command /absolute/path/reference MODEL_GGUF glm on README_PATH
|
||||
```
|
||||
|
||||
The driver uses power100, Low, context32768, temperature0.6, top-p0.95,
|
||||
top-k0, min-p0, seed42, SSD streaming off and graph-selected GLM chunks.
|
||||
Warmup is a separate session (up to32 tokens), followed by README Summary,
|
||||
lighthouse Story, and Python `is_prime` in one ongoing chat, to natural EOS.
|
||||
There is no total-runtime watchdog. Missing models fail; no downloads occur.
|
||||
DeepSeek is supported by the driver's `deepseek` family argument, with the
|
||||
installed DSpark support GGUF required for acceleration-on; it has not yet been
|
||||
validated by this follow-up's GLM runs.
|
||||
|
||||
Optional `DS4_REFERENCE_CANARY=/absolute/path/ds4-server` starts the **same native
|
||||
probe and monitor implementation** as the UI/harness in a separate process.
|
||||
The `gpu-canary` CLI accepts phase labels on stdin and ends on EOF. A readiness
|
||||
handshake waits for the first successful probe before model loading. Its
|
||||
readiness/stall/sample reports are not model-progress watchdog heartbeats.
|
||||
Clean throughput runs leave this variable unset. An external observer and an
|
||||
in-process observer must not be treated as identical OS scheduling conditions.
|
||||
|
||||
`DS4_REFERENCE_IN_PROCESS_CANARY=1` instead enables a native probe thread inside
|
||||
the reference process. It links the same `native/metal/ds4_canary.m` used by
|
||||
DS4Server, with the same4096-byte blit, separate queue and100ms cadence, rather
|
||||
than duplicating a Metal implementation. This bridge contains no model code.
|
||||
Its Rust monitor logs per-sample phases/timing but is not the UI event loop.
|
||||
The optional external observer can also be enabled simultaneously. Neither is
|
||||
enabled for clean reference throughput. `reference --canary-self-test` exercises
|
||||
readiness, two phases and clean shutdown without loading a model.
|
||||
|
||||
The initial external-probe integration test exposed a startup race: a phase
|
||||
could end before the executable initialized. It was not worked around by
|
||||
loosening the assertion; the adapter now requires a readiness handshake. The
|
||||
initial diagnostic without that handshake is retained, not a full-startup proof.
|
||||
|
||||
## Measurements and limitations
|
||||
|
||||
All raw receipts, outputs and failed attempts are retained under
|
||||
`local-eval-results/glm-scheduling-20260911.7VSaeW/` (ignored, local evidence).
|
||||
The before binary is the secured `02db096` implementation, SHA256
|
||||
`ab289f067a7a01c22113eec76aa896638d83e392242192e1440d14ed11524d5c`.
|
||||
|
||||
Fresh complete AR runs (after, then before), same output/reasoning/tokens/EOS:
|
||||
|
||||
| Turn | Tokens | Before decode t/s | After decode t/s | Before prefill ms | After prefill ms |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Summary |640|24.467|25.403|7561|6034|
|
||||
| Story |1102|23.703|24.135|262|255|
|
||||
| Python |200|24.726|25.401|305|302|
|
||||
|
||||
These are single sequential pairs, not drift-controlled medians. The large
|
||||
summary-prefill difference cannot be attributed to a decode-only flush change.
|
||||
No hard decode regression was observed; full performance parity is not proven.
|
||||
|
||||
The matched-bootstrap original DS4 AR reference completed naturally at
|
||||
26.618/25.123/25.825 decode t/s, with779/1047/198 output tokens. Its generated
|
||||
text differs from Rust despite matching initial prompt tokens/settings; later
|
||||
contexts therefore also differ. This is not an exact-output performance pair.
|
||||
Reference prefill timers measure session sync; Rust's current GLM `prefill_ms`
|
||||
still includes the observed UI phase. Do not silently equate those intervals.
|
||||
|
||||
The new Rust MTP run retains the previous625/1026/196 completion tokens and
|
||||
identical output/reasoning. Its draft acceptance fractions are241/385 (62.6%),
|
||||
342/685 (49.9%) and95/102 (93.1%). Python therefore has the expected higher
|
||||
acceptance; MTP's benefit is workload-dependent, not uniformly absent.
|
||||
|
||||
The fresh MTP before/after pair also preserves every output/reasoning token
|
||||
and natural completion:
|
||||
|
||||
| Turn | Tokens | Before decode t/s | After decode t/s | Before prefill ms | After prefill ms |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||
| Summary |625|19.988|23.154|9303|6190|
|
||||
| Story |1026|17.728|19.295|306|272|
|
||||
| Python |196|27.843|29.506|347|322|
|
||||
|
||||
The same sequential-run/drift limitation applies. This establishes no observed
|
||||
hard regression, not a controlled causal speedup or reference-parity acceptance.
|
||||
|
||||
### Canary placement and timestamp attribution
|
||||
|
||||
The full `after-mtp-dual-canary` run had simultaneous internal/external probes.
|
||||
The internal probe recorded867 successful samples, with a prefill maximum
|
||||
of490.740ms and decode maximum3.180ms. The external probe was ready before model
|
||||
launch and continued until after termination:3004 successful samples, overall
|
||||
maximum3.331ms (startup), and1.572ms while labelled `preparing` across the model
|
||||
lifetime. That external label is deliberately not turn/phase attribution.
|
||||
Both probes stopped cleanly; no sample failed or reached2s. This is diagnostic
|
||||
evidence, not a clean throughput run or a compositor-frame test.
|
||||
|
||||
Thus the previous `completed_ms` cannot be interpreted as a measured systemwide
|
||||
GPU blockade. It includes host-side waiting and completion delivery. The optional
|
||||
shared native probe now also records commit-to-GPU-start (`gpu_wait_ms`),
|
||||
GPU-start-to-end (`gpu_interval_ms`), and GPU-end-to-host-return (`host_return_ms`).
|
||||
Metal's GPU timestamps use system mach time; the probe uses `mach_absolute_time`
|
||||
and the native timebase for those differences, not `CLOCK_MONOTONIC`. Missing or
|
||||
inconsistent timestamps remain null, not zero. The GPU interval includes possible
|
||||
GPU scheduling/preemption, not exclusively active blit execution. See Apple's
|
||||
[GPUStartTime documentation](https://developer.apple.com/documentation/metal/mtlcommandbuffer/gpustarttime).
|
||||
|
||||
The model-free Metal integration check verifies phase coverage, valid nonnegative
|
||||
intervals and their bounds against wall completion; the synthetic unit check
|
||||
retains null timing when unavailable. The existing UI uses the same enhanced
|
||||
native probe, but its stats panel still displays the existing wall latency fields.
|
||||
Neither model work nor disabled-canary execution invokes the new timestamp work.
|
||||
|
||||
The first full Rust timestamp run (`after-mtp-timeline`) preserved every MTP
|
||||
output/reasoning token and EOS. All902 samples had valid Metal timestamps and
|
||||
none failed. Its worst prefill sample was300.003ms:299.892ms before GPU start,
|
||||
0.001917ms GPU interval, and0.105958ms after GPU end. The decode maximum was
|
||||
3.733ms. This directly rules out delayed host return as the dominant cause of
|
||||
that prefill sample; the queued probe waits for GPU execution. It does not show
|
||||
that a different application's rendering queue is delayed by the same amount.
|
||||
|
||||
The subsequent extraction into the shared native object changes no probe work:
|
||||
direct `[cb commit]` replaces the wrapper whose model-queue-only hook never
|
||||
applied to this separate canary queue. Both native bindings pass their model-free
|
||||
checks after extraction. Final current product binary SHA256:
|
||||
`50b4b4abbbdc45ff600c1f46d0bec611879249ac8e4d8291d22d656b9c6e9a5d`;
|
||||
standalone reference binary:
|
||||
`ccc7a8a774cb1c202add6dba60b04dffe3597822b15a34e22c7e4a5574b50adf`;
|
||||
shared probe source:
|
||||
`dd3abc34088ee27ba0759f01a291b9b714114420295252d63e85fd6f326fddab`.
|
||||
|
||||
Answer correctness is checked separately from natural termination. Rust AR,
|
||||
Rust MTP and reference AR passed their generated assertions plus5011 `is_prime`
|
||||
cases (-10 through5000). The preliminary reference MTP output passed its own
|
||||
five assertions but failed347 additional cases, first at49: it omits the
|
||||
`i + 2` divisor test. This is a failed generated Python answer, not by itself
|
||||
evidence of an engine defect. It must not be reported as a successful code
|
||||
benchmark merely because EOS was reached. Details are in `python-check.json`.
|
||||
|
||||
### Reference clean MTP and record integrity
|
||||
|
||||
The clean `reference-mtp-clean` run (both canaries disabled) completed all turns
|
||||
with the same593/872/167 tokens, text, stop tokens and failed Python answer as
|
||||
the diagnostic reference run. Its prefill times were5462.306/351.803/434.414ms;
|
||||
decode21.483/16.754/26.143t/s. The preceding in-process diagnostic measured
|
||||
23.623/19.287/30.440t/s. This spread must not be disguised as a port speedup or
|
||||
accepted2% parity: it is one sequential comparison with different probe state,
|
||||
not controlled repeated clean medians. Canary-on throughput is not the baseline.
|
||||
|
||||
The first internal reference run reported788 successful probes, but only787
|
||||
were independently parseable: a watchdog resource record interrupted one
|
||||
canary JSON record at a pipe-read boundary. That failed record is preserved in
|
||||
`reference-mtp-inline.stderr.log`, not silently counted as missing/zero latency.
|
||||
The supervisor now forwards complete lines in one locked stream write (with a
|
||||
64KiB cap for newline-free output), while watchdog progress still consumes every
|
||||
incoming chunk immediately. EOF flushes partial output. A split-record regression
|
||||
test and all existing memory/start/continuation/long-run watchdog tests pass.
|
||||
The reference diagnostic is repeated as `reference-mtp-inline-records` for a
|
||||
fully parseable receipt; the earlier run is retained as the failure evidence.
|
||||
|
||||
That repeated reference run completed with **771/771 parseable, successful,
|
||||
fully timestamped samples** and identical593/872/167 generated tokens/text/EOS.
|
||||
Prefill p95/max was233.106/264.292ms (48 samples); decode p95/max was
|
||||
0.226/19.338ms (713 samples). The worst prefill probe waited264.195ms before
|
||||
GPU start, ran over0.001750ms, and returned to the host0.092083ms after GPU end.
|
||||
No probe reached2s. The reference's prefill samples also include its short
|
||||
warmup; the worst sample occurred during the measured summary prefill.
|
||||
Its diagnostic throughput was23.893/19.177/32.090t/s, not the clean baseline.
|
||||
|
||||
| Matched native in-process probe | Prefill p95 ms | Prefill max ms | Decode max ms |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| DS4Server, `after-mtp-timeline` |289.978|300.003|3.733|
|
||||
| Original DS4, `reference-mtp-inline-records` |233.106|264.292|19.338|
|
||||
|
||||
These sequential diagnostics reproduce the same GPU-start-wait phenomenon in
|
||||
the golden master. They do not excuse the remaining Rust prefill cost, establish
|
||||
statistical latency equivalence, or measure another application's compositor.
|
||||
Moving inference to another thread cannot by itself reproduce the independent
|
||||
process's scheduling conditions; process isolation is a distinct architectural
|
||||
option, not implemented or declared proven as a UI fix here.
|
||||
|
||||
Verification at this checkpoint: release all-target/all-feature build; release
|
||||
all-target/all-feature Clippy with warnings denied; rustfmt and diff checks;
|
||||
11 model-eval unit tests; both model-free native probe bindings; all4 supervisor
|
||||
tests; earlier live GLM verifier/progress/HC guards and the full AR/MTP chats.
|
||||
The updated supervisor fixes measurement transport, not inference scheduling.
|
||||
|
||||
## Sampling versus model execution — continued investigation
|
||||
|
||||
The prior follow-up made concrete progress (execution fixes plus a fair native
|
||||
in-process latency reference), but did not establish the full three-model,
|
||||
AR/speculative2% goal. This continuation addresses the different GLM outputs
|
||||
before treating their different ongoing histories as matched performance work.
|
||||
|
||||
`reference --sampler-fixture` runs the original public `ds4_sample_logits` without
|
||||
loading a model or using Metal. The checked-in
|
||||
`tests/fixtures/ds4-sampling-ec7642c.json` contains64 cases: four vocabulary sizes,
|
||||
eight temperature/top-k/top-p/min-p settings, seeds0/42,32 consecutive tokens
|
||||
per case and the final RNG state. The original Rust test failed40 of64 cases.
|
||||
The shared DS4/GLM sampler now preserves the first argmax tie and original
|
||||
negative sentinel, skips RNG consumption for greedy/all-invalid and the DS4
|
||||
full-vocabulary min-p fallback, and preserves seed0 until the original RNG's
|
||||
zero-state substitution. Qwen's independent MTPLX sampler is untouched.
|
||||
All64 oracle cases and the16 enabled sampling tests pass. Crucially, the positive
|
||||
temperature/top-p benchmark cases at seed42 already passed before the fix:
|
||||
these edge corrections are not the explanation for the observed GLM chat gap.
|
||||
|
||||
Optional `DS4_REFERENCE_LOGITS_TRACE` records the first32 summary logit rows
|
||||
through the public original session API. It requires AR mode, creates a new
|
||||
file rather than overwriting one, and does not change generated tokens or RNG.
|
||||
The full `reference-ar-logits` chat retained exactly the779/1047/198 tokens,
|
||||
text and stop tokens of `reference-ar-bootstrap`. Its timings are diagnostic,
|
||||
not a clean performance baseline. The binary trace contains19,824,640 bytes
|
||||
(32 rows of154,880 little-endian floats). Its path is serialized as an OsString
|
||||
and decoded losslessly by the replay test.
|
||||
|
||||
`glm53_reference_logits_replay_separates_sampling_from_execution` first samples
|
||||
those original C-produced rows through the production Rust sampler: **all32
|
||||
tokens match**. It then opens the installed GLM at Power100/context32768,
|
||||
prefills the same9-token bootstrap and exact summary suffix, and advances only
|
||||
with reference-selected tokens. Thus histories never diverge during comparison.
|
||||
On the Rust-generated rows the test **fails at step17**, choosing906 instead of
|
||||
the reference320. Already the first post-prefill row has max absolute difference
|
||||
5.722162 and RMS difference0.851167. All32 per-step row errors are retained in
|
||||
`logits-replay.stderr.log`; the watched test terminates normally with failure
|
||||
status in9s. This is a new, deliberately retained red parity test, not a passed
|
||||
live validation or a speed result. No DS4/GLM/Metal/CPU diagnostic override was
|
||||
present in the parent environment.
|
||||
|
||||
The next localization belongs in the model execution path: compare existing
|
||||
original DS4 per-layer tensor dumps with the corresponding Rust HC/KDA/DSA/FFN
|
||||
stages, starting at the first bootstrap/prefill block. Do not explain this away
|
||||
as stochastic output variation or hide it with a lower chunk/power setting.
|
||||
No speculative numerical tolerance or new scheduling workaround was applied.
|
||||
|
||||
## Root cause: GLM 5.2 chunk boundary applied to GLM 5.3
|
||||
|
||||
The active original indexed GLM 5.3 path deliberately keeps full2048-token
|
||||
chunks across both the old2048 indexer threshold and the4096/8192 dense-attention
|
||||
threshold. Rust was still applying the GLM 5.2 top-k boundary: after the9-token
|
||||
bootstrap it evaluated2039 tokens, whereas DS4 evaluated2048. This changes the
|
||||
recurrent prefill computation, despite identical total prompt tokens.
|
||||
|
||||
The original layer0 bootstrap `attn_out` and `ffn_out` dumps matched Rust
|
||||
bit-for-bit. The original position9 dumps contain2048*4096 floats, establishing
|
||||
the actual chunk geometry rather than inferring it from configuration.
|
||||
Detailed HC dump hooks elsewhere in DS4 belong to an inactive dense path and
|
||||
were not used as evidence for the active indexed execution.
|
||||
|
||||
Rust now retains complete GLM 5.3 chunks and splits only the attention slices
|
||||
at the dense/sparse boundary, as DS4 does. This also removes the incorrect
|
||||
whole-pair sparse override for a two-row verifier crossing that boundary.
|
||||
GLM 5.2 retains its old top-k splitting. Unit checks cover both families and
|
||||
the4096/8192 attention transitions. No smaller chunk, delay, or power reduction
|
||||
was introduced.
|
||||
|
||||
After this correction, the same fixed-history replay is green: **all32 full
|
||||
154880-value logit rows are bit-identical** to the original trace (max absolute
|
||||
and RMS error both0), and all sampled tokens agree. This run had no stage
|
||||
instrumentation enabled. Evidence is retained under
|
||||
`local-eval-results/glm-stage-20260911.rwQBaJ/mixed-replay.*.log`.
|
||||
The earlier red replay remains historical evidence, not the current result.
|
||||
The optional Rust stage reader exists only under `cfg(test)` and validates
|
||||
tensor geometry before comparing values; it adds no production GPU drains.
|
||||
|
||||
The live verifier at frontier4095/context32768 passed across the4096 boundary,
|
||||
including acceptance, rejection, rewind to either retained frontier, scalar
|
||||
fallback and recurrent-state restoration (`mixed-boundary.*.log`,70.64s).
|
||||
The32-row replay alone is not a complete performance or output-parity claim.
|
||||
The initial source-only note about one-token suffixes was incomplete: the
|
||||
shared UI/headless consumer already routes one-token extensions through scalar
|
||||
execution. The actual remaining crossover was two/three-token extensions;
|
||||
see the subsequent common-consumer correction below.
|
||||
|
||||
### Complete chats after the chunk correction
|
||||
|
||||
Fresh clean runs used the same ongoing workload, Power100/Low, native EOS,
|
||||
separate warmup and no active canary. Rust executable SHA256:
|
||||
`4b23c04325c931854b98c23bd2c98df8a5c2362927aa9b1faed65019d07fd40d`.
|
||||
The original reference retained its prior tokens/text/stops exactly.
|
||||
|
||||
| Mode / turn | Rust tokens | DS4 tokens | Rust decode t/s | DS4 decode t/s | Output + thinking identical |
|
||||
| --- | ---: | ---: | ---: | ---: | --- |
|
||||
| AR Summary |779|779|24.512|20.955|yes|
|
||||
| AR Story |1047|1047|22.956|19.744|yes|
|
||||
| AR Python |198|198|23.436|20.650|yes|
|
||||
| MTP Summary |593|593|20.235|21.427|yes|
|
||||
| MTP Story |905|872|18.039|18.007|no|
|
||||
| MTP Python |169|167|29.443|29.511|no|
|
||||
|
||||
All six Rust turns and six reference turns ended naturally. AR prompt/cached
|
||||
counts also match exactly. Receipts: `clean-comparison.json`,
|
||||
`mixed-ar-output-check.json`, `mixed-mtp-output-check.json` in the stage evidence
|
||||
directory. The AR reference was materially slower than earlier clean runs;
|
||||
these sequential pairs are not a controlled speedup or2% acceptance claim.
|
||||
MTP Summary is about5.6% slower in Rust in this pair; the later MTP throughput
|
||||
numbers do not compare identical histories. Prefill UI-phase and original
|
||||
session-sync timers still have different boundaries (raw values in the receipt).
|
||||
Control-loop maxima of40–59ms are not GPU canary or compositor measurements.
|
||||
|
||||
### Second root cause: MTP stop token retained in the ongoing frontier
|
||||
|
||||
Although MTP Summary text/thinking and593 emitted tokens match, Rust starts
|
||||
Story with3229 cached tokens and3249 prompt tokens; DS4 uses3228/3248.
|
||||
The shared Rust generation consumer returned on an MTP stop token without
|
||||
rewinding the already evaluated block. Both normal and raw original DS4 agent
|
||||
consumers call `ds4_session_rewind(block_start + ti)` at that point. The standalone
|
||||
reference's stop handling therefore agrees with its real agent, not just an
|
||||
arbitrary benchmark convention.
|
||||
|
||||
The shared UI/headless consumer now calls `rewind_speculative_output`, a thin
|
||||
GLM adapter over the existing two-row rollback, to keep exactly
|
||||
`prompt_tokens + emitted_tokens` before retaining the chat.
|
||||
This restores the saved two-row KDA state and replays the retained row; it does
|
||||
not merely truncate IDs or re-render generated text. Invalid frontiers fail
|
||||
explicitly. Both sampled and greedy generation use this consumer. Qwen's own
|
||||
whole-turn controller is unchanged. Other model-specific speculative stop
|
||||
contracts are not claimed validated by this GLM change.
|
||||
The live verifier regression now exercises that same consumer rollback path.
|
||||
`align_prompt` is intentionally not used: it retains one fewer token to force
|
||||
logit recomputation during prompt synchronization, which is a different contract.
|
||||
The full post-frontier-fix MTP measurement (`frontier-mtp.*.log`) now matches
|
||||
the original for **all three turns**: text, thinking, emitted token count,
|
||||
prompt count, cached frontier and natural stop. Emitted counts are593/872/167;
|
||||
Story starts at3228 cached/3248 prompt tokens, Python at4120/4145. The executable
|
||||
SHA256 is `b964336d64fbb90b3a9ca595a4705eda02e7afe9c39aedb4ea775e0d52fcf20e`.
|
||||
`frontier-mtp-output-check.json` has three entries with every equality true;
|
||||
the checked `jq -e` assertion requires all three entries and all five properties.
|
||||
Decode rates are23.880/19.357/31.643t/s, versus21.427/18.007/29.511 in the directly
|
||||
preceding clean original MTP run. This is one sequential pair, not repeated2%
|
||||
acceptance. The Python answer is now exactly the reference's previously checked
|
||||
incorrect answer (first counterexample49); matching the oracle does not waive
|
||||
the independent generated-code quality failure.
|
||||
|
||||
### Final regression and responsiveness diagnostics
|
||||
|
||||
The final strict replay passes with bit-equal logits at all32 steps. The two
|
||||
original layer0/position9 stage tensors each contain8388608 floats and also
|
||||
match bit-for-bit (`final-replay.*.log`,10.21s). The updated live verifier at4095
|
||||
passes through the same rollback entrypoint used by the consumer, including
|
||||
invalid/unchanged-frontier checks, rejection and both retained rows
|
||||
(`final-boundary.*.log`,82.27s). The five enabled GLM unit tests pass.
|
||||
|
||||
`final-canary` retained identical full MTP output/frontiers. Its in-memory
|
||||
summary reports840 samples, no failures, prefill p95/max395.611/483.166ms,
|
||||
decode max3.820ms and no sample crossing the configured2s threshold. However,
|
||||
strict raw-log parsing found an interleaved canary/resource JSON record: the
|
||||
model-eval parent inherited the child's stderr, and both processes serialized
|
||||
JSON fragments to that descriptor. This raw file is retained as a **failed
|
||||
record-integrity diagnostic**, not silently filtered into a complete sample set.
|
||||
|
||||
The model-eval supervisor now pipes child stderr and forwards complete lines
|
||||
under the parent's shared stderr lock, the same lock used by resource samples.
|
||||
Diagnostics do not refresh inference progress deadlines. Reader failures are
|
||||
reported on join. This fixes the app harness counterpart of the earlier
|
||||
standalone watchdog forwarding issue; it changes measurement transport, not
|
||||
GPU scheduling or the UI inference graph.
|
||||
|
||||
The directly following original DS4 in-process probe run
|
||||
(`final-reference-canary`) has841/841 parseable samples, no failures, unchanged
|
||||
reference tokens/text/stops, prefill p95/max373.220/388.090ms and decode max4.370ms.
|
||||
The worst prefill sample spent387.964ms before GPU start,0.002875ms over its
|
||||
GPU interval and0.122ms returning to the host. Thus substantial prefill queue
|
||||
waiting still occurs in the original oracle; the larger Rust spike is not
|
||||
declared equivalent or explained away.
|
||||
|
||||
The repeated Rust run after the forwarding correction (`final-canary-records`)
|
||||
completed the entire chat in83.744s and preserved all output/frontier fields.
|
||||
Every JSON record beginning with `{` in stderr was parsed with `fromjson`
|
||||
(no error suppression): **776/776 canary records and82/82 resource records**
|
||||
match the independently reported totals. The checked receipt is
|
||||
`final-canary-records-check.json`. There are no probe failures or observed2s
|
||||
threshold crossings. Prefill p95/max is119.507/247.181ms; decode max1.854ms.
|
||||
The worst sample waits247.062ms before GPU start, spans0.001750ms on the GPU,
|
||||
and returns after0.115458ms. This lower maximum is not attributed to the
|
||||
transport-only fix: the prior483ms Rust and388ms original spikes remain recorded,
|
||||
and scheduling/throughput variability still requires repeated paired testing.
|
||||
The optional probe remains off by default; no negligible-overhead claim is made.
|
||||
|
||||
Final source verification: release all-target/all-feature build and Clippy
|
||||
with warnings denied; rustfmt/diff checks; five GLM unit tests; eleven
|
||||
model-eval unit tests; sixteen sampling tests including the64-case original
|
||||
sampler fixture; strict live logits/stage and consumer-rollback boundary tests.
|
||||
The final CLI SHA256 is
|
||||
`cbe04f8ce8f8d2fcb6c82b97c3d85b7bed561418893621a6a653d344d1aa6d85`.
|
||||
The previously good bundle remains unchanged at SHA256
|
||||
`ea4d555c2faf0940d9cbcf76d8638ca614a9cb2c6b034e3b2f80aeef86b0b339`.
|
||||
|
||||
## Common prompt timing and DS4 CPU sampling follow-up
|
||||
|
||||
Evidence for this continuation is under
|
||||
`local-eval-results/glm-paired-20260911.eClfCS/`. The preceding goal turn made
|
||||
verified progress (chunk scheduling and stop-token frontier fixes); it did not
|
||||
establish the full six-cell performance goal.
|
||||
|
||||
The shared consumer now uses DS4's GLM5.3 resumed-prefill crossover of2 tokens,
|
||||
not the generic4-token threshold. DS4 explicitly documents this choice as
|
||||
measured on M5 Max/GB10 (`ds4.c:36784`). One-token continuations were already
|
||||
scalar; cold/vision paths and the separate MTPLX whole-turn controller are
|
||||
unchanged. The enabled crossover test covers GLM5.3 versus GLM5.2/DeepSeek.
|
||||
|
||||
DeepSeek/GLM now publish the existing `PromptTiming` at the shared prompt-
|
||||
evaluation boundary: after restoration/bootstrap, around actual suffix execution
|
||||
including its progress callbacks, before decode/checkpoint storage. Exact cache
|
||||
hits report zero evaluated work. Separately unmeasured restore/history components
|
||||
are `null`, not fabricated zeros; Qwen continues reporting the same measured
|
||||
numeric values through `Some`. The new metric test and existing Qwen progress/
|
||||
decode-timer test pass. The ordinary UI-prefill timer remains separately visible.
|
||||
|
||||
A fresh clean AR pair kept all three outputs/thinking/token counts/frontiers
|
||||
identical. Rust's engine-prefill times were5235.183/269.769/320.767ms, original
|
||||
DS4 session-sync8160.750/495.857/479.216ms; Rust decode24.811/23.114/23.410t/s
|
||||
versus16.633/17.333/19.054. These large sequential-run differences are not a
|
||||
controlled speedup or a completed repeat matrix (`baseline-ar-comparison.json`).
|
||||
The reference driver now additionally queries and checks actual engine power100
|
||||
after load, rather than only recording its requested options.
|
||||
|
||||
The CPU sampler still differed algorithmically: Rust sorted the full vocabulary
|
||||
and drew from renormalized probabilities, while DS4 first tries a512-candidate
|
||||
heap and draws from raw retained weights. A CPU-only replay uses the existing32
|
||||
full logit rows, one32-draw warmup and16 measured batches (512 draws). The same
|
||||
small runner serves the independent original public `ds4_sample_logits` and the
|
||||
production Rust sampler. It loads no model and performs no Metal work; both are
|
||||
supervised with1GiB memory/start30s/idle30s limits. The original public function
|
||||
allocates a scratch buffer per call, unlike its session API, so its microbenchmark
|
||||
is not an exact measure of session-sampler overhead.
|
||||
|
||||
Before alignment Rust took2.645ms/draw versus original0.834ms, with all512 tokens
|
||||
equal. The aligned Rust path initially measured0.401ms/draw with the same512
|
||||
tokens (`sampler-{before,after}-rust.json`, `sampler-reference.stdout.log`).
|
||||
It uses stdlib `BinaryHeap`, DS4's logit/index tie order, bounded-nucleus fallback
|
||||
without advancing RNG, original raw cumulative sampling, full-vocabulary/min-p
|
||||
fallback and the original expf-verified log-space rejection boundary. Top-k
|
||||
retains the original1024 cap. Separate distribution materialization for
|
||||
speculative correction and Qwen's MTPLX sampler are untouched.
|
||||
|
||||
All64 original sampler fixture cases and17 enabled sampling tests pass, as does
|
||||
the added missing-mass/near-one fallback, RNG and signed-zero tie check. Release
|
||||
all-target/all-feature build and warnings-denied Clippy pass. The new executable
|
||||
SHA256 is `ece6aed3601fb402e6dba6ac2e289d6e0c2dc86663600c3d4b1a4cc07e8fb42c`.
|
||||
The first full post-sampler AR and MTP pairs both preserve all three outputs,
|
||||
thinking, completion/prompt/cached counts and natural stops. The independently
|
||||
queried reference engine reports power100. Receipts are
|
||||
`sampler-{ar,mtp}-comparison.json`; these are single pairs, not the repeat matrix.
|
||||
|
||||
| Mode / turn | Rust / original engine-prefill ms | Rust / original decode t/s |
|
||||
| --- | ---: | ---: |
|
||||
| AR Summary | 5208.845 / 5548.775 | 26.129 / 24.711 |
|
||||
| AR Story | 267.263 / 287.268 | 24.553 / 23.728 |
|
||||
| AR Python | 319.127 / 341.355 | 24.964 / 24.520 |
|
||||
| MTP Summary | 6574.400 / 5427.866 | 23.935 / 23.628 |
|
||||
| MTP Story | 273.240 / 278.956 | 19.419 / 19.089 |
|
||||
| MTP Python | 315.279 / 360.621 | 33.140 / 31.919 |
|
||||
|
||||
The Summary MTP prefill regression in this pair remains visible despite the
|
||||
slightly faster Rust decode. Reversed-order repetitions are needed to distinguish
|
||||
run variability from a repeatable graph cost. AR before/after the sampler keeps
|
||||
the entire chat output identical and improves decode by5.310/6.225/6.638% in this
|
||||
one sequential comparison (`sampler-ar-before-after.json`); no controlled causal
|
||||
end-to-end percentage is inferred from that pair alone.
|
||||
|
||||
MTP is not universally beneficial in the original either: its Story decode is
|
||||
19.089t/s versus23.728 AR, while Python is31.919 versus24.520. Rust's full MTP
|
||||
cycle receipts show228/366,289/584 and82/86 accepted drafts respectively
|
||||
(62.3%,49.5%,95.3%). The corresponding complete decode-loop time per cycle is
|
||||
67.69/76.89/58.60ms. At1.62/1.49/1.94 emitted tokens per cycle, the Python case
|
||||
amortizes the extra draft/verification work much better. These are whole-cycle
|
||||
averages, not isolated kernel timings: the existing `verifier_ms` includes other
|
||||
cycle work and must not be presented as an exclusive verification stage.
|
||||
AR and MTP have different natural histories, so their t/s comparison is not a
|
||||
matched-token microbenchmark. The previously recorded Python correctness failure
|
||||
also remains open even though both implementations produce the same code.
|
||||
|
||||
### Reversed-order pairs: acceptance still fails
|
||||
|
||||
Both modes were repeated in original-then-Rust order, serially without builds
|
||||
or canary probes. All twelve measured answers in these four processes again
|
||||
match text/thinking/counts/cache frontiers and end naturally; all watchdogs
|
||||
exit successfully. No slow run was discarded (`repeat2-*-comparison.json`).
|
||||
|
||||
| Mode / turn | Rust / original engine-prefill ms | Rust / original decode t/s |
|
||||
| --- | ---: | ---: |
|
||||
| AR Summary | 6738.280 / 5283.041 | 23.569 / 25.535 |
|
||||
| AR Story | 309.647 / 281.591 | 21.743 / 24.186 |
|
||||
| AR Python | 387.381 / 329.667 | 21.107 / 24.898 |
|
||||
| MTP Summary | 9240.178 / 8835.102 | 17.706 / 17.678 |
|
||||
| MTP Story | 358.905 / 402.447 | 15.146 / 14.581 |
|
||||
| MTP Python | 403.605 / 491.547 | 25.843 / 23.655 |
|
||||
|
||||
AR decode now misses by7.70/10.10/15.23%; MTP Summary prefill misses by4.38%.
|
||||
The subsequent original MTP run is itself much slower than its first run.
|
||||
This excludes neither a Rust scheduling difference nor changing device clocks;
|
||||
it does preclude a pass based on the favorable first pair or a selected median.
|
||||
The required third pair and full six-cell acceptance remain outstanding.
|
||||
|
||||
Rust AR emits exactly9445/12612/2424 command buffers in both repetitions, with
|
||||
the same outputs, but its GPU timestamp-interval sums increase from
|
||||
34536/42061/8084ms to39285/47601/9590ms (`ar-drift-comparison.json`). Those sums
|
||||
are `GPUEndTime - GPUStartTime` and may include preemption; they are not exclusive
|
||||
kernel or hardware-clock measurements. The slowdown is not explained by changed
|
||||
token counts or extra command buffers, and is not declared thermal throttling.
|
||||
During the sequence, a read-only process snapshot showed only the intended
|
||||
reference model process. macOS reported no recorded thermal/performance warning
|
||||
and normal VM pressure (1), which does not exclude frequency changes. The
|
||||
AGX PerformanceStatistics snapshot exposes utilization but no frequency field.
|
||||
Hardware was freshly checked: Apple M5 Max,128GiB,18 logical CPUs.
|
||||
|
||||
## Remaining acceptance
|
||||
|
||||
- Compare repeated clean throughput pairs; GLM ongoing histories now match in
|
||||
both modes, but sequential run variability does not establish2% performance parity.
|
||||
- Localize the remaining GLM prefill cost against original DS4's active indexed
|
||||
path, now that in-process GPU-start waiting is observable on both sides.
|
||||
The engine-prefill timer is now exposed separately from UI-phase timing;
|
||||
use that aligned boundary in the paired comparisons.
|
||||
No chunk reduction or extra waits are justified by these measurements alone.
|
||||
- Verify the corrected short-extension crossover live where needed, and other
|
||||
model-specific speculative stop contracts; the recorded GLM workload does not
|
||||
cover every possible interaction. Full chats pass after CPU-sampler alignment;
|
||||
repeated timing acceptance remains separate.
|
||||
- Validate the remaining SSD expert-streaming cases separately from resident
|
||||
scheduling. This is unrelated to replacing DS4 KV checkpoint persistence.
|
||||
- Complete the DeepSeek AR/DSpark reference cells and Qwen residual performance
|
||||
analysis. Interactive confirmations are not a substitute for the six-cell
|
||||
numerical acceptance matrix.
|
||||
|
||||
No bundle replacement, commit or push has been performed by this follow-up so far.
|
||||
All processes have terminated. The subsequent DeepSeek comparison and its
|
||||
separate bootstrap/DSpark findings are recorded in
|
||||
[DeepSeek follow-up](deepseek-reference-followup-20260911.md).
|
||||
@@ -7931,6 +7931,19 @@ Verifikation 2026-09-05, ohne GPU-Modelllauf oder breite Gates:
|
||||
- `git diff --check`: bestanden. Keine Durchsatzverbesserung aus diesen Tests
|
||||
abgeleitet. Vor der neuen DSpark-Messung bleiben P02/P03/P04 maßgeblich.
|
||||
|
||||
**Gemeinsamer DS4/GLM-Sampler angeglichen (11.09.):** Der CPU-only-Replay mit
|
||||
32 echten GLM-Logit-Zeilen und512 gemessenen Auswahlen zeigte2,645ms pro Schritt
|
||||
in Rust gegenüber0,834ms über die originale öffentliche DS4-Funktion. Die Rust-
|
||||
Implementierung benutzt nun den begrenzten512-Kandidaten-Heap, originale Logit-
|
||||
Tie-Reihenfolge, rohe CDF-Summen und DS4-Fallbacks statt vollständiger Sortierung
|
||||
mit erneuter Normalisierung. Initial0,401ms/Schritt bei identischen512 Tokens;
|
||||
64 Original-Fixturefälle und17 Sampling-Tests bestanden. Qwens MTPLX-Sampler und
|
||||
die separate Verteilungsberechnung für spekulative Korrektur bleiben unverändert.
|
||||
Ein kompletter GLM-AR-Chat bewahrt Ausgabe/Thinking/Frontiers und zeigt im einzelnen
|
||||
Vorher-/Nachher-Paar5,3–6,6% höheren Decode-Durchsatz. Das ersetzt weder die
|
||||
DSpark-Modusabnahme noch die wiederholte Gesamtmatrix. Details im
|
||||
[GLM follow-up](glm-scheduling-followup-20260911.md).
|
||||
|
||||
### P14 — GLM-MTP tatsächlich batchen wie DS4
|
||||
|
||||
**Produktpfad umgesetzt, Referenz-Performanceabnahme offen (10.09.):**
|
||||
@@ -7996,6 +8009,61 @@ Priorität P1; nach P04, P13; Quellcodedifferenz belegt.
|
||||
|
||||
### P15 — DS/GLM-Scheduling, Kaltstart und gemeinsame Runtime abgleichen
|
||||
|
||||
**Numerische GLM-Abweichung reproduziert (Fortsetzung11.09.):**
|
||||
64 modellfreie Original-DS4-Sampler-Fälle sichern Tokens und RNG-Zustand ab;
|
||||
Greedy-Tie/RNG- und Seed0-Abweichungen sind korrigiert, ohne Qwen zu ändern.
|
||||
Die verwendeten positiven Seed42-Benchmarkfälle waren davon nicht betroffen.
|
||||
Der neue feste-Historie-Test auf32 echten Original-DS4-Logit-Zeilen trennt die
|
||||
Ursachen: Rust-Sampling derselben Zeilen liefert32/32 identische Tokens; Rust-
|
||||
GPU-Ausführung derselben Eingaben liefert bereits nach Prefill max5,722/RMS0,851
|
||||
Logit-Abweichung und bei Schritt17 ein anderes Sample. Der Test bleibt bewusst
|
||||
rot und ignoriert für normale modellfreie Läufe. Die Referenz selbst bewahrt
|
||||
den vollständigen AR-Chat exakt trotz optionalem Trace. Nächste Priorität ist
|
||||
die erste divergierende HC/KDA/DSA/FFN-Stufe, nicht das Durchsatzetikett auf
|
||||
auseinanderlaufenden Chathistorien. Belege/Kommandos und Grenzen stehen im
|
||||
[GLM follow-up](glm-scheduling-followup-20260911.md).
|
||||
|
||||
**Fortsetzung 11.09.:** Der Benutzer bestätigt jetzt normalen GLM-Decode und
|
||||
benutzbaren, aber gegenüber Qwen weniger geschmeidigen Prefill. Kein vollständiger
|
||||
Freeze behauptet. Die unten am10.09. identifizierte residente skalare Flush-
|
||||
Abweichung ist in beiden GLM-Schleifen korrigiert (periodisch alle4, ohne letzten
|
||||
Layer/SSD-Streaming); Prefill-Fortschritt meldet nur abgeschlossene bestehende
|
||||
Drains und den Batchabschluss. Frische vollständige AR/MTP-Vorher-/Nachher-Chats
|
||||
bewahren Tokens/Text/Thinking/EOS ohne beobachtete harte Regression; einzelne
|
||||
Laufpaare sind kein kontrollierter Speedup-Beweis. Der ausdrücklich genehmigte
|
||||
Standalone-DS4-Benchmark benutzt originale DS4-Objekte ausschließlich außerhalb
|
||||
des Produkts und reproduziert Low, separaten Bootstrap und laufende Chathistorie.
|
||||
Die separate Canary-Prozessplatzierung war kein fairer Vergleich zum internen
|
||||
Canary: gleichzeitig intern490,740ms versus extern höchstens3,331ms. Neue native
|
||||
Metal-Zeitstempel lokalisieren eine300ms-Prefill-Probe fast vollständig vor dem
|
||||
GPU-Start, nicht in der Host-Rückmeldung. Dieselbe Probe innerhalb des originalen
|
||||
DS4-Prozesses reproduziert264,292ms Prefill-Spitze, ebenfalls vor dem GPU-Start;
|
||||
771/771 Datensätze sind nach Korrektur der Watchdog-Ausgabeweiterleitung sauber
|
||||
auswertbar. Die restliche Kosten-/Latenzdifferenz und weitere Abnahme bleiben in
|
||||
[GLM follow-up](glm-scheduling-followup-20260911.md). P15 bleibt offen.
|
||||
|
||||
**GLM-Ausführungsursache lokalisiert (11.09.):** Rust übernahm die GLM-5.2-
|
||||
Top-k-Chunkgrenze für GLM5.3: nach9 Bootstrap-Tokens2039 statt der originalen2048
|
||||
Tokens. Nun bleiben GLM5.3-Chunks wie im aktiven DS4-Pfad vollständig; nur die
|
||||
Attention-Slices werden an der Dense/Sparse-Grenze geteilt, auch für kreuzende
|
||||
Verifier-Paare. Im unverändert vorgegebenen32-Schritt-Replay sind danach alle
|
||||
Logit-Vektoren bitidentisch zur Originalreferenz, vorher lagen deutliche Fehler
|
||||
bereits im ersten Vektor vor. Vollständige AR/MTP-Chats, Grenzfall-Livetest und
|
||||
Performance-Neumessung bleiben erforderlich; dies allein schließt P15 nicht.
|
||||
|
||||
**Nachprüfung desselben Tages:** Der vollständige AR-Chat stimmt nun in allen
|
||||
drei Antworten samt Thinking, Tokenzahlen und Prompt-/Cache-Grenzen überein.
|
||||
Bei MTP wurde zusätzlich ein im gemeinsamen UI-/Harness-Consumer behaltenes
|
||||
Stop-Token gefunden: der zweite Prompt hatte3249 statt3248 Tokens. Der Consumer
|
||||
nutzt nun GLMs vorhandenen Zwei-Zeilen-Rollback, analog zur Stop-Behandlung des
|
||||
originalen DS4-Agenten. Danach stimmt auch der vollständige MTP-Chat in allen
|
||||
drei Antworten und Grenzen überein (593/872/167 ausgegebene Tokens). Der
|
||||
Attention-Grenzfall4095+2 wurde live geprüft. Der strikte32-Logit-Replay und die
|
||||
großen Layer0-Tensorvergleiche sind bitidentisch. Einzelne neue Geschwindigkeits-
|
||||
paare und diese konkrete Workload schließen nicht die gesamte2%-Matrix oder
|
||||
alle Interaktionsfälle ab; P15 bleibt offen. Ergebnisse und Einschränkungen im
|
||||
[GLM follow-up](glm-scheduling-followup-20260911.md).
|
||||
|
||||
**Aktiver GLM-Pfad und durchgehender Canary geprüft (10.09.):**
|
||||
`local-eval-results/glm-scheduling-canary-20260910.T1ABR1/{manifest.md,comparison.json}`
|
||||
enthält zwei vollständige AR/MTP-Chats bis EOS bei Power100/Low und Canary an.
|
||||
@@ -8061,6 +8129,61 @@ Priorität P1; nach P01, P04, P13; Source-Inventar vorhanden, Detailbeweis offen
|
||||
|
||||
Priorität P1; nach den relevanten P01–P15; offen.
|
||||
|
||||
**Original-DS4-Vergleich erweitert (11.09.):** GLM-AR/MTP bewahren nach dem
|
||||
CPU-Sampler-Abgleich in je zwei vollständigen Paaren Ausgabe/Thinking/Frontiers.
|
||||
Die umgekehrte Reihenfolge zeigt dennoch AR-Decode-Rückstände bis15,23% und
|
||||
MTP-Summary-Prefill bis4,38%; auch die Originalreferenz schwankt stark. Kein
|
||||
Median-Pass, keine thermische Erklärung ohne Taktbeleg. Details und unveränderte
|
||||
Command-Buffer-Zahlen im [GLM follow-up](glm-scheduling-followup-20260911.md).
|
||||
|
||||
Der erste DeepSeek-Vergleich ist ausdrücklich ungültig: Der Referenztreiber
|
||||
forderte2048 statt des UI-/DS4-Automatikwerts4096 an. Zusätzlich fügte der
|
||||
gemeinsame Produkt-Renderer bei einem reinen Bootstrap-Cache fälschlich EOS vor
|
||||
dem ersten User-Prompt ein (2742 statt2741 Tokens). Der Tokenizer selbst stimmt.
|
||||
Ein gezielter CPU-Test reproduziert diesen UI-/Headless-Fehler vor der Korrektur;
|
||||
nach dem Nichtleer-History-Guard bestehen DeepSeek- und GLM-Originalfixtures.
|
||||
Referenztreiber auf Automatik angeglichen, keine Produkt-Chunkverkleinerung.
|
||||
Das korrigierte vollständige DeepSeek-AR-Paar stimmt nun in allen Antworten,
|
||||
Thinking-, Token- und Cache-Feldern überein, verfehlt aber im Einzelpaar noch
|
||||
das Durchsatzziel. Das neue DSpark-Paar scheitert funktional: erste Antwort
|
||||
bereits647 statt1208 Tokens bei identischem Prompt; anschließend jeweils ein
|
||||
zusätzlich behaltenes EOS im Rust-Cache. Die normale Beendigung und bestandene
|
||||
Python-Tests ändern diesen roten Vergleich nicht. Nächste zusammenhängende
|
||||
DSpark-Einheit: erster divergenter Proposal-/Verify-Zyklus, Stop-/Capture-Frontier
|
||||
und zusätzliche Vollvokabular-Readbacks gemeinsam gegen Original DS4 prüfen
|
||||
und angleichen, dann neu messen. Konkrete Quellen, Rohdaten und Tests im
|
||||
[DeepSeek follow-up](deepseek-reference-followup-20260911.md).
|
||||
|
||||
**DSpark-Zyklus-/Zustandsabgleich fortgesetzt:** Der neue vollständige
|
||||
Diagnoselauf stimmt jetzt in allen1755 Originalzyklen einschließlich Warmup,
|
||||
Antworten, Thinking und Cachepositionen überein. Ursachen waren fehlende
|
||||
HC-/Attention-Norm beim Support-KV-Aufbau, der abweichende Einzeldraft-Verifier,
|
||||
Seed-/Teilannahme-Capture, Cachefenster-/Scheduler-/EOS-Übergänge und fehlende
|
||||
Aktivierungsquantisierung des tatsächlich installierten Q8-Confidence-Kopfs.
|
||||
Zusammenhängende Quellen-/Taskliste und Rot→Grün-Belege stehen im verlinkten
|
||||
Follow-up. Early-Confidence-Gate und Verifier-Readbacks/Submission sind inzwischen
|
||||
ebenfalls angeglichen; alle1755 Originalzyklen bleiben exakt gleich. Noch keine
|
||||
Leistungsabnahme: saubere Wiederholungspaare zeigen starke zeitliche Drift in
|
||||
beiden Implementierungen. Die CPU-Worker-Policy ist ebenfalls angeglichen:
|
||||
dauerhafte12 Threads inklusive Aufrufer, weiterhin1755 identische Zyklen.
|
||||
Ein CPU-only-Dispatchvergleich zeigt einen kleinen Gewinn, keine Erklärung für
|
||||
den verbleibenden Gesamtgap. Verifier-/Frontier-Puffer werden inzwischen über
|
||||
Zyklen wiederverwendet; der32-Zyklen-Test prüft Pufferidentität und bytegenaue
|
||||
Rücknahme, der vollständige Chat weiterhin alle1755 Originalzyklen. Auch der
|
||||
Exact-Sampling-Test besteht. Die jüngsten beiden Paare verfehlen jedoch weiter
|
||||
Teile des Durchsatzziels; kein belegter Gesamtgewinn durch diese Änderung.
|
||||
|
||||
**Messbedingung nach Nutzerklarstellung (11.09.):** Parallel laufen Videos und
|
||||
beanspruchen einen Teil der GPU. Die genaue zeitliche Überlappung früherer
|
||||
Messungen ist unbekannt. Die jüngsten seriellen Paare bleiben als Rohdaten
|
||||
erhalten, gelten aber nicht als kontrollierte2%-Abnahme oder kausaler
|
||||
Vorher/Nachher-Beleg. Weder alle Differenzen auf Videos schieben noch thermische
|
||||
Drosselung behaupten. Keine weiteren Durchsatzserien während dieser Nutzung;
|
||||
Funktionsprüfungen und Code-Abgleich bleiben möglich. Die vollständige
|
||||
Sechsfeldmatrix, Responsivitätsabnahme und Zähler-Scope-Bereinigung bleiben offen.
|
||||
Der Nutzer hat den Abschluss und Commit/Push dieses Zwischenstands freigegeben;
|
||||
die vorgeschriebenen Commit-Gates umfassen auch den erneuten Bundle-Bau.
|
||||
|
||||
**Aktuelles interaktives Test-Bundle bereitgestellt (10.09.,22:32):**
|
||||
`make bundle` und strikte Codesign-Prüfung bestanden. Das Bundle enthält den
|
||||
zuletzt vermessenen Release-Code und identische Metal-Ressourcen; der genaue
|
||||
|
||||
Reference in New Issue
Block a user