74 lines
5.6 KiB
Markdown
74 lines
5.6 KiB
Markdown
# Issue 106: complete MTP final-primary state
|
|
|
|
Qwen's round-head stop/token-limit exit now re-arms its emitted primary for the
|
|
existing final commit, exactly as pinned MTPLX 2.12.2
|
|
`9882703f3105363ddc37eca9f97aa09a1d387112`, `generation.py:13001-13019` does.
|
|
That commit updates committed MTP history and forwards the primary once. Sampling,
|
|
acceptance, RNG, snapshot ownership and other exit branches are unchanged.
|
|
|
|
Qwen checkpoint magic is now `QWMTKV02`. Old `QWMTKV01` files are rejected before
|
|
tensor allocation; Generator's existing incompatible-cache path removes the file,
|
|
resets the owner and rebuilds cold. This deliberately invalidates all old Qwen
|
|
checkpoints, including AR files, because V1 does not certify complete MTP state.
|
|
It does not alter the DS4/GLM format or their owners. This is a DS4Server persistence
|
|
version change around MTPLX's existing inference behavior.
|
|
|
|
There is no in-process executable hot-upgrade mechanism: resident sessions belong
|
|
to the process-local Generator. Installing a new executable takes effect on
|
|
restart, which drops all resident state. Runtime engine changes also drop the old
|
|
Generator (`runtime.rs`, `run_command`). New processes cannot import old resident
|
|
objects, and old processes' subsequently written V1 files are still rejected.
|
|
|
|
## Acceptance review
|
|
|
|
| Requirement | Evidence |
|
|
| --- | --- |
|
|
| Real pre-fix reproduction | Parent `5ecf8ab`, unchanged model, 12 first-primary cases: key 4, all 12 trunk offsets 3, MTP offset 2, pending absent and no error. `before.jsonl`. |
|
|
| Actually persisted bad state | A real `QWMTKV01` file was saved from the sampled depth-1 reproduction, with IDs `[9707,11,1075,314]`. File identity and hash in `checks.json`; the 116 MB cache is not checked into Git. |
|
|
| Full state, not offsets alone | All 12 cases now have trunk 4 and history 3. Every encoded tensor's raw bytes, dtype, shape and metadata, including GDN/conv, PLE, QSA/KV, MTP history, logits and hidden state, match independent teacher forcing. `native-state.jsonl`. |
|
|
| Exactly once and next turn | Teacher forcing restores the prompt, explicitly appends one MTP history row and forwards one primary. All corrected snapshots match it, persist/reopen exactly, restore four tokens, then match the full state after suffix `[1075]`. |
|
|
| All upstream exits | Eight cases compare token sequences, cache lengths and stop origin with the pinned upstream scripted-model test: bonus, deferred correction, accepted draft, sampled correction, first/fresh primary, and both max-token depths. Native tests script only logits; real model computation and caches still execute. Context copy is disabled in both oracle and native matrix so the intended draft/verify routes are reached. `oracle.jsonl`, `native-matrix.jsonl`. |
|
|
| No false final capture on failure | Cancellation at finalization returns an error; failed final forward leaves pending/error and no safe final arrays; disabling final capture leaves pending. `native-state.jsonl`. |
|
|
| Old cache recovery in product | The real old file is rejected and removed by Generator, and recovery reports zero reused tokens. Its entire newly persisted state and response equal an independent cold request. `native-product.jsonl`. |
|
|
| Existing persistence contracts | Model-free checkpoint roundtrip, corruption, failed restore, cancellation and atomic-write regression run under the Metal supervisor. |
|
|
| Other families | Production diff is confined to Qwen's MTP exit and checkpoint signature. DS4/GLM retain `DS4RKV01`/version 2 and their existing pending/state behavior; shared RNG and inference helpers are unchanged. Ordinary cargo tests retain DS4 token/RNG controls. |
|
|
|
|
The former bug-asserting `issue103_mtp_final_receipts` diagnostic is replaced by
|
|
these positive-contract regressions. The only new Execution hooks are `cfg(test)`
|
|
scripted logits/failure injection, absent from release builds. No Python is called
|
|
by the product. These receipts establish correctness, not speed.
|
|
|
|
## Reproduction
|
|
|
|
Use the unchanged Bare-Speed revision
|
|
`74559cdf34fbfc0b593de72d17e93f37fd4f9ea7`, M5 Max 128 GiB, SSD PLE,
|
|
MLX 0.32.2 for the isolated oracle. Artifact/environment hashes were refreshed in
|
|
`../issue104-ar/environment.json`. Native raw prompt is `[9707,11,1075]`,
|
|
chunk 2048, seed 0, depths 1/2/3, temperatures 0/.6.
|
|
|
|
To regenerate an actual old file, use an isolated checkout of `5ecf8ab`, apply
|
|
`baseline-save.patch`, build its tests and run `issue103_mtp_final_receipts` with
|
|
`DS4_ISSUE106_BAD_CHECKPOINT` pointing to the output file. The patch only adds
|
|
persistence to the old diagnostic; it does not manufacture a header or cache.
|
|
|
|
Build current tests with `cargo test --all-features --bin ds4-server --no-run`.
|
|
Set `DS4_QWEN38_ARTIFACTS` to the installed model, `DS4_ISSUE106_BAD_CHECKPOINT`
|
|
to that real old file, and `DS4_ISSUE106_DIR` to a temporary output directory.
|
|
Run each ignored test separately and serially through
|
|
`target/debug/test-supervisor 114688 60 120 --command <test-executable>
|
|
<test-name> --ignored --nocapture --test-threads=1`:
|
|
|
|
- `issue106_complete_state_and_checkpoint_recovery`
|
|
- `issue106_product_discards_old_checkpoint_and_recovers_cold`
|
|
- `issue106_upstream_exit_matrix` (also `MTPLX_CONTEXT_COPY=0`,
|
|
`DS4_ISSUE106_ORACLE` pointing to fresh oracle JSONL).
|
|
|
|
Generate the oracle with `MTPLX_CONTEXT_COPY=0 <isolated-python>
|
|
tools/issue106-reference.py <pinned-mtplx-checkout>`, with pytest and the pinned
|
|
reference dependencies installed in that isolated environment.
|
|
Run `qwen_checkpoint_roundtrip_and_failed_restore_preserve_live_state` separately
|
|
with a 4096 MiB supervisor cap and `DS4_QWEN38_CHECKPOINT_DIR` set.
|
|
|
|
All four mandatory commit gates and individual supervisor outcomes are recorded
|
|
in `checks.json`.
|