Apply Qwen vision RoPE per row across table boundaries
This commit is contained in:
1 parent
dccf81971f
commit
892f8a70b9
11 files changed
+580
-21
No files matched your search
@@ -0,0 +1,57 @@
|
||||
# Issue 108: row-wise attention positions
|
||||
|
||||
Qwen attention now implements pinned MTPLX's `_vision_chunk_cos_sin` route:
|
||||
covered rows use the original table slice and M-RoPE graph; uncovered rows use
|
||||
plain RoPE at `index + delta`; mixed chunks concatenate those two tables. The
|
||||
YaRN amplitude applies to both portions. With the shipped scaling of one, wholly
|
||||
inside/outside calls build the same numerical graphs as before. The text-only
|
||||
branch and indexer remain unchanged. Production changes are confined to two
|
||||
private Qwen modules, with no DS4/GLM callers or shared numerical changes.
|
||||
|
||||
This fixes a **latent module mismatch**. Review of `vision::prepare`, owner setup,
|
||||
model dispatch, suffix prefill and MTP history found no new product trigger:
|
||||
`prepare` builds a table over the complete rendered prompt; prefill stops within
|
||||
it, and decoding starts after it. The final post-image text positions are equal
|
||||
across axes at `index + delta`. A normal MTP window crossing that text boundary
|
||||
is therefore not a reproduction. No state rebase, prompt mutation or stale-table
|
||||
restore was added. No new DS4Server feature is involved.
|
||||
|
||||
## Verification
|
||||
|
||||
`tools/issue108-reference.py` runs the pinned MTPLX revision
|
||||
`9882703f3105363ddc37eca9f97aa09a1d387112` in isolated MLX 0.32.2. It uses the
|
||||
oracle's actual frequency constructor and chunk helper, plus independent NumPy
|
||||
per-row checks. Its 56 cases cover small and shipped head geometry (dim256,
|
||||
24 query heads, rotary64), F32/BF16, scaling 1/1.25, wholly inside/outside,
|
||||
unequal image-axis straddling, equal axes, absent/empty tables, and a full-prompt
|
||||
post-image text tail. Native cosine/sine and Q/K outputs match every oracle byte.
|
||||
The text-tail and equal-axis crossings also match plain shifted RoPE exactly.
|
||||
|
||||
The owned `vision_positions` case in `issue103_vision_position_receipts` now
|
||||
requires corrected cached keys. Actual Attention calls compare one mixed chunk
|
||||
with the same input split at the table end: outputs, valid K/V, offsets and the
|
||||
next call's output/state agree exactly. The unrelated sparse-QSA and attention
|
||||
Gate diagnostic cases retain their current separate ownership (#107/#109).
|
||||
|
||||
Existing whole-attention fixtures exercise the unchanged text/vision controls,
|
||||
cache hashes and execution routes in both TF32 modes, with lazy and prepared
|
||||
inputs. The mandatory Rust formatting, Clippy, bundle and complete ordinary test
|
||||
gates are recorded in `checks.json`; the adjacent JSONL receipts retain the
|
||||
owned test results. These are correctness checks, not performance claims.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```sh
|
||||
local-eval-results/mtplx-reference-env-0.32.2/bin/python \
|
||||
tools/issue108-reference.py /path/to/pinned/mtplx /tmp/issue108-oracle
|
||||
cargo test --all-features --bin ds4-server --no-run
|
||||
cargo build --bin test-supervisor
|
||||
export DS4_ISSUE108_ORACLE=/tmp/issue108-oracle
|
||||
# Use the test executable printed by cargo, not cargo itself in the supervisor.
|
||||
target/debug/test-supervisor 4096 60 120 --command TEST_EXECUTABLE \
|
||||
issue108_attention_chunk_positions --ignored --nocapture --test-threads=1
|
||||
target/debug/test-supervisor 4096 60 120 --command TEST_EXECUTABLE \
|
||||
issue103_vision_position_receipts --ignored --nocapture --test-threads=1
|
||||
# Run mtplx_qsa_attention_module_matches_reference the same way with
|
||||
# MLX_ENABLE_TF32=1 and then MLX_ENABLE_TF32=0, serially.
|
||||
```
|
||||
Reference in new issue
Block a user