105 lines
6.2 KiB
Markdown
105 lines
6.2 KiB
Markdown
# Issue 107: image-aware sparse QSA
|
|
|
|
Qwen vision now retains the indexer's sparse selection. The indexer receives the
|
|
request's position table and delta, rotates queries row by row and pooled keys at
|
|
each block's first token, and uses the eager preparation route selected by pinned
|
|
MTPLX `9882703f3105363ddc37eca9f97aa09a1d387112`. Its compiled/fused preparation
|
|
is ineligible with image positions; the existing sparse selectors/consumers remain
|
|
eligible. Budgets, thresholds, text preparation and attention projection RoPE are
|
|
unchanged. Issue 108 still owns the attention projection's table-straddling case.
|
|
|
|
The indexer's new helper implements `_vision_position_cos_sin` using the native
|
|
array operations, including I32 positions, clipped table gather, per-row selection,
|
|
axis gather, trigonometry and YaRN amplitude. Signed I32 Gather reuses the existing
|
|
U32 payload-copy kernel: it performs no value arithmetic or float conversion.
|
|
Tests include signed extrema on both front-axis and general gather routes.
|
|
|
|
Image identities now include a numerical policy: 1 for image-aware sparse QSA,
|
|
0 for the old dense policy (also MTPLX's explicit diagnostic rollback flag).
|
|
Missing fields deserialize as 0. Old vision checkpoints are rejected before
|
|
tensor materialization, obsolete live/resident owners are discarded, and stale
|
|
owners cannot save a checkpoint. Identical image bytes and token spans do not
|
|
bypass this policy. Text-only identities remain empty and compatible. The V1
|
|
signature rejection delivered in issue 106 remains in force; this issue also
|
|
rejects affected V2 vision files without invalidating unaffected V2 text files.
|
|
|
|
Request prefix matching additionally rejects a restore ending strictly inside an
|
|
image span, matching MTPLX's image-safe restore boundary. Complete image spans and
|
|
preceding text prefixes remain eligible. Existing image digests continue to
|
|
distinguish different pixels with identical image-pad token IDs.
|
|
|
|
## Acceptance and review
|
|
|
|
- The owned `vision_sparse_qsa` regression now requires equal sparse output for
|
|
text and a vision context with equal text positions. Separate issue 108/109
|
|
diagnostic assertions remain unchanged.
|
|
- Real indexer geometry: four heads, dim128, ratio4, top-k512, 2,072 raw key rows,
|
|
518 pooled blocks. Query rows and block starts cross a 2,060-row position table.
|
|
Table, equal-axis, delta-only and empty-table cases match pinned MTPLX exactly
|
|
in F32 and BF16; independent NumPy math verifies the F32 values. Normal fused
|
|
selectors remain active while compiled preparation is correctly bypassed.
|
|
- A real 2,095-token, two-image request exceeds the sparse budget. Corrected final
|
|
model logits match MTPLX byte-for-byte: SHA256
|
|
`8b97664316bec6113c85b7dfca9b09967391f019ea1064b83d81a3d9d5191354`.
|
|
Pre-fix logits differ (`a79aa02a98542adeea636f22c85fc5f929308dc5dc75658d4072e6f275d22abe`).
|
|
- The actual pre-fix dense-vision checkpoint is 412,628,836 bytes, SHA256
|
|
`7ddc663d4f337ff272000a7237dac05a224262bf25bc88257e784547b2056326`.
|
|
Tests reject its normal disk load and import its original tensor payload only
|
|
through a test helper to verify stale live/resident rejection and save refusal.
|
|
- Product tests cover unchanged images, identical complete-state restore/reopen
|
|
and follow-up, text edits before/between/after two images, different pixels with
|
|
identical pad IDs, cancellation/resend, a prefix ending inside an image, and
|
|
unaffected text checkpoints. Comparisons cover all serialized tensors and state
|
|
metadata, not only response text.
|
|
- DS4/GLM do not call the modified Qwen modules. Their sampling, RNG, vision and
|
|
checkpoint formats are unchanged. This is MTPLX behavior plus DS4Server's
|
|
application-specific checkpoint policy integration.
|
|
- The 12 affected whole-attention fixture rows were regenerated from the pinned
|
|
oracle (profiles 7 and 9, TF32 on/off). All 56 fixture rows pass with both lazy
|
|
and prepared inputs, including output/state hashes, consumer routes, compiled
|
|
core lifecycle and counters. Existing text-only indexer route fixtures pass.
|
|
|
|
All four commit gates passed. The initial general test run hit an unrelated
|
|
`dev_brain` fixture whose temporary short Git hash was all digits and therefore
|
|
parsed as a YAML integer. Its focused retry and the subsequent full suite passed;
|
|
no unrelated source change was included. `checks.json` records both outcomes.
|
|
|
|
The tests and receipts are correctness evidence, not performance measurements.
|
|
See `checks.json` and the accompanying JSONL files for actual run outcomes.
|
|
|
|
## Reproduction
|
|
|
|
Use the same unchanged Bare-Speed artifact and environment documented in
|
|
`../issue104-ar/environment.json`, including the pinned vision tower. The input
|
|
image is `local-eval-results/qwen-vision-input/image.png` (1024 square pixels),
|
|
repeated twice; tests generate an independent 1024-square blue control.
|
|
|
|
An isolated checkout of parent `a8cad88` plus `baseline-capture.patch` regenerates
|
|
the actual old snapshot with `issue107_vision_product`. Set `DS4_ISSUE107_DIR` to
|
|
its output directory. Do not replace a new file's header to fake old state.
|
|
|
|
Generate fresh reference results in the isolated MLX 0.32.2 environment:
|
|
|
|
```sh
|
|
python tools/issue107-model-reference.py PINNED_MTPLX MODEL IMAGE ORACLE_DIR
|
|
python tools/issue107-indexer-reference.py PINNED_MTPLX INDEXER_DIR
|
|
```
|
|
|
|
Build native tests with `cargo test --all-features --bin ds4-server --no-run`.
|
|
Run ignored tests one at a time via `test-supervisor MEMORY_MIB 60 120 --command
|
|
TEST_EXECUTABLE TEST_NAME --ignored --nocapture --test-threads=1`. Use 114688 MiB
|
|
for the product test and 4096 MiB for model-free tests. Set
|
|
`DS4_QWEN38_ARTIFACTS`, `DS4_QWEN_VISION_IMAGE`, `DS4_ISSUE107_DIR`,
|
|
`DS4_ISSUE107_BAD_CHECKPOINT`, `DS4_ISSUE107_ORACLE_DIR`, and
|
|
`DS4_ISSUE107_INDEXER` to the corresponding files/directories.
|
|
|
|
Owned tests: `issue107_vision_product`,
|
|
`issue107_real_geometry_query_and_block_start_positions`, and the
|
|
`vision_sparse_qsa` case in `issue103_vision_position_receipts`.
|
|
|
|
To refresh just the affected whole-attention receipts, set `PYTHONPATH` to the
|
|
pinned checkout and run `qsa_attention_module_fixtures(profile_filter=7)` and
|
|
`qsa_attention_module_fixtures(profile_filter=9)` from
|
|
`runpy.run_path("tools/mtplx-kernel-fixtures.py")` in separate processes with
|
|
`MLX_ENABLE_TF32=1` and `0`. Replace only those profile/TF32 rows in the fixture.
|