Files

Issue 107: image-aware sparse QSA

Qwen vision now retains the indexer's sparse selection. The indexer receives the request's position table and delta, rotates queries row by row and pooled keys at each block's first token, and uses the eager preparation route selected by pinned MTPLX 9882703f3105363ddc37eca9f97aa09a1d387112. Its compiled/fused preparation is ineligible with image positions; the existing sparse selectors/consumers remain eligible. Budgets, thresholds, text preparation and attention projection RoPE are unchanged. Issue 108 still owns the attention projection's table-straddling case.

The indexer's new helper implements _vision_position_cos_sin using the native array operations, including I32 positions, clipped table gather, per-row selection, axis gather, trigonometry and YaRN amplitude. Signed I32 Gather reuses the existing U32 payload-copy kernel: it performs no value arithmetic or float conversion. Tests include signed extrema on both front-axis and general gather routes.

Image identities now include a numerical policy: 1 for image-aware sparse QSA, 0 for the old dense policy (also MTPLX's explicit diagnostic rollback flag). Missing fields deserialize as 0. Old vision checkpoints are rejected before tensor materialization, obsolete live/resident owners are discarded, and stale owners cannot save a checkpoint. Identical image bytes and token spans do not bypass this policy. Text-only identities remain empty and compatible. The V1 signature rejection delivered in issue 106 remains in force; this issue also rejects affected V2 vision files without invalidating unaffected V2 text files.

Request prefix matching additionally rejects a restore ending strictly inside an image span, matching MTPLX's image-safe restore boundary. Complete image spans and preceding text prefixes remain eligible. Existing image digests continue to distinguish different pixels with identical image-pad token IDs.

Acceptance and review

  • The owned vision_sparse_qsa regression now requires equal sparse output for text and a vision context with equal text positions. Separate issue 108/109 diagnostic assertions remain unchanged.
  • Real indexer geometry: four heads, dim128, ratio4, top-k512, 2,072 raw key rows, 518 pooled blocks. Query rows and block starts cross a 2,060-row position table. Table, equal-axis, delta-only and empty-table cases match pinned MTPLX exactly in F32 and BF16; independent NumPy math verifies the F32 values. Normal fused selectors remain active while compiled preparation is correctly bypassed.
  • A real 2,095-token, two-image request exceeds the sparse budget. Corrected final model logits match MTPLX byte-for-byte: SHA256 8b97664316bec6113c85b7dfca9b09967391f019ea1064b83d81a3d9d5191354. Pre-fix logits differ (a79aa02a98542adeea636f22c85fc5f929308dc5dc75658d4072e6f275d22abe).
  • The actual pre-fix dense-vision checkpoint is 412,628,836 bytes, SHA256 7ddc663d4f337ff272000a7237dac05a224262bf25bc88257e784547b2056326. Tests reject its normal disk load and import its original tensor payload only through a test helper to verify stale live/resident rejection and save refusal.
  • Product tests cover unchanged images, identical complete-state restore/reopen and follow-up, text edits before/between/after two images, different pixels with identical pad IDs, cancellation/resend, a prefix ending inside an image, and unaffected text checkpoints. Comparisons cover all serialized tensors and state metadata, not only response text.
  • DS4/GLM do not call the modified Qwen modules. Their sampling, RNG, vision and checkpoint formats are unchanged. This is MTPLX behavior plus DS4Server's application-specific checkpoint policy integration.
  • The 12 affected whole-attention fixture rows were regenerated from the pinned oracle (profiles 7 and 9, TF32 on/off). All 56 fixture rows pass with both lazy and prepared inputs, including output/state hashes, consumer routes, compiled core lifecycle and counters. Existing text-only indexer route fixtures pass.

All four commit gates passed. The initial general test run hit an unrelated dev_brain fixture whose temporary short Git hash was all digits and therefore parsed as a YAML integer. Its focused retry and the subsequent full suite passed; no unrelated source change was included. checks.json records both outcomes.

The tests and receipts are correctness evidence, not performance measurements. See checks.json and the accompanying JSONL files for actual run outcomes.

Reproduction

Use the same unchanged Bare-Speed artifact and environment documented in ../issue104-ar/environment.json, including the pinned vision tower. The input image is local-eval-results/qwen-vision-input/image.png (1024 square pixels), repeated twice; tests generate an independent 1024-square blue control.

An isolated checkout of parent a8cad88 plus baseline-capture.patch regenerates the actual old snapshot with issue107_vision_product. Set DS4_ISSUE107_DIR to its output directory. Do not replace a new file's header to fake old state.

Generate fresh reference results in the isolated MLX 0.32.2 environment:

python tools/issue107-model-reference.py PINNED_MTPLX MODEL IMAGE ORACLE_DIR
python tools/issue107-indexer-reference.py PINNED_MTPLX INDEXER_DIR

Build native tests with cargo test --all-features --bin ds4-server --no-run. Run ignored tests one at a time via test-supervisor MEMORY_MIB 60 120 --command TEST_EXECUTABLE TEST_NAME --ignored --nocapture --test-threads=1. Use 114688 MiB for the product test and 4096 MiB for model-free tests. Set DS4_QWEN38_ARTIFACTS, DS4_QWEN_VISION_IMAGE, DS4_ISSUE107_DIR, DS4_ISSUE107_BAD_CHECKPOINT, DS4_ISSUE107_ORACLE_DIR, and DS4_ISSUE107_INDEXER to the corresponding files/directories.

Owned tests: issue107_vision_product, issue107_real_geometry_query_and_block_start_positions, and the vision_sparse_qsa case in issue103_vision_position_receipts.

To refresh just the affected whole-attention receipts, set PYTHONPATH to the pinned checkout and run qsa_attention_module_fixtures(profile_filter=7) and qsa_attention_module_fixtures(profile_filter=9) from runpy.run_path("tools/mtplx-kernel-fixtures.py") in separate processes with MLX_ENABLE_TF32=1 and 0. Replace only those profile/TF32 rows in the fixture.