130 lines
7.9 KiB
Markdown
130 lines
7.9 KiB
Markdown
# Issue #100: DeepSeek V4 Flash Vision Experimental
|
|
|
|
The catalog ID is `deepseek-v4-flash-vision-exp`. The main model and image
|
|
encoder are required together, including for text-only use; DSpark uses the
|
|
separate Vision-Exp support file. Model Manager verifies both hashes and GGUF
|
|
metadata before making the model available. Existing 0731 files remain separate.
|
|
The session database migration accepts the new model without changing existing
|
|
session IDs or content.
|
|
|
|
## Reference and architecture
|
|
|
|
The behavioral oracle is DS4 commit
|
|
`0aaea5a238fb41a35106a551e73c8409dfb751ac`. This is existing DS4 functionality;
|
|
MTPLX is the Qwen oracle and has no corresponding DeepSeek implementation.
|
|
The three GGUF artifacts are pinned by size and SHA256 in `src/model.rs`, from
|
|
`antirez/deepseek-v4-gguf` revision
|
|
`f71f23d552d664e523b422157b2befbf74040380`. Their required source checkpoint is
|
|
`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` revision
|
|
`e46e16bf6035c6f317eb2ac7458eb0362926d402`.
|
|
|
|
All image decoding, resizing, patch construction, encoder graph execution,
|
|
prompt layout, attention-mask construction, model loading, and cache ownership
|
|
are Rust. Objective-C only binds Metal buffers and dispatches GPU primitives.
|
|
No DS4 C application or Python model implementation is linked, compiled into,
|
|
or invoked by the product. The independent Python comparison in
|
|
`tools/issue100-vision-reference.py` is test-only.
|
|
|
|
| Requirement / DS4 behavior | Implementation |
|
|
| --- | --- |
|
|
| Distinct checkpoint and drafter | Three catalog artifacts; strict revision, variant, tensor inventory, and RMS epsilon checks; wrong drafter/encoder rejection |
|
|
| Image preprocessing | EXIF orientation; decoded RGB identity; DS4 bicubic resize, centered padding, 14-pixel channel-major patches, minimum pixels and 381-token pre-alignment budget |
|
|
| Vision tower | 32 Rust-dispatched ViT blocks, 2-D RoPE, DS4 BF16 rounding points, 3x3 aligner reorder, GELU projection to 4096 dimensions |
|
|
| Image prompt | Position-dependent leading pads, paired-row permutation, newline/pad/end sentinel vectors, synthetic token kinds, whole-image prefill chunks |
|
|
| Visual language-model execution | Image embeddings broadcast to all hyperconnection streams, bidirectional image-window attention, causal compressed keys with indexer masks, visual MoE routing biases and ordinary text hash routing |
|
|
| Continued conversations | Historical image identity and positions retained with the exact generated token frontier; new images laid out at their actual appended positions |
|
|
| Cache safety | Decoded-pixel fingerprints, position/count checks, changed/reordered-image invalidation, resident-session ownership, checkpoint serialization and validation |
|
|
| App / API use | Shared model selector, model manager, image/tool-result input, API model ID, reasoning modes, statistics, configuration, and evaluator support |
|
|
|
|
DeepSeek image identity in DS4Server's disk checkpoints is a DS4Server storage
|
|
extension. It validates every serialized image span against the synthetic token
|
|
stream before installing the restored state. Other model checkpoint formats
|
|
and payloads are unchanged.
|
|
|
|
## Numerical and regression evidence
|
|
|
|
The Rust encoder was reviewed operation by operation against
|
|
`ds4_gpu_deepseek4_vision_encode`. The reused BF16 matrix bridge is byte-identical
|
|
at the function level (SHA256
|
|
`621818731c2d43989a8fd08f186d1054ecdb49a91cb1453591f07bc2613d3fef`).
|
|
`metal/glm53_bf16.metal` is byte-identical to that DS4 revision (SHA256
|
|
`27d93dcfe81b09f93ed65cd986aaa0b9c1ca1df7fc43e58c15da533284d50db5`).
|
|
The shared vision kernels used here also match; the reference's unrelated GLM
|
|
merger-layernorm barrier change is outside the DeepSeek graph.
|
|
|
|
The independent published DeepSeek PyTorch encoder comparison on the fixed red
|
|
square produces 100x4096 aligned values, cosine similarity 0.99500793, RMSE
|
|
0.00568503, and maximum absolute difference 0.10229492. This is not a bitwise
|
|
DS4 oracle comparison: the published Python graph uses BF16 module operations,
|
|
while DS4 accepts FP32 patches and specifies its own BF16 rounding boundaries.
|
|
DS4's operation order and rounding are retained in the Rust graph.
|
|
|
|
Focused checks include DS4's published image-layout and attention-bound vectors,
|
|
image-size limits, real sidecar tensor validation, wrong-checkpoint rejection,
|
|
all DeepSeek reasoning modes in image continuations, mixed text/image routing
|
|
with exact selected experts and analytical normalized weights, and raw/mixed
|
|
attention over a wrapped KV ring with independently computed visible-key means.
|
|
The existing real-model 0731 resident-session and GLM 5.3 verifier tests pass.
|
|
|
|
## Reproducing the checks
|
|
|
|
Download the three catalog files at the artifact revision above into
|
|
`local-eval-results/issue100-artifacts/`. The ignored `issue100_managed_intake`
|
|
test hard-links these into `local-eval-results/issue100-managed/`, runs the real
|
|
SHA256/intake pipeline, and writes normal verification markers only after success.
|
|
|
|
```sh
|
|
cargo test --all-features issue100
|
|
cargo test --all-features issue100_checkpoint_metadata -- --ignored --nocapture
|
|
cargo test --all-features issue100_visual_text -- --ignored --nocapture
|
|
cargo test --all-features issue100_managed_intake -- --ignored --nocapture
|
|
```
|
|
|
|
Run Metal tests serially in separate processes under `test-supervisor`, with the
|
|
child output redirected to a separate log. Copy the compiled test binary outside
|
|
`target/` before starting a longer test. The encoder accepts
|
|
`DS4_ISSUE100_IMAGE` and `DS4_ISSUE100_ENCODER_OUTPUT` (dump prefix); its filter is
|
|
`issue100_encoder_real_weights`. The small primitive filter is
|
|
`issue100_visual_routing_and_attention`. The production generation/cache test
|
|
filter is `issue100_full_model_images_cache_and_dspark` and uses the committed
|
|
PNG fixtures here. `DS4_ISSUE100_KEEP_VISION=1` selects persistent encoder
|
|
residency; `DS4_ISSUE100_SSD=1` selects a 4096-expert SSD cache.
|
|
|
|
The full-model test requires the intake test first and checks real image answers,
|
|
target-only/DSpark agreement with actual drafting, warm/disk continuation,
|
|
same-token-layout image replacement, multiple images, tool images, and text-only
|
|
use after image requests. All three isolated runs passed on the M5 Max, using
|
|
the real managed artifacts after successful SHA256 and metadata intake.
|
|
|
|
|
|
## Final isolated validation
|
|
|
|
| Runtime mode | Result | Duration | DSpark drafted / accepted |
|
|
| --- | --- | --- | --- |
|
|
| Resident, transient encoder | Pass | 15.10 s | 15 / 10 |
|
|
| Resident, persistent encoder | Pass | 15.94 s | 15 / 10 |
|
|
| SSD, 4096-expert cache and preload | Pass | 31.46 s | 30 / 0 |
|
|
|
|
Each fresh process checked red-square and blue-circle recognition, image order,
|
|
new user and tool images, changed images with identical synthetic prompt tokens,
|
|
warm and disk-restored continuation, exact checkpoint logits, and text-only use
|
|
after images. DSpark output matched target-only output within each runtime mode;
|
|
the resident and SSD answers use different wording, so these receipts do not
|
|
claim token equivalence between storage modes. Zero accepted drafts in the SSD
|
|
fixture is a measured result, not a disabled drafter.
|
|
|
|
The SSD log retains 129 existing mapped-range diagnostics (three per layer) from
|
|
the GPU-copy hotlist preload attempt. The existing Rust loader then successfully
|
|
seeds through the host-loading primitive after finishing GPU commands. Both
|
|
preload primitives exist in the DS4 reference; this change retains the existing
|
|
DS4Server fallback. These are not image-encoder failures. Resource logs are
|
|
process accounting and do not measure total file-backed GPU residency.
|
|
|
|
All commit gates passed: rustfmt, Clippy with warnings denied, signed app bundle,
|
|
and all-feature tests (323 application tests plus eight tool/supervisor tests;
|
|
247 hardware/artifact-dependent tests are ignored by that generic suite).
|
|
The relevant ignored tests were run explicitly as documented above. The bundle
|
|
contains the new Metal kernel and its DS4 license. Logs alongside this document
|
|
record intake, encoder/primitive checks, family regression controls, the three
|
|
full-model runs, and the commit gates.
|