Support DeepSeek V4 Flash Vision Experimental
This commit is contained in:
Georg Bauer committed
1 parent
7c18e738cd
commit
9bbe12e6b3
52 files changed
+4988
-100
No files matched your search
@@ -0,0 +1,129 @@
|
||||
# Issue #100: DeepSeek V4 Flash Vision Experimental
|
||||
|
||||
The catalog ID is `deepseek-v4-flash-vision-exp`. The main model and image
|
||||
encoder are required together, including for text-only use; DSpark uses the
|
||||
separate Vision-Exp support file. Model Manager verifies both hashes and GGUF
|
||||
metadata before making the model available. Existing 0731 files remain separate.
|
||||
The session database migration accepts the new model without changing existing
|
||||
session IDs or content.
|
||||
|
||||
## Reference and architecture
|
||||
|
||||
The behavioral oracle is DS4 commit
|
||||
`0aaea5a238fb41a35106a551e73c8409dfb751ac`. This is existing DS4 functionality;
|
||||
MTPLX is the Qwen oracle and has no corresponding DeepSeek implementation.
|
||||
The three GGUF artifacts are pinned by size and SHA256 in `src/model.rs`, from
|
||||
`antirez/deepseek-v4-gguf` revision
|
||||
`f71f23d552d664e523b422157b2befbf74040380`. Their required source checkpoint is
|
||||
`deepseek-ai/DeepSeek-V4-Flash-Vision-Exp` revision
|
||||
`e46e16bf6035c6f317eb2ac7458eb0362926d402`.
|
||||
|
||||
All image decoding, resizing, patch construction, encoder graph execution,
|
||||
prompt layout, attention-mask construction, model loading, and cache ownership
|
||||
are Rust. Objective-C only binds Metal buffers and dispatches GPU primitives.
|
||||
No DS4 C application or Python model implementation is linked, compiled into,
|
||||
or invoked by the product. The independent Python comparison in
|
||||
`tools/issue100-vision-reference.py` is test-only.
|
||||
|
||||
| Requirement / DS4 behavior | Implementation |
|
||||
| --- | --- |
|
||||
| Distinct checkpoint and drafter | Three catalog artifacts; strict revision, variant, tensor inventory, and RMS epsilon checks; wrong drafter/encoder rejection |
|
||||
| Image preprocessing | EXIF orientation; decoded RGB identity; DS4 bicubic resize, centered padding, 14-pixel channel-major patches, minimum pixels and 381-token pre-alignment budget |
|
||||
| Vision tower | 32 Rust-dispatched ViT blocks, 2-D RoPE, DS4 BF16 rounding points, 3x3 aligner reorder, GELU projection to 4096 dimensions |
|
||||
| Image prompt | Position-dependent leading pads, paired-row permutation, newline/pad/end sentinel vectors, synthetic token kinds, whole-image prefill chunks |
|
||||
| Visual language-model execution | Image embeddings broadcast to all hyperconnection streams, bidirectional image-window attention, causal compressed keys with indexer masks, visual MoE routing biases and ordinary text hash routing |
|
||||
| Continued conversations | Historical image identity and positions retained with the exact generated token frontier; new images laid out at their actual appended positions |
|
||||
| Cache safety | Decoded-pixel fingerprints, position/count checks, changed/reordered-image invalidation, resident-session ownership, checkpoint serialization and validation |
|
||||
| App / API use | Shared model selector, model manager, image/tool-result input, API model ID, reasoning modes, statistics, configuration, and evaluator support |
|
||||
|
||||
DeepSeek image identity in DS4Server's disk checkpoints is a DS4Server storage
|
||||
extension. It validates every serialized image span against the synthetic token
|
||||
stream before installing the restored state. Other model checkpoint formats
|
||||
and payloads are unchanged.
|
||||
|
||||
## Numerical and regression evidence
|
||||
|
||||
The Rust encoder was reviewed operation by operation against
|
||||
`ds4_gpu_deepseek4_vision_encode`. The reused BF16 matrix bridge is byte-identical
|
||||
at the function level (SHA256
|
||||
`621818731c2d43989a8fd08f186d1054ecdb49a91cb1453591f07bc2613d3fef`).
|
||||
`metal/glm53_bf16.metal` is byte-identical to that DS4 revision (SHA256
|
||||
`27d93dcfe81b09f93ed65cd986aaa0b9c1ca1df7fc43e58c15da533284d50db5`).
|
||||
The shared vision kernels used here also match; the reference's unrelated GLM
|
||||
merger-layernorm barrier change is outside the DeepSeek graph.
|
||||
|
||||
The independent published DeepSeek PyTorch encoder comparison on the fixed red
|
||||
square produces 100x4096 aligned values, cosine similarity 0.99500793, RMSE
|
||||
0.00568503, and maximum absolute difference 0.10229492. This is not a bitwise
|
||||
DS4 oracle comparison: the published Python graph uses BF16 module operations,
|
||||
while DS4 accepts FP32 patches and specifies its own BF16 rounding boundaries.
|
||||
DS4's operation order and rounding are retained in the Rust graph.
|
||||
|
||||
Focused checks include DS4's published image-layout and attention-bound vectors,
|
||||
image-size limits, real sidecar tensor validation, wrong-checkpoint rejection,
|
||||
all DeepSeek reasoning modes in image continuations, mixed text/image routing
|
||||
with exact selected experts and analytical normalized weights, and raw/mixed
|
||||
attention over a wrapped KV ring with independently computed visible-key means.
|
||||
The existing real-model 0731 resident-session and GLM 5.3 verifier tests pass.
|
||||
|
||||
## Reproducing the checks
|
||||
|
||||
Download the three catalog files at the artifact revision above into
|
||||
`local-eval-results/issue100-artifacts/`. The ignored `issue100_managed_intake`
|
||||
test hard-links these into `local-eval-results/issue100-managed/`, runs the real
|
||||
SHA256/intake pipeline, and writes normal verification markers only after success.
|
||||
|
||||
```sh
|
||||
cargo test --all-features issue100
|
||||
cargo test --all-features issue100_checkpoint_metadata -- --ignored --nocapture
|
||||
cargo test --all-features issue100_visual_text -- --ignored --nocapture
|
||||
cargo test --all-features issue100_managed_intake -- --ignored --nocapture
|
||||
```
|
||||
|
||||
Run Metal tests serially in separate processes under `test-supervisor`, with the
|
||||
child output redirected to a separate log. Copy the compiled test binary outside
|
||||
`target/` before starting a longer test. The encoder accepts
|
||||
`DS4_ISSUE100_IMAGE` and `DS4_ISSUE100_ENCODER_OUTPUT` (dump prefix); its filter is
|
||||
`issue100_encoder_real_weights`. The small primitive filter is
|
||||
`issue100_visual_routing_and_attention`. The production generation/cache test
|
||||
filter is `issue100_full_model_images_cache_and_dspark` and uses the committed
|
||||
PNG fixtures here. `DS4_ISSUE100_KEEP_VISION=1` selects persistent encoder
|
||||
residency; `DS4_ISSUE100_SSD=1` selects a 4096-expert SSD cache.
|
||||
|
||||
The full-model test requires the intake test first and checks real image answers,
|
||||
target-only/DSpark agreement with actual drafting, warm/disk continuation,
|
||||
same-token-layout image replacement, multiple images, tool images, and text-only
|
||||
use after image requests. All three isolated runs passed on the M5 Max, using
|
||||
the real managed artifacts after successful SHA256 and metadata intake.
|
||||
|
||||
|
||||
## Final isolated validation
|
||||
|
||||
| Runtime mode | Result | Duration | DSpark drafted / accepted |
|
||||
| --- | --- | --- | --- |
|
||||
| Resident, transient encoder | Pass | 15.10 s | 15 / 10 |
|
||||
| Resident, persistent encoder | Pass | 15.94 s | 15 / 10 |
|
||||
| SSD, 4096-expert cache and preload | Pass | 31.46 s | 30 / 0 |
|
||||
|
||||
Each fresh process checked red-square and blue-circle recognition, image order,
|
||||
new user and tool images, changed images with identical synthetic prompt tokens,
|
||||
warm and disk-restored continuation, exact checkpoint logits, and text-only use
|
||||
after images. DSpark output matched target-only output within each runtime mode;
|
||||
the resident and SSD answers use different wording, so these receipts do not
|
||||
claim token equivalence between storage modes. Zero accepted drafts in the SSD
|
||||
fixture is a measured result, not a disabled drafter.
|
||||
|
||||
The SSD log retains 129 existing mapped-range diagnostics (three per layer) from
|
||||
the GPU-copy hotlist preload attempt. The existing Rust loader then successfully
|
||||
seeds through the host-loading primitive after finishing GPU commands. Both
|
||||
preload primitives exist in the DS4 reference; this change retains the existing
|
||||
DS4Server fallback. These are not image-encoder failures. Resource logs are
|
||||
process accounting and do not measure total file-backed GPU residency.
|
||||
|
||||
All commit gates passed: rustfmt, Clippy with warnings denied, signed app bundle,
|
||||
and all-feature tests (323 application tests plus eight tool/supervisor tests;
|
||||
247 hardware/artifact-dependent tests are ignored by that generic suite).
|
||||
The relevant ignored tests were run explicitly as documented above. The bundle
|
||||
contains the new Metal kernel and its DS4 license. Logs alongside this document
|
||||
record intake, encoder/primitive checks, family regression controls, the three
|
||||
full-model runs, and the commit gates.
|
||||
Reference in new issue
Block a user