DS4Server/docs/issue100-vision/README.md

Issue #100: DeepSeek V4 Flash Vision Experimental

The catalog ID is deepseek-v4-flash-vision-exp. The main model and image encoder are required together, including for text-only use; DSpark uses the separate Vision-Exp support file. Model Manager verifies both hashes and GGUF metadata before making the model available. Existing 0731 files remain separate. The session database migration accepts the new model without changing existing session IDs or content.

Reference and architecture

The behavioral oracle is DS4 commit 0aaea5a238fb41a35106a551e73c8409dfb751ac. This is existing DS4 functionality; MTPLX is the Qwen oracle and has no corresponding DeepSeek implementation. The three GGUF artifacts are pinned by size and SHA256 in src/model.rs, from antirez/deepseek-v4-gguf revision f71f23d552d664e523b422157b2befbf74040380. Their required source checkpoint is deepseek-ai/DeepSeek-V4-Flash-Vision-Exp revision e46e16bf6035c6f317eb2ac7458eb0362926d402.

All image decoding, resizing, patch construction, encoder graph execution, prompt layout, attention-mask construction, model loading, and cache ownership are Rust. Objective-C only binds Metal buffers and dispatches GPU primitives. No DS4 C application or Python model implementation is linked, compiled into, or invoked by the product. The independent Python comparison in tools/issue100-vision-reference.py is test-only.

Requirement / DS4 behavior	Implementation
Distinct checkpoint and drafter	Three catalog artifacts; strict revision, variant, tensor inventory, and RMS epsilon checks; wrong drafter/encoder rejection
Image preprocessing	EXIF orientation; decoded RGB identity; DS4 bicubic resize, centered padding, 14-pixel channel-major patches, minimum pixels and 381-token pre-alignment budget
Vision tower	32 Rust-dispatched ViT blocks, 2-D RoPE, DS4 BF16 rounding points, 3x3 aligner reorder, GELU projection to 4096 dimensions
Image prompt	Position-dependent leading pads, paired-row permutation, newline/pad/end sentinel vectors, synthetic token kinds, whole-image prefill chunks
Visual language-model execution	Image embeddings broadcast to all hyperconnection streams, bidirectional image-window attention, causal compressed keys with indexer masks, visual MoE routing biases and ordinary text hash routing
Continued conversations	Historical image identity and positions retained with the exact generated token frontier; new images laid out at their actual appended positions
Cache safety	Decoded-pixel fingerprints, position/count checks, changed/reordered-image invalidation, resident-session ownership, checkpoint serialization and validation
App / API use	Shared model selector, model manager, image/tool-result input, API model ID, reasoning modes, statistics, configuration, and evaluator support

DeepSeek image identity in DS4Server's disk checkpoints is a DS4Server storage extension. It validates every serialized image span against the synthetic token stream before installing the restored state. Other model checkpoint formats and payloads are unchanged.

Numerical and regression evidence

The Rust encoder was reviewed operation by operation against ds4_gpu_deepseek4_vision_encode. The reused BF16 matrix bridge is byte-identical at the function level (SHA256 621818731c2d43989a8fd08f186d1054ecdb49a91cb1453591f07bc2613d3fef). metal/glm53_bf16.metal is byte-identical to that DS4 revision (SHA256 27d93dcfe81b09f93ed65cd986aaa0b9c1ca1df7fc43e58c15da533284d50db5). The shared vision kernels used here also match; the reference's unrelated GLM merger-layernorm barrier change is outside the DeepSeek graph.

The independent published DeepSeek PyTorch encoder comparison on the fixed red square produces 100x4096 aligned values, cosine similarity 0.99500793, RMSE 0.00568503, and maximum absolute difference 0.10229492. This is not a bitwise DS4 oracle comparison: the published Python graph uses BF16 module operations, while DS4 accepts FP32 patches and specifies its own BF16 rounding boundaries. DS4's operation order and rounding are retained in the Rust graph.

Focused checks include DS4's published image-layout and attention-bound vectors, image-size limits, real sidecar tensor validation, wrong-checkpoint rejection, all DeepSeek reasoning modes in image continuations, mixed text/image routing with exact selected experts and analytical normalized weights, and raw/mixed attention over a wrapped KV ring with independently computed visible-key means. The existing real-model 0731 resident-session and GLM 5.3 verifier tests pass.

Reproducing the checks

Download the three catalog files at the artifact revision above into local-eval-results/issue100-artifacts/. The ignored issue100_managed_intake test hard-links these into local-eval-results/issue100-managed/, runs the real SHA256/intake pipeline, and writes normal verification markers only after success.

cargo test --all-features issue100
cargo test --all-features issue100_checkpoint_metadata -- --ignored --nocapture
cargo test --all-features issue100_visual_text -- --ignored --nocapture
cargo test --all-features issue100_managed_intake -- --ignored --nocapture

Run Metal tests serially in separate processes under test-supervisor, with the child output redirected to a separate log. Copy the compiled test binary outside target/ before starting a longer test. The encoder accepts DS4_ISSUE100_IMAGE and DS4_ISSUE100_ENCODER_OUTPUT (dump prefix); its filter is issue100_encoder_real_weights. The small primitive filter is issue100_visual_routing_and_attention. The production generation/cache test filter is issue100_full_model_images_cache_and_dspark and uses the committed PNG fixtures here. DS4_ISSUE100_KEEP_VISION=1 selects persistent encoder residency; DS4_ISSUE100_SSD=1 selects a 4096-expert SSD cache.

The full-model test requires the intake test first and checks real image answers, target-only/DSpark agreement with actual drafting, warm/disk continuation, same-token-layout image replacement, multiple images, tool images, and text-only use after image requests. All three isolated runs passed on the M5 Max, using the real managed artifacts after successful SHA256 and metadata intake.

Final isolated validation

Runtime mode	Result	Duration	DSpark drafted / accepted
Resident, transient encoder	Pass	15.10 s	15 / 10
Resident, persistent encoder	Pass	15.94 s	15 / 10
SSD, 4096-expert cache and preload	Pass	31.46 s	30 / 0

Each fresh process checked red-square and blue-circle recognition, image order, new user and tool images, changed images with identical synthetic prompt tokens, warm and disk-restored continuation, exact checkpoint logits, and text-only use after images. DSpark output matched target-only output within each runtime mode; the resident and SSD answers use different wording, so these receipts do not claim token equivalence between storage modes. Zero accepted drafts in the SSD fixture is a measured result, not a disabled drafter.

The SSD log retains 129 existing mapped-range diagnostics (three per layer) from the GPU-copy hotlist preload attempt. The existing Rust loader then successfully seeds through the host-loading primitive after finishing GPU commands. Both preload primitives exist in the DS4 reference; this change retains the existing DS4Server fallback. These are not image-encoder failures. Resource logs are process accounting and do not measure total file-backed GPU residency.

All commit gates passed: rustfmt, Clippy with warnings denied, signed app bundle, and all-feature tests (323 application tests plus eight tool/supervisor tests; 247 hardware/artifact-dependent tests are ignored by that generic suite). The relevant ignored tests were run explicitly as documented above. The bundle contains the new Metal kernel and its DS4 license. Logs alongside this document record intake, encoder/primitive checks, family regression controls, the three full-model runs, and the commit gates.