Use the current Gemma 4 E4B Instruct model at 4-bit, with temperature 0 and thinking disabled, to describe and interrogate image content so the DeepSeek model can reason over it. When DeepSeek needs a specific fact from an image, it asks the auxiliary model a focused question and receives deterministic textual tool output.
Use MLX Swift LM through a bundled one-shot helper process rather than extending the custom DeepSeek inference engine. The helper owns the auxiliary model's independent MLX and Metal state and releases it completely when the process exits.
Gemma 4 E4B is the preferred auxiliary model. The current model family handles text, images, audio, and video, so the same sidecar boundary can later add listening and video understanding without changing model families. Current MLX Swift LM already implements the Gemma 4 VLM and processor, so DS4Server does not need a custom Gemma inferencer. Phase one remains image analysis so the memory and lifecycle design can be proven in isolation.
The model integration would be using internal tools, not a chat integration, those model runs are one-shot invocations, so no context handling - the only context is the actual file.
Memory is tight, but the baseline must keep DeepSeek's mmap, executor, and active KV cache alive while serializing the auxiliary invocation. Clean file-backed DeepSeek pages may be reclaimed by macOS while the helper is active. Explicit expert-cache release and page-discard advice are a measured fallback, not a prerequisite.
The existing agent tool loop invokes the auxiliary model and inserts its bounded textual result before DeepSeek continues. The same loaded DeepSeek generator and KV context remain intact; the normal path does not reconstruct the model.
The existing generation queue serializes DeepSeek and auxiliary work. The server remains listening while an image is analyzed, and concurrent requests queue rather than overlapping GPU inference or forcing a server restart.
Workable implementation plan (2026-08-30)
Decision and scope
This has no corresponding DS4 functionality; it is a new DS4Server feature.
Implement phase one as an agent-only image-analysis tool backed by the current 4-bit Gemma 4 E4B Instruct model (mlx-community/gemma-4-e4b-it-4bit). Keep the loaded DeepSeek generator and its KV cache alive. Run the auxiliary model in a bundled, one-shot Swift/MLX helper process and let that process exit after each tool call. Do not add a second model to DS4Server's current native Metal executor, and do not unload/reconstruct DeepSeek by default.
Gemma 4 E4B is the required default, not a compatibility option. Google positions E4B as the stronger small on-device member of the current Gemma 4 family: 4.5B effective parameters (8B including per-layer embeddings), a 128K context window, image and audio input, variable-resolution vision, OCR, charts, UI understanding, and configurable thinking. The exact 4-bit E4B checkpoint is registered by the current released MLX Swift LM VLM factory.
Pin MLX Swift LM 3.31.4 or a newer release that has passed the Gemma 4 image fixture and memory gates. Do not retain any older model family as a silent fallback.
Raise DS4Server's global macOS deployment target to the minimum required by the pinned MLX/Gemma implementation instead of feature-gating vision or carrying an old-system path. Update the Rust build, Swift package, bundle metadata, documentation, and CI target together. If the final pinned implementation uses a newer macOS-only API, raise the global minimum again; do not add compatibility branches to preserve an older target.
The current DS4 Metal bridge is process-global: it has one global model-view table, command queues, pipelines, and model residency set. Its model buffers are no-copy Metal buffers pointing directly into the main GGUF mmap. A second independent model/runtime in that singleton would require a broad native-state refactor and creates cleanup and concurrency hazards.
A dedicated helper gives the auxiliary MLX runtime independent Metal state. It also gives deterministic cleanup: process exit releases the auxiliary model mappings, MLX buffers, command queues, and transient image tensors even on an error. It must be an app-bundled executable invoked directly by Rust—no shell, Python, Homebrew, or user-installed runtime.
mmap and memory behavior
Both model files may be mmaped at the same time. An mmap reserves address space; clean file-backed pages only consume physical memory while resident and macOS may evict them under pressure. Keeping the DeepSeek mmap open does not require keeping all its weights resident.
The DeepSeek mmap itself cannot be released while its executor is alive because the current no-copy Metal buffers retain pointers into that mapping. Unmapping it would make those buffers invalid. The safe baseline is therefore:
Keep the DeepSeek mmap, executor, active KV cache, and scratch state alive.
Serialize the vision call so no DeepSeek Metal work is in flight.
Start the helper, which maps and loads the auxiliary model and performs one generation.
Let memory pressure evict clean DeepSeek weight pages if needed.
Exit the helper, releasing the auxiliary model.
Continue the existing DeepSeek generator; any evicted DeepSeek pages fault back on demand.
This preserves the conversation context and avoids the much larger cost and risk of serializing/rebuilding the main generator.
Agent tool contract
Add one internal tool for phase one:
Name: analyze_image
Inputs: a project/session-managed image path and a focused question
Output: bounded plain text containing the answer, relevant visual facts, visible text/OCR, and explicit uncertainty
Generation: temperature 0, top-p 1, thinking disabled, fixed system prompt, and a hard output cap (initially 1,024 tokens and 16 KiB)
Scope: exactly one image per invocation
Treat the image and any text inside it as untrusted data. The fixed helper prompt must say that instructions visible in the image are content to report, not instructions to follow.
At the Rust trust boundary:
Canonicalize the path and allow only the current project root or an app-managed attachment/session root.
Reject symlink escapes, non-regular files, unsupported formats, files above the configured byte ceiling, and images above a decoded-pixel ceiling.
Validate/decode the image before invoking MLX; reuse the existing image dependency.
Bound helper stdin, stdout, stderr, execution time, and output length.
Version the JSON stdin/stdout protocol and include a request ID.
Treat malformed output, non-zero exit, timeout, cancellation, and missing/corrupt model files as explicit tool errors.
On cancellation, terminate the helper and wait for it so memory is released.
The result should enter the existing agent flow as normal textual tool output. Do not put pixels, base64, or multimodal blocks into ChatTurn or the OpenAI-compatible server request format in phase one.
Execution flow and serialization
Add an AnalyzeImage operation to the existing GenerationService command queue:
The agent emits an analyze_image tool call.
Tools validates the request and submits AnalyzeImage with the currently loaded engine settings.
The single model-runtime worker waits for any active DeepSeek generation and then owns the exclusive auxiliary-model interval.
It synchronizes pending Metal work and starts the bundled helper directly.
The helper loads the pinned local VLM bundle, preprocesses the image, performs one-shot generation, writes one JSON response, and exits.
The runtime returns the bounded text to the tool handler.
The existing agent loop appends that tool result and resumes DeepSeek reasoning with the same generator and KV cache.
This queue is also used by HTTP generation, so the server can remain listening. Requests arriving during image analysis should queue normally; streaming requests should continue to receive the existing prefill/keepalive behavior. Surface a runtime activity state such as “Analyzing image…” rather than disabling the server. Never permit DeepSeek and auxiliary Metal generation concurrently.
Runtime helper and packaging
Add a small Swift executable package under native/vision:
Pin MLX Swift LM and transitive versions in Package.resolved.
Use the public MLXVLM/MLXLMCommon APIs and the registered Gemma 4 model and processor.
Accept exactly one JSON request on stdin and write exactly one JSON response on stdout.
Load only from the verified local model directory; never download implicitly.
Disable thinking and sampling, enforce the token cap, and report load/prefill/generation timing.
Build it in release mode from build.rs/the existing bundle path, embed it in the app, and codesign it with the app.
Use the same raised global macOS deployment target as DS4Server; there is no older-system feature gate.
Do not add a Rust abstraction with multiple backends or a custom VLM inference engine. There is one tested backend in phase one.
Model catalog integration
Auxiliary assets are a model bundle, not a chat ModelChoice. Add a separate AuxiliaryModelId and a pinned manifest for the required config, tokenizer, processor, and safetensor files.
Reuse the existing model transfer primitives for download resume, byte progress, SHA-256 verification, deletion, and error reporting. The manifest must pin an immutable repository revision plus the expected size and hash of every file. Store the completed bundle in the normal model catalog (for example, models/vision-gemma-4-e4b/) and display aggregate stored/expected size and verification state in the model manager.
There must be no mutable “main” reference, implicit helper download, or deletion after a call. Once downloaded and verified, the auxiliary model remains available for future calls and experiments until the user explicitly removes it.
Conditional low-memory pause path
Start without explicit DeepSeek offloading and measure it. macOS already has the ability to reclaim clean mmap-backed weight pages, and adding manual lifecycle machinery before evidence would make the riskiest part of the engine more complex.
If the acceptance measurements show that the lowest supported memory configuration cannot run the helper reliably, add one scoped Generator::pause_for_auxiliary() guard:
Synchronize all DeepSeek command buffers.
Release the SSD-streamed expert cache and any inactive resident-session allocations.
Detach/end the model residency set.
Apply POSIX_MADV_DONTNEED only to clean main/support weight ranges while retaining the mappings and no-copy buffers.
Keep active KV cache, scratch/logit state needed for continuation, and generator metadata alive.
On guard drop—success, tool error, timeout, cancellation, or panic—recreate the residency set and restore a usable executor.
Do not eagerly warm the full GGUF; allow the next DeepSeek pass to fault the required pages back.
Never call full ds4_gpu_cleanup, unmap the GGUF, or leave Metal buffers pointing at released virtual memory. Full generator checkpoint/drop/reopen/restore is only a final fallback if measured hardware evidence proves that retaining KV plus mappings cannot fit.
Expected implementation touch points
native/vision/: pinned Swift helper package and protocol implementation
build.rs, bundle/build metadata: build, embed, and sign the helper
src/vision.rs: request/response protocol, validation, direct child-process lifecycle
Metal bridge only if the measured conditional pause guard is required
Verification and acceptance criteria
Functional
Analyze real PNG, JPEG, GIF, and WebP fixtures covering photos, screenshots, charts, documents/OCR, and spatial questions.
Repeated temperature-zero runs produce materially identical answers.
A prompt-injection image is described but its embedded instructions are not followed.
Invalid MIME/header, oversize bytes, excessive decoded pixels, symlink escape, missing file, corrupt model, and missing helper all produce clear bounded errors.
Cancellation during model load and generation terminates/reaps the helper and leaves DeepSeek immediately usable.
Unit-test the JSON protocol and path limits with a fake helper; add an app-level test using the real bundled helper/model where hardware permits.
Context and correctness
The same loaded DeepSeek generator and active conversation KV cache continue after the tool call.
The next DeepSeek result matches the behavior of inserting the same description as ordinary tool text; no hidden context reset or token-accounting change.
DS4 token processing, KV accounting, and generation behavior remain unchanged outside the new tool result.
No image bytes or auxiliary prompt tokens are counted as DeepSeek context.
Memory and concurrency
Measure on the target M5 hardware and the lowest supported RAM configuration, with 4K, 32K, and maximum practical active contexts, and with both resident and SSD-streamed DeepSeek modes (including DSpark):
physical footprint and memory pressure before, during, and after the helper;
helper peak footprint and whether it returns after exit;
latency to first auxiliary token and total tool time;
first-token and steady-state DeepSeek speed immediately after the tool;
cold-page/refault penalty on the first resumed DeepSeek pass;
unchanged KV checksum/continuation across the tool interval.
Also verify:
a simultaneous HTTP request queues while the server remains alive;
no two GPU inference paths overlap;
helper crash, timeout, and cancellation all restore normal DeepSeek generation;
there is no hidden model re-download;
no main generator reconstruction occurs on the normal path.
Add the conditional pause guard only if these results demonstrate a real memory failure or unacceptable pressure, and record the before/after measurements in the implementation issue comment.
Build and distribution
A clean release build embeds and signs the helper.
The installed app works without Python, Homebrew, Xcode, or network access once the auxiliary bundle is present.
The app, helper, bundle metadata, and CI all declare the same raised minimum macOS version; no runtime old-system branch remains.
Model catalog deletion removes only the explicitly selected auxiliary bundle.
Existing release, Clippy, formatting, bundle, and all-feature test gates stay green.
Deferred follow-ups
Keep these out of the first implementation:
direct multimodal OpenAI/API request blocks or chat composer attachments;
persistent auxiliary conversation/KV state;
a resident helper/model cache;
a custom DS4 Metal VLM graph;
audio and video.
After the image path is proven, video and audio can reuse the same Gemma 4 E4B tool/helper boundary. Each should remain a separate follow-up with bounded media preprocessing and real MLX Swift processor fixtures before it is enabled.
Use the current Gemma 4 E4B Instruct model at 4-bit, with temperature 0 and thinking disabled, to describe and interrogate image content so the DeepSeek model can reason over it. When DeepSeek needs a specific fact from an image, it asks the auxiliary model a focused question and receives deterministic textual tool output.
Use MLX Swift LM through a bundled one-shot helper process rather than extending the custom DeepSeek inference engine. The helper owns the auxiliary model's independent MLX and Metal state and releases it completely when the process exits.
Gemma 4 E4B is the preferred auxiliary model. The current model family handles text, images, audio, and video, so the same sidecar boundary can later add listening and video understanding without changing model families. Current MLX Swift LM already implements the Gemma 4 VLM and processor, so DS4Server does not need a custom Gemma inferencer. Phase one remains image analysis so the memory and lifecycle design can be proven in isolation.
The model integration would be using internal tools, not a chat integration, those model runs are one-shot invocations, so no context handling - the only context is the actual file.
Memory is tight, but the baseline must keep DeepSeek's mmap, executor, and active KV cache alive while serializing the auxiliary invocation. Clean file-backed DeepSeek pages may be reclaimed by macOS while the helper is active. Explicit expert-cache release and page-discard advice are a measured fallback, not a prerequisite.
The existing agent tool loop invokes the auxiliary model and inserts its bounded textual result before DeepSeek continues. The same loaded DeepSeek generator and KV context remain intact; the normal path does not reconstruct the model.
The existing generation queue serializes DeepSeek and auxiliary work. The server remains listening while an image is analyzed, and concurrent requests queue rather than overlapping GPU inference or forcing a server restart.
---
## Workable implementation plan (2026-08-30)
### Decision and scope
This has no corresponding DS4 functionality; it is a new DS4Server feature.
Implement phase one as an agent-only image-analysis tool backed by the current 4-bit Gemma 4 E4B Instruct model (`mlx-community/gemma-4-e4b-it-4bit`). Keep the loaded DeepSeek generator and its KV cache alive. Run the auxiliary model in a bundled, one-shot Swift/MLX helper process and let that process exit after each tool call. Do not add a second model to DS4Server's current native Metal executor, and do not unload/reconstruct DeepSeek by default.
Gemma 4 E4B is the required default, not a compatibility option. Google positions E4B as the stronger small on-device member of the current Gemma 4 family: 4.5B effective parameters (8B including per-layer embeddings), a 128K context window, image and audio input, variable-resolution vision, OCR, charts, UI understanding, and configurable thinking. The exact 4-bit E4B checkpoint is registered by the current released MLX Swift LM VLM factory.
- https://huggingface.co/google/gemma-4-31B
- https://github.com/ml-explore/mlx-swift-lm/blob/3.31.4/Libraries/MLXVLM/VLMModelFactory.swift
- https://github.com/ml-explore/mlx-swift-lm/releases/tag/3.31.4
Pin MLX Swift LM 3.31.4 or a newer release that has passed the Gemma 4 image fixture and memory gates. Do not retain any older model family as a silent fallback.
Raise DS4Server's global macOS deployment target to the minimum required by the pinned MLX/Gemma implementation instead of feature-gating vision or carrying an old-system path. Update the Rust build, Swift package, bundle metadata, documentation, and CI target together. If the final pinned implementation uses a newer macOS-only API, raise the global minimum again; do not add compatibility branches to preserve an older target.
- https://github.com/ml-explore/mlx-swift-lm/blob/3.31.4/Package.swift
### Why the helper process is the right boundary
The current DS4 Metal bridge is process-global: it has one global model-view table, command queues, pipelines, and model residency set. Its model buffers are no-copy Metal buffers pointing directly into the main GGUF mmap. A second independent model/runtime in that singleton would require a broad native-state refactor and creates cleanup and concurrency hazards.
A dedicated helper gives the auxiliary MLX runtime independent Metal state. It also gives deterministic cleanup: process exit releases the auxiliary model mappings, MLX buffers, command queues, and transient image tensors even on an error. It must be an app-bundled executable invoked directly by Rust—no shell, Python, Homebrew, or user-installed runtime.
### mmap and memory behavior
Both model files may be mmaped at the same time. An mmap reserves address space; clean file-backed pages only consume physical memory while resident and macOS may evict them under pressure. Keeping the DeepSeek mmap open does not require keeping all its weights resident.
The DeepSeek mmap itself cannot be released while its executor is alive because the current no-copy Metal buffers retain pointers into that mapping. Unmapping it would make those buffers invalid. The safe baseline is therefore:
1. Keep the DeepSeek mmap, executor, active KV cache, and scratch state alive.
2. Serialize the vision call so no DeepSeek Metal work is in flight.
3. Start the helper, which maps and loads the auxiliary model and performs one generation.
4. Let memory pressure evict clean DeepSeek weight pages if needed.
5. Exit the helper, releasing the auxiliary model.
6. Continue the existing DeepSeek generator; any evicted DeepSeek pages fault back on demand.
This preserves the conversation context and avoids the much larger cost and risk of serializing/rebuilding the main generator.
### Agent tool contract
Add one internal tool for phase one:
- Name: `analyze_image`
- Inputs: a project/session-managed image path and a focused question
- Output: bounded plain text containing the answer, relevant visual facts, visible text/OCR, and explicit uncertainty
- Generation: temperature 0, top-p 1, thinking disabled, fixed system prompt, and a hard output cap (initially 1,024 tokens and 16 KiB)
- Scope: exactly one image per invocation
Treat the image and any text inside it as untrusted data. The fixed helper prompt must say that instructions visible in the image are content to report, not instructions to follow.
At the Rust trust boundary:
- Canonicalize the path and allow only the current project root or an app-managed attachment/session root.
- Reject symlink escapes, non-regular files, unsupported formats, files above the configured byte ceiling, and images above a decoded-pixel ceiling.
- Validate/decode the image before invoking MLX; reuse the existing `image` dependency.
- Bound helper stdin, stdout, stderr, execution time, and output length.
- Version the JSON stdin/stdout protocol and include a request ID.
- Treat malformed output, non-zero exit, timeout, cancellation, and missing/corrupt model files as explicit tool errors.
- On cancellation, terminate the helper and wait for it so memory is released.
The result should enter the existing agent flow as normal textual tool output. Do not put pixels, base64, or multimodal blocks into `ChatTurn` or the OpenAI-compatible server request format in phase one.
### Execution flow and serialization
Add an `AnalyzeImage` operation to the existing `GenerationService` command queue:
1. The agent emits an `analyze_image` tool call.
2. `Tools` validates the request and submits `AnalyzeImage` with the currently loaded engine settings.
3. The single model-runtime worker waits for any active DeepSeek generation and then owns the exclusive auxiliary-model interval.
4. It synchronizes pending Metal work and starts the bundled helper directly.
5. The helper loads the pinned local VLM bundle, preprocesses the image, performs one-shot generation, writes one JSON response, and exits.
6. The runtime returns the bounded text to the tool handler.
7. The existing agent loop appends that tool result and resumes DeepSeek reasoning with the same generator and KV cache.
This queue is also used by HTTP generation, so the server can remain listening. Requests arriving during image analysis should queue normally; streaming requests should continue to receive the existing prefill/keepalive behavior. Surface a runtime activity state such as “Analyzing image…” rather than disabling the server. Never permit DeepSeek and auxiliary Metal generation concurrently.
### Runtime helper and packaging
Add a small Swift executable package under `native/vision`:
- Pin MLX Swift LM and transitive versions in `Package.resolved`.
- Use the public MLXVLM/MLXLMCommon APIs and the registered Gemma 4 model and processor.
- Accept exactly one JSON request on stdin and write exactly one JSON response on stdout.
- Load only from the verified local model directory; never download implicitly.
- Disable thinking and sampling, enforce the token cap, and report load/prefill/generation timing.
- Build it in release mode from `build.rs`/the existing bundle path, embed it in the app, and codesign it with the app.
- Use the same raised global macOS deployment target as DS4Server; there is no older-system feature gate.
Do not add a Rust abstraction with multiple backends or a custom VLM inference engine. There is one tested backend in phase one.
### Model catalog integration
Auxiliary assets are a model bundle, not a chat `ModelChoice`. Add a separate `AuxiliaryModelId` and a pinned manifest for the required config, tokenizer, processor, and safetensor files.
Reuse the existing model transfer primitives for download resume, byte progress, SHA-256 verification, deletion, and error reporting. The manifest must pin an immutable repository revision plus the expected size and hash of every file. Store the completed bundle in the normal model catalog (for example, `models/vision-gemma-4-e4b/`) and display aggregate stored/expected size and verification state in the model manager.
There must be no mutable “main” reference, implicit helper download, or deletion after a call. Once downloaded and verified, the auxiliary model remains available for future calls and experiments until the user explicitly removes it.
### Conditional low-memory pause path
Start without explicit DeepSeek offloading and measure it. macOS already has the ability to reclaim clean mmap-backed weight pages, and adding manual lifecycle machinery before evidence would make the riskiest part of the engine more complex.
If the acceptance measurements show that the lowest supported memory configuration cannot run the helper reliably, add one scoped `Generator::pause_for_auxiliary()` guard:
1. Synchronize all DeepSeek command buffers.
2. Release the SSD-streamed expert cache and any inactive resident-session allocations.
3. Detach/end the model residency set.
4. Apply `POSIX_MADV_DONTNEED` only to clean main/support weight ranges while retaining the mappings and no-copy buffers.
5. Keep active KV cache, scratch/logit state needed for continuation, and generator metadata alive.
6. On guard drop—success, tool error, timeout, cancellation, or panic—recreate the residency set and restore a usable executor.
7. Do not eagerly warm the full GGUF; allow the next DeepSeek pass to fault the required pages back.
Never call full `ds4_gpu_cleanup`, unmap the GGUF, or leave Metal buffers pointing at released virtual memory. Full generator checkpoint/drop/reopen/restore is only a final fallback if measured hardware evidence proves that retaining KV plus mappings cannot fit.
### Expected implementation touch points
- `native/vision/`: pinned Swift helper package and protocol implementation
- `build.rs`, bundle/build metadata: build, embed, and sign the helper
- `src/vision.rs`: request/response protocol, validation, direct child-process lifecycle
- `src/runtime.rs`: `AnalyzeImage` operation, serialization, cancellation, activity events
- `src/agent.rs`: tool schema/handler and access to `GenerationService`
- `src/model.rs` and `src/model/transfer.rs`: auxiliary bundle manifest, verification, and transfer reuse
- model-manager UI: auxiliary bundle download/verify/delete/status
- Metal bridge only if the measured conditional pause guard is required
### Verification and acceptance criteria
#### Functional
- Analyze real PNG, JPEG, GIF, and WebP fixtures covering photos, screenshots, charts, documents/OCR, and spatial questions.
- Repeated temperature-zero runs produce materially identical answers.
- A prompt-injection image is described but its embedded instructions are not followed.
- Invalid MIME/header, oversize bytes, excessive decoded pixels, symlink escape, missing file, corrupt model, and missing helper all produce clear bounded errors.
- Cancellation during model load and generation terminates/reaps the helper and leaves DeepSeek immediately usable.
- Unit-test the JSON protocol and path limits with a fake helper; add an app-level test using the real bundled helper/model where hardware permits.
#### Context and correctness
- The same loaded DeepSeek generator and active conversation KV cache continue after the tool call.
- The next DeepSeek result matches the behavior of inserting the same description as ordinary tool text; no hidden context reset or token-accounting change.
- DS4 token processing, KV accounting, and generation behavior remain unchanged outside the new tool result.
- No image bytes or auxiliary prompt tokens are counted as DeepSeek context.
#### Memory and concurrency
Measure on the target M5 hardware and the lowest supported RAM configuration, with 4K, 32K, and maximum practical active contexts, and with both resident and SSD-streamed DeepSeek modes (including DSpark):
- physical footprint and memory pressure before, during, and after the helper;
- helper peak footprint and whether it returns after exit;
- latency to first auxiliary token and total tool time;
- first-token and steady-state DeepSeek speed immediately after the tool;
- cold-page/refault penalty on the first resumed DeepSeek pass;
- unchanged KV checksum/continuation across the tool interval.
Also verify:
- a simultaneous HTTP request queues while the server remains alive;
- no two GPU inference paths overlap;
- helper crash, timeout, and cancellation all restore normal DeepSeek generation;
- there is no hidden model re-download;
- no main generator reconstruction occurs on the normal path.
Add the conditional pause guard only if these results demonstrate a real memory failure or unacceptable pressure, and record the before/after measurements in the implementation issue comment.
#### Build and distribution
- A clean release build embeds and signs the helper.
- The installed app works without Python, Homebrew, Xcode, or network access once the auxiliary bundle is present.
- The app, helper, bundle metadata, and CI all declare the same raised minimum macOS version; no runtime old-system branch remains.
- Model catalog deletion removes only the explicitly selected auxiliary bundle.
- Existing release, Clippy, formatting, bundle, and all-feature test gates stay green.
### Deferred follow-ups
Keep these out of the first implementation:
- direct multimodal OpenAI/API request blocks or chat composer attachments;
- persistent auxiliary conversation/KV state;
- a resident helper/model cache;
- a custom DS4 Metal VLM graph;
- audio and video.
After the image path is proven, video and audio can reuse the same Gemma 4 E4B tool/helper boundary. Each should remain a separate follow-up with bounded media preprocessing and real MLX Swift processor fixtures before it is enabled.
hugo
added the idea label 2026-08-14 20:28:16 +00:00
hugo
added the wontfix label 2026-09-05 21:29:25 +00:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Use the current Gemma 4 E4B Instruct model at 4-bit, with temperature 0 and thinking disabled, to describe and interrogate image content so the DeepSeek model can reason over it. When DeepSeek needs a specific fact from an image, it asks the auxiliary model a focused question and receives deterministic textual tool output.
Use MLX Swift LM through a bundled one-shot helper process rather than extending the custom DeepSeek inference engine. The helper owns the auxiliary model's independent MLX and Metal state and releases it completely when the process exits.
Gemma 4 E4B is the preferred auxiliary model. The current model family handles text, images, audio, and video, so the same sidecar boundary can later add listening and video understanding without changing model families. Current MLX Swift LM already implements the Gemma 4 VLM and processor, so DS4Server does not need a custom Gemma inferencer. Phase one remains image analysis so the memory and lifecycle design can be proven in isolation.
The model integration would be using internal tools, not a chat integration, those model runs are one-shot invocations, so no context handling - the only context is the actual file.
Memory is tight, but the baseline must keep DeepSeek's mmap, executor, and active KV cache alive while serializing the auxiliary invocation. Clean file-backed DeepSeek pages may be reclaimed by macOS while the helper is active. Explicit expert-cache release and page-discard advice are a measured fallback, not a prerequisite.
The existing agent tool loop invokes the auxiliary model and inserts its bounded textual result before DeepSeek continues. The same loaded DeepSeek generator and KV context remain intact; the normal path does not reconstruct the model.
The existing generation queue serializes DeepSeek and auxiliary work. The server remains listening while an image is analyzed, and concurrent requests queue rather than overlapping GPU inference or forcing a server restart.
Workable implementation plan (2026-08-30)
Decision and scope
This has no corresponding DS4 functionality; it is a new DS4Server feature.
Implement phase one as an agent-only image-analysis tool backed by the current 4-bit Gemma 4 E4B Instruct model (
mlx-community/gemma-4-e4b-it-4bit). Keep the loaded DeepSeek generator and its KV cache alive. Run the auxiliary model in a bundled, one-shot Swift/MLX helper process and let that process exit after each tool call. Do not add a second model to DS4Server's current native Metal executor, and do not unload/reconstruct DeepSeek by default.Gemma 4 E4B is the required default, not a compatibility option. Google positions E4B as the stronger small on-device member of the current Gemma 4 family: 4.5B effective parameters (8B including per-layer embeddings), a 128K context window, image and audio input, variable-resolution vision, OCR, charts, UI understanding, and configurable thinking. The exact 4-bit E4B checkpoint is registered by the current released MLX Swift LM VLM factory.
Pin MLX Swift LM 3.31.4 or a newer release that has passed the Gemma 4 image fixture and memory gates. Do not retain any older model family as a silent fallback.
Raise DS4Server's global macOS deployment target to the minimum required by the pinned MLX/Gemma implementation instead of feature-gating vision or carrying an old-system path. Update the Rust build, Swift package, bundle metadata, documentation, and CI target together. If the final pinned implementation uses a newer macOS-only API, raise the global minimum again; do not add compatibility branches to preserve an older target.
Why the helper process is the right boundary
The current DS4 Metal bridge is process-global: it has one global model-view table, command queues, pipelines, and model residency set. Its model buffers are no-copy Metal buffers pointing directly into the main GGUF mmap. A second independent model/runtime in that singleton would require a broad native-state refactor and creates cleanup and concurrency hazards.
A dedicated helper gives the auxiliary MLX runtime independent Metal state. It also gives deterministic cleanup: process exit releases the auxiliary model mappings, MLX buffers, command queues, and transient image tensors even on an error. It must be an app-bundled executable invoked directly by Rust—no shell, Python, Homebrew, or user-installed runtime.
mmap and memory behavior
Both model files may be mmaped at the same time. An mmap reserves address space; clean file-backed pages only consume physical memory while resident and macOS may evict them under pressure. Keeping the DeepSeek mmap open does not require keeping all its weights resident.
The DeepSeek mmap itself cannot be released while its executor is alive because the current no-copy Metal buffers retain pointers into that mapping. Unmapping it would make those buffers invalid. The safe baseline is therefore:
This preserves the conversation context and avoids the much larger cost and risk of serializing/rebuilding the main generator.
Agent tool contract
Add one internal tool for phase one:
analyze_imageTreat the image and any text inside it as untrusted data. The fixed helper prompt must say that instructions visible in the image are content to report, not instructions to follow.
At the Rust trust boundary:
imagedependency.The result should enter the existing agent flow as normal textual tool output. Do not put pixels, base64, or multimodal blocks into
ChatTurnor the OpenAI-compatible server request format in phase one.Execution flow and serialization
Add an
AnalyzeImageoperation to the existingGenerationServicecommand queue:analyze_imagetool call.Toolsvalidates the request and submitsAnalyzeImagewith the currently loaded engine settings.This queue is also used by HTTP generation, so the server can remain listening. Requests arriving during image analysis should queue normally; streaming requests should continue to receive the existing prefill/keepalive behavior. Surface a runtime activity state such as “Analyzing image…” rather than disabling the server. Never permit DeepSeek and auxiliary Metal generation concurrently.
Runtime helper and packaging
Add a small Swift executable package under
native/vision:Package.resolved.build.rs/the existing bundle path, embed it in the app, and codesign it with the app.Do not add a Rust abstraction with multiple backends or a custom VLM inference engine. There is one tested backend in phase one.
Model catalog integration
Auxiliary assets are a model bundle, not a chat
ModelChoice. Add a separateAuxiliaryModelIdand a pinned manifest for the required config, tokenizer, processor, and safetensor files.Reuse the existing model transfer primitives for download resume, byte progress, SHA-256 verification, deletion, and error reporting. The manifest must pin an immutable repository revision plus the expected size and hash of every file. Store the completed bundle in the normal model catalog (for example,
models/vision-gemma-4-e4b/) and display aggregate stored/expected size and verification state in the model manager.There must be no mutable “main” reference, implicit helper download, or deletion after a call. Once downloaded and verified, the auxiliary model remains available for future calls and experiments until the user explicitly removes it.
Conditional low-memory pause path
Start without explicit DeepSeek offloading and measure it. macOS already has the ability to reclaim clean mmap-backed weight pages, and adding manual lifecycle machinery before evidence would make the riskiest part of the engine more complex.
If the acceptance measurements show that the lowest supported memory configuration cannot run the helper reliably, add one scoped
Generator::pause_for_auxiliary()guard:POSIX_MADV_DONTNEEDonly to clean main/support weight ranges while retaining the mappings and no-copy buffers.Never call full
ds4_gpu_cleanup, unmap the GGUF, or leave Metal buffers pointing at released virtual memory. Full generator checkpoint/drop/reopen/restore is only a final fallback if measured hardware evidence proves that retaining KV plus mappings cannot fit.Expected implementation touch points
native/vision/: pinned Swift helper package and protocol implementationbuild.rs, bundle/build metadata: build, embed, and sign the helpersrc/vision.rs: request/response protocol, validation, direct child-process lifecyclesrc/runtime.rs:AnalyzeImageoperation, serialization, cancellation, activity eventssrc/agent.rs: tool schema/handler and access toGenerationServicesrc/model.rsandsrc/model/transfer.rs: auxiliary bundle manifest, verification, and transfer reuseVerification and acceptance criteria
Functional
Context and correctness
Memory and concurrency
Measure on the target M5 hardware and the lowest supported RAM configuration, with 4K, 32K, and maximum practical active contexts, and with both resident and SSD-streamed DeepSeek modes (including DSpark):
Also verify:
Add the conditional pause guard only if these results demonstrate a real memory failure or unacceptable pressure, and record the before/after measurements in the implementation issue comment.
Build and distribution
Deferred follow-ups
Keep these out of the first implementation:
After the image path is proven, video and audio can reuse the same Gemma 4 E4B tool/helper boundary. Each should remain a separate follow-up with bounded media preprocessing and real MLX Swift processor fixtures before it is enabled.
not needed anymore due to fast vision capable models and ds4 flash exp