Add resumable background model downloads

PLAN.md

@@ -15,15 +15,16 @@ sessions, tools, and turn orchestration.
 | App shell | Implemented | Native Iced window, Codex-inspired two-pane layout, dark theme, embedded SVG icons, `Command-,` Preferences panel | Functional chat content and native app packaging |
 | Projects and sessions | Implemented for metadata | Native folder picker, project-name dialog, create/select/delete projects and sessions | Transcripts, KV payloads, rename/archive, and model binding |
 | Persistence | Implemented for metadata and preferences | SQLite in Application Support, Diesel ORM, embedded migrations, constrained defaults and CRUD tests | Messages, downloaded-artifact metadata, and session-runtime migrations |
-| Model runtime | Preferences only | Persisted model choice, DFlash toggle, and idle timeout | Downloads, loading, inference, idle unloading, and Metal integration |
+| Model runtime | Download workflow implemented | Persisted model choice, DSpark toggle, idle timeout, and Rust-native background main/DSpark downloads with progress, ETA, stop, restart-resume, size checks, and SHA-256 verification | Loading, inference, idle unloading, and Metal integration |
 | Local endpoint | Not started | Nothing listening | OpenAI-compatible HTTP and streaming surfaces |
 | Agent | UI shell only | Empty conversation pane and composer placeholder | Messages, generation, tools, approvals, compaction, and cancellation |
 | Dev brain | Not started | No vault access | Obsidian vault selection, durable access, and agent knowledge/memory tools |
 | Release | Not started | Development binary builds and tests | `.app` packaging, signing, entitlements, and notarization |
 
 The application currently manages durable project/session metadata and model
-preferences. It does not yet download or load a model, serve HTTP, save chat
-messages, or run an agent.
+preferences, and can download the selected model's main and optional DSpark
+GGUF into managed Application Support storage without blocking the UI. It does
+not yet load a model, serve HTTP, save chat messages, or run an agent.
 
 ## Phase 0 — project and session shell
 
@@ -42,8 +43,9 @@ Exit criterion: restart the app and see the same project/session tree.
 
 ## Phase 1 — model library and on-demand loading
 
-Status: **partially implemented**. Preferences and their schema are complete;
-model downloads, loading, inference, and idle unloading remain open.
+Status: **partially implemented**. Preferences and the complete background
+download workflow are implemented; loading, inference, and idle unloading
+remain open.
 
 1. Port the narrow GGUF loader, tokenizer, prompt renderer, and Metal session
    API behind a small Rust `Engine` surface modeled on `ds4.h`:
@@ -58,6 +60,48 @@ model downloads, loading, inference, and idle unloading remain open.
 5. Move engine work off the UI thread and support cooperative cancellation at
    the safe session boundaries already defined by `ds4_session_sync`.
 
+### DS4 KV-cache port and shared session core
+
+KV-cache behavior is part of the DS4 engine contract, not a new application
+cache to redesign. Port the existing `ds4_session` and `ds4_kvstore` behavior
+to Rust while preserving its semantics and on-disk compatibility:
+
+- A live engine session owns one mutable inference timeline: exact rendered
+  tokens, logits, and the DS4-specific KV graph state. Callers provide the full
+  token prefix and the Rust port of `ds4_session_sync` must retain the longest
+  compatible prefix, evaluate only its suffix, or safely rewind, invalidate,
+  and rebuild exactly when DS4 does.
+- Carry over DS4's safe cancellation/checkpoint boundaries, common-prefix
+  matching, canonical rewrite and replay, snapshots, payload staging,
+  save/load, strip-to-transcript behavior, and atomic replacement. The engine
+  continues to own the opaque KV payload format; UI, HTTP, and persistence code
+  must not reinterpret tensor state.
+- Port `ds4_kvstore` policy rather than replacing it with a generic cache:
+  exact token and rendered-text prefix lookup, cold and continued aligned
+  checkpoints, model/quantization/context/payload-ABI compatibility checks,
+  tool-call replay metadata, disk-budget enforcement, hit tracking, and
+  eviction behavior.
+- Use one process-wide Rust session registry and KV checkpoint store for both
+  local chat and the HTTP server. A remote request bound to an application
+  session resolves to the same logical DS4 session as the GUI and mutations are
+  serialized so two callers never write one live timeline concurrently.
+  Stateless HTTP requests still participate in the same compatible-prefix
+  checkpoint pool, so a prefix produced locally can be reused remotely and
+  vice versa.
+- Store that shared checkpoint pool under Application Support in
+  `kv-cache/`, using the DS4 key/header/payload rules. Session rows hold only an
+  optional reference to their current checkpoint; the large opaque payload is
+  not copied into separate GUI and HTTP caches.
+- Sharing does not merge unrelated conversations or expose one session's
+  transcript to another. It shares the DS4 implementation, checkpoint pool, and
+  explicitly bound logical session; each unrelated live conversation keeps its
+  own mutable session state.
+- Engine unload must first leave every reusable live timeline in a valid
+  persisted checkpoint. Reload restores a compatible checkpoint immediately;
+  if model identity, quantization, context size, payload ABI, or rendered
+  history is incompatible, rebuild from the transcript and replace the stale
+  checkpoint only after the new state is valid.
+
 ### Lazy lifecycle
 
 - Start the application and the local HTTP endpoint with no model loaded.
@@ -82,24 +126,45 @@ model downloads, loading, inference, and idle unloading remain open.
 - Provide a Preferences panel from the sidebar and the standard
   macOS `Command-,` shortcut.
 - Let the user choose the active model. Default to `deepseek-v4-flash`.
-- Let the user enable or disable DFlash for the selected model. Disable the
-  control for models that do not support DFlash.
+- Let the user enable or disable DSpark for the selected model. Disable the
+  control for models that do not support DSpark.
 - Let the user configure the inactivity timeout; default to 10 minutes.
 - Persist preferences in the application SQLite database. Changing preferences
   never loads a model by itself.
 
 ### Targeted downloads and storage
 
+- The Preferences panel can download the selected model's main GGUF and the
+  DSpark support GGUF when enabled. A dedicated worker thread performs HTTP and
+  verification work so Preferences and the rest of the application stay
+  interactive.
+- Use an in-process Rust HTTP client with Rustls rather than invoking a system
+  downloader. Stream response chunks directly into a `.part` file and use a
+  Rust SHA-256 implementation for verification.
+- Report aggregate bytes present, total bytes, bytes missing, percentage,
+  smoothed transfer rate, current artifact/verification phase, and estimated
+  time remaining. Keep the detailed display in Preferences and a persistent
+  status bar with a Stop action while the transfer runs.
+- Stop is non-destructive: close the active transfer and retain its `.part`
+  file. Starting the download again in the same process or after quitting and
+  relaunching must derive progress from that file and issue an HTTP range
+  request from its exact byte length. If the server ignores the range request,
+  safely restart that artifact from byte zero rather than appending corrupt
+  data.
+- Promote a `.part` file only after its exact size and SHA-256 are verified. A
+  stopped verification keeps the complete partial file so the next run can
+  verify and promote it without downloading it again.
 - Store managed model artifacts under
   `~/Library/Application Support/DS4Server.rfc1437.de/models/<model-id>/` and
   mmap/load them from there. Do not place managed models in the app bundle or a
   project directory.
-- Download only the selected model's main artifact. Download its DFlash
-  artifact only when DFlash is enabled; never prefetch other catalog models or
+- Download only the selected model's main artifact. Download its DSpark
+  artifact only when DSpark is enabled; never prefetch other catalog models or
   optional artifacts.
-- Verify size/checksum before making a download usable, resume partial
-  downloads, and remove failed temporary files without touching a valid model.
-- When the selected model changes or DFlash is disabled, offer removal of the
+- Remove checksum-invalid partial files without touching a valid model. Keep
+  partial files after cancellation, application exit, network failure, or an
+  interrupted verification so they remain restart-resumable.
+- When the selected model changes or DSpark is disabled, offer removal of the
   now-unused artifact; do not delete large existing downloads without explicit
   confirmation.
 
@@ -128,41 +193,52 @@ Status: **open; not started**.
 4. Port exact sampled tool-call replay and deterministic canonicalization from
    `ds4_server.c` so a client's normalized JSON does not destroy KV-prefix
    reuse.
-5. Bind API conversations to engine sessions and persist/restore KV payloads
-   using the engine-owned serialization format. Add resident batching only
-   after single-session correctness is established.
+5. Route API conversations through the shared session registry and KV store
+   defined in Phase 1. Responses `conversation` bindings and any other
+   persistent request identity map to the same application session used by the
+   GUI; stateless requests use shared DS4 prefix lookup. Persist/restore only
+   through the engine-owned serialization format. Add resident batching only
+   after single-session and cross-surface correctness are established.
 6. Display endpoint address, model, active requests, token rates, and errors in
    the GUI.
 
 Exit criterion: the upstream DwarfStar server tests pass against the app for
 non-streaming and streaming Chat Completions and Responses calls, including a
-multi-turn tool call.
+multi-turn tool call. A compatible prefix created by local chat must register a
+remote cache hit, a remote checkpoint must be reusable by local chat, and a
+GUI/HTTP race on one bound session must serialize without corrupting its token
+or KV frontier.
 
 ## Phase 3 — durable agent sessions
 
 Status: **partially open**. Project/session metadata rows exist; transcripts,
 model binding, and KV payload persistence are not implemented.
 
-Replace each metadata-only session with a directory:
+Extend each metadata-only session with transcript and shared-checkpoint
+metadata:
 
 ```text
 Application Support/DS4Server.rfc1437.de/
   data.sqlite3
   models/<model-id>/
     main.gguf
-    dflash.gguf       # only when enabled
-  sessions/<session-id>/
-    kv.bin
+    dspark.gguf       # only when enabled
+  kv-cache/
+    <ds4-cache-key>.kv
 ```
 
-- Add migrated message/transcript tables to SQLite. Keep only the engine-owned
-  large KV payload in per-session files, written with atomic replacement.
+- Add migrated message/transcript tables and an optional current-checkpoint key
+  to SQLite. Keep large engine-owned KV payloads only in the process-wide Phase
+  1 store, written with atomic replacement; do not introduce separate
+  agent-only or HTTP-only cache formats.
 - Record model identity, context size, rendered token history, title,
   timestamps, and working directory.
 - Restore compatible KV immediately. If KV is absent or incompatible, rebuild
   from the transcript and show prefill progress.
-- Add strip/archive behavior equivalent to the native agent: retain transcript
-  while removing the large KV payload.
+- Add strip/archive behavior equivalent to the native agent: retain the
+  transcript and clear its checkpoint reference. Remove the cache file only
+  when it is not reusable by another live/session prefix; otherwise normal DS4
+  disk-budget eviction reclaims it.
 - Version formats before the first release and add migrations only when a real
   format change exists.
 
@@ -176,7 +252,8 @@ visual placeholders with no message or agent runtime behind them.
 
 1. Build the classic chat surface: transcript, reasoning disclosure, tool-call
    cards, composer, queued input, stop button, prefill progress, and generation
-   statistics.
+   statistics. Local turns must call the same session registry and KV path used
+   by the HTTP endpoint.
 2. Port the native agent turn loop from `ds4_agent.c`, including model-specific
    system prompts and DeepSeek DSML/GLM tool-call parsing.
 3. Implement the local tool set with the project directory as its boundary:
@@ -231,6 +308,7 @@ and the preferences shortcut have unit coverage. Model fixtures, integration
 tests, and release packaging remain open.
 
 - Keep unit coverage narrow: persistence round trips, format compatibility,
+  DS4 KV prefix/rewrite/store/load parity, cross-surface session binding,
   request parsing, and tool-boundary checks.
 - Reuse `../ds4` fixtures for prompt rendering, sampling, server streaming, and
   KV correctness; run live Metal tests only when a supported GGUF is available.
@@ -241,13 +319,11 @@ tests, and release packaging remain open.
 
 ## Recommended next milestones
 
-1. Add the targeted download manager and model catalog, storing only selected
-   artifacts under Application Support with resumable, verified downloads.
-2. Implement the single-engine lifecycle and local generation path: lazy load,
+1. Implement the single-engine lifecycle and local generation path: lazy load,
    shared concurrent acquisition, activity tracking, and idle unload.
-3. Put the OpenAI-compatible endpoint on that same engine lifecycle so local
+2. Put the OpenAI-compatible endpoint on that same engine lifecycle so local
    chats and HTTP requests cannot load duplicate engines.
-4. Add transcript/KV persistence, then connect the existing conversation UI and
+3. Add transcript/KV persistence, then connect the existing conversation UI and
    finally port the agent tools.
-5. Add the opt-in Dev Brain tool after local file/search boundaries and agent
+4. Add the opt-in Dev Brain tool after local file/search boundaries and agent
    approvals are stable.