hugo/DS4Server
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat: first cut at actual token generation and model loading

Browse Source
This commit is contained in:
Georg Bauer
2026-07-24 17:44:41 +02:00
parent 616b550849
commit ff984bb96a
38 changed files with 66885 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files

56
PLAN.md
View File

@@ -12,23 +12,23 @@ sessions, tools, and turn orchestration.
| Area | Status | Available now | Still open |
| --- | --- | --- | --- |
| App shell | Implemented | Native multi-window Iced app, Codex-inspired layout, native Application/Edit/Window menus, Preferences panel, and Model Manager window | Functional chat content and native app packaging |
| App shell | Implemented | Native multi-window Iced app, Codex-inspired layout, native Application/Edit/Window menus, Preferences panel, Model Manager, and streaming composer | Native app release packaging |
| Projects and sessions | Implemented for metadata | Native folder picker, project-name dialog, create/select/delete projects and sessions | Transcripts, KV payloads, rename/archive, and model binding |
| Persistence | Implemented for metadata and preferences | SQLite in Application Support, Diesel ORM, embedded migrations, constrained defaults and CRUD tests | Messages, downloaded-artifact metadata, and session-runtime migrations |
| Model runtime | Model intake implemented | Typed DS4 preferences, managed downloads, mmap-backed GGUF v3 loading, complete DS4 shape/quantization validation, DeepSeek/GLM tokenization, and prompt rendering | Metal inference, sessions/KV state, lifecycle coordination, and idle unloading |
| Model runtime | DeepSeek Flash vertical implemented | Rust-owned mmap/model/session graph, reused Metal kernels, 2K compressed-attention generation, DS4 sampling, lazy background loading, cancellation, and idle unloading | Ratio-4 sparse indexer beyond 2K, DSpark/SSD/steering, GLM/Pro, KV checkpoint persistence, and shared acquisition for HTTP |
| Local endpoint | Not started | Nothing listening | OpenAI-compatible HTTP and streaming surfaces |
| Agent | UI shell only | Empty conversation pane and composer placeholder | Messages, generation, tools, approvals, compaction, and cancellation |
| Agent | Basic local chat implemented | Multi-turn role rendering, streamed DeepSeek Flash text, composer send/stop, and in-memory transcript | Durable transcripts, reasoning presentation, tools, approvals, compaction, statistics, and KV-backed resume |
| A2UI | Future extension | bDS2 provides the reference structured render-tool contract | Native inline surfaces for local chat after core chat and tools are stable |
| Dev brain | Not started | No vault access | Obsidian vault selection, durable access, and agent knowledge/memory tools |
| Release | Not started | Development binary builds and tests | `.app` packaging, signing, entitlements, and notarization |
The application currently manages durable project/session metadata and model
preferences. Its separate Model Manager can download, fully validate, and
delete main and DSpark GGUF artifacts in managed Application Support storage
without blocking the UI. The engine can mmap a configured model, validate its
metadata and complete tensor layout, load its tokenizer, and render a prompt;
the app does not yet acquire that engine for inference, serve HTTP, save chat
messages, or run an agent.
The application manages durable project/session metadata and model preferences.
Its Model Manager downloads, validates, and deletes managed GGUF artifacts
without blocking the UI. Local chat now lazily acquires the Rust DeepSeek Flash
executor, streams generated text, supports cancellation and multi-turn role
replay, and unloads model resources after the configured idle timeout. The
current Rust graph supports DS4 compressed attention through 2K context; the sparse ratio-4 indexer,
durable transcripts/KV checkpoints, HTTP service, and agent tools remain open.
## Phase 0 — project and session shell
@@ -47,27 +47,27 @@ Exit criterion: restart the app and see the same project/session tree.
## Phase 1 — model library and on-demand loading
Status: **partially implemented**. The typed DS4 preference contract, shared
effective-settings builder, background download workflow, and model-intake
boundary are implemented; Metal inference, session state, and idle lifecycle
coordination remain open.
Status: **partially implemented**. The typed preference/download/intake path and
a DeepSeek Flash Rust/Metal vertical are implemented, including compressed KV
through 2K context, DS4 sampling, a single background owner, cancellation, and
idle unload. Sparse indexed attention, persistent KV/session sync, other model
families, and optional execution modes remain open.
1. **Implemented:** the typed model/runtime preference contract described below
is complete before chat, the HTTP endpoint, or the engine lifecycle.
2. **Partially implemented:** the mmap-backed GGUF loader, DeepSeek/GLM
tokenizer, prompt renderer, and model `open`/`close` ownership boundary are
complete. The Metal session API modeled on `ds4.h` remains:
`create_session`, `sync`, `sample`, and KV save/load.
2. **Partially implemented:** mmap-backed loading, tokenizer/prompt rendering,
Rust-owned Flash session creation/evaluation/sampling, and model ownership
are complete. Prefix sync plus KV save/load remain.
3. **Implemented:** retain the tested model restrictions from DwarfStar. Main
and DSpark artifacts reject invalid GGUF versions, catalog mismatches, and
incompatible metadata, tensor shapes, offsets, or quantization before
promotion and again before model use.
4. Reuse the existing `.metal` kernels and preserve mmap-backed resident
4. **Partially implemented:** reuse the existing `.metal` kernels and preserve mmap-backed resident
loading plus the explicit SSD-streaming path. Keep Objective-C only at the
Metal interop boundary.
5. Keep one engine resident at most, matching DwarfStar's instance-lock and
5. **Implemented for local chat:** keep one engine resident at most, matching DwarfStar's instance-lock and
memory assumptions.
6. Move engine work off the UI thread and support cooperative cancellation at
6. **Implemented for local chat:** move engine work off the UI thread and support cooperative cancellation at
the safe session boundaries already defined by `ds4_session_sync`.
### DS4 KV-cache port and shared session core
@@ -328,10 +328,11 @@ the same conversation without prefill when its KV payload is compatible.
## Phase 4 — agent chat and tools
Status: **UI shell only**. The conversation area, composer, and icons are
visual placeholders with no message or agent runtime behind them.
Status: **basic local chat implemented**. The conversation area streams
multi-turn DeepSeek Flash output and supports Stop; durable messages, tool
turns, and the coding-agent runtime remain open.
1. Build the classic chat surface: transcript, reasoning disclosure, tool-call
1. **Partially implemented:** build the classic chat surface: transcript, reasoning disclosure, tool-call
cards, composer, queued input, stop button, prefill progress, and generation
statistics. Local turns must call the same session registry and KV path used
by the HTTP endpoint.
@@ -481,9 +482,10 @@ tests, and release packaging remain open.
## Recommended next milestones
1. Implement the single-engine lifecycle and local generation path: lazy load,
shared concurrent acquisition, activity tracking, and idle unload.
2. Put the OpenAI-compatible endpoint on that same engine lifecycle so local
1. Finish the Flash session core as one larger slice: port the ratio-4 indexer
and indexed attention for full configured context, then add prefix sync and
compatible KV checkpoint save/load.
2. Put the OpenAI-compatible endpoint on the same engine lifecycle so local
chats and HTTP requests cannot load duplicate engines.
3. Add transcript/KV persistence, then connect the existing conversation UI and
finally port the agent tools.