initial commit

PLAN.md

@@ -0,0 +1,133 @@
+# DS4Server implementation plan
+
+Bundle/application identifier: `DS4Server.rfc1437.de`
+
+The GUI is a Rust/Iced reimplementation of the user-facing model server and
+coding-agent workflows in `../ds4`. The existing engine remains the behavioral
+reference: `ds4.c`/`ds4.h` define model and session behavior,
+`ds4_server.c` defines the compatible API, and `ds4_agent.c` defines agent
+sessions, tools, and turn orchestration.
+
+## Phase 0 — project and session shell (implemented)
+
+- Show projects in a sidebar with their sessions nested underneath.
+- Create projects with a native macOS folder picker and a project-name dialog.
+- Create, select, and delete projects and sessions.
+- Persist projects and sessions in SQLite under
+  `~/Library/Application Support/DS4Server.rfc1437.de/`, using Diesel ORM
+  queries and embedded Diesel migrations.
+- Never delete workspace contents when removing app metadata.
+
+Exit criterion: restart the app and see the same project/session tree.
+
+## Phase 1 — model library and on-demand loading
+
+1. Port the narrow GGUF loader, tokenizer, prompt renderer, and Metal session
+   API behind a small Rust `Engine` surface modeled on `ds4.h`:
+   `open`, `close`, `create_session`, `sync`, `sample`, and KV save/load.
+2. Retain the tested model restrictions from DwarfStar; reject unsupported
+   shapes and quantizations before allocating model state.
+3. Reuse the existing `.metal` kernels and preserve mmap-backed resident
+   loading plus the explicit SSD-streaming path. Keep Objective-C only at the
+   Metal interop boundary.
+4. Add a model library screen that records GGUF paths and loading options.
+   Load only after the user chooses a model. Expose
+   `Unloaded → Loading → Ready/Failed → Unloading` state and progress in Iced.
+5. Keep one engine resident initially, matching DwarfStar's instance-lock and
+   memory assumptions. Switching models unloads the current engine first.
+6. Move engine work off the UI thread and support cooperative cancellation at
+   the safe session boundaries already defined by `ds4_session_sync`.
+
+Persist model references and tuning options in Application Support. For a
+sandboxed distribution, store security-scoped bookmarks rather than assuming
+raw paths remain accessible.
+
+Exit criterion: select a supported DeepSeek V4 or GLM 5.2 GGUF, load it without
+blocking the UI, run a deterministic prompt, unload it, and pass the matching
+`../ds4` token-output fixture.
+
+## Phase 2 — OpenAI-compatible local endpoint
+
+1. Add a localhost-only HTTP service owned by the loaded engine. Start and stop
+   it from the GUI; never expose a LAN listener without an explicit setting.
+2. Implement the DwarfStar compatibility surface in this order:
+   - `GET /v1/models`
+   - `POST /v1/chat/completions`
+   - `POST /v1/responses`
+   - `POST /v1/completions`
+3. Support streaming SSE, reasoning output, usage accounting, cancellation,
+   sampling parameters, tool schemas, and tool choice.
+4. Port exact sampled tool-call replay and deterministic canonicalization from
+   `ds4_server.c` so a client's normalized JSON does not destroy KV-prefix
+   reuse.
+5. Bind API conversations to engine sessions and persist/restore KV payloads
+   using the engine-owned serialization format. Add resident batching only
+   after single-session correctness is established.
+6. Display endpoint address, model, active requests, token rates, and errors in
+   the GUI.
+
+Exit criterion: the upstream DwarfStar server tests pass against the app for
+non-streaming and streaming Chat Completions and Responses calls, including a
+multi-turn tool call.
+
+## Phase 3 — durable agent sessions
+
+Replace each metadata-only session with a directory:
+
+```text
+Application Support/DS4Server.rfc1437.de/
+  data.sqlite3
+  sessions/<session-id>/
+    kv.bin
+```
+
+- Add migrated message/transcript tables to SQLite. Keep only the engine-owned
+  large KV payload in per-session files, written with atomic replacement.
+- Record model identity, context size, rendered token history, title,
+  timestamps, and working directory.
+- Restore compatible KV immediately. If KV is absent or incompatible, rebuild
+  from the transcript and show prefill progress.
+- Add strip/archive behavior equivalent to the native agent: retain transcript
+  while removing the large KV payload.
+- Version formats before the first release and add migrations only when a real
+  format change exists.
+
+Exit criterion: quit during normal use, relaunch, select a session, and resume
+the same conversation without prefill when its KV payload is compatible.
+
+## Phase 4 — agent chat and tools
+
+1. Build the classic chat surface: transcript, reasoning disclosure, tool-call
+   cards, composer, queued input, stop button, prefill progress, and generation
+   statistics.
+2. Port the native agent turn loop from `ds4_agent.c`, including model-specific
+   system prompts and DeepSeek DSML/GLM tool-call parsing.
+3. Implement the local tool set with the project directory as its boundary:
+   - shell command execution
+   - file read and continuation
+   - file write
+   - anchored edit
+   - text/file search
+4. Stream assistant tokens and partial tool calls into Iced while inference and
+   tools run on workers. Preserve cooperative interruption and a valid KV
+   prefix at every stop point.
+5. Require confirmation for commands or writes outside the project boundary,
+   destructive actions, and network tools. Add web search/page visiting only
+   after the local tool loop is stable.
+6. Port context compaction and system-prompt reinjection after ordinary
+   multi-turn operation is correct.
+
+Exit criterion: create a project session, ask the model to inspect and edit a
+small fixture repository, review each tool result in the GUI, restart, and
+continue from the persisted session.
+
+## Verification and packaging
+
+- Keep unit coverage narrow: persistence round trips, format compatibility,
+  request parsing, and tool-boundary checks.
+- Reuse `../ds4` fixtures for prompt rendering, sampling, server streaming, and
+  KV correctness; run live Metal tests only when a supported GGUF is available.
+- Preserve the DwarfStar/llama.cpp notices required by `../ds4/LICENSE` for
+  reused or adapted source and kernels.
+- Produce a signed `.app` with the exact identifier above, then add hardened
+  runtime, entitlements, notarization, and update delivery as release work.