From ee7e5696a5b8ec6b5567e52c5861815ccefb628f Mon Sep 17 00:00:00 2001 From: Georg Bauer Date: Mon, 27 Jul 2026 17:08:13 +0200 Subject: [PATCH] chore: updated plan for dev brain --- PLAN.md | 132 ++++++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 124 insertions(+), 8 deletions(-) diff --git a/PLAN.md b/PLAN.md index 96fa841..066d58c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -4,12 +4,128 @@ Only unfinished implementation work belongs here. DwarfStar remains the behavioral oracle for model execution, token processing, context accounting, KV-cache behavior, the HTTP API, and the built-in agent loop. -## 1. Add Dev Brain support +## Dev Brain: a project-backed LLM Wiki -- Let the user select one Obsidian vault and give the local agent bounded - Markdown memory_search, memory_read, memory_create, and memory_append - operations outside project folders. -- the dev brain uses simple markdown lines to separate multiple memories -- check LLM wiki ideas online to see how obdsidian should be used by the LLM - as a memory storage -- dev brain can be enabled/disabled in preferences to reduce system prompt +Dev Brain implements the LLM Wiki idea described by Andrej Karpathy. The +registered projects remain the authoritative, changing source material. A +user-selected Obsidian vault is the derived, human-readable body of knowledge +that the agent maintains from those projects. Its purpose is to compile durable +knowledge about architecture, behavior, decisions, invariants, workflows, and +relationships once, keep it aligned as the projects change, and make it cheap +for later agents to retrieve the relevant context. + +This is not a general Markdown memory store. The vault contains coherent topic +pages with explicit provenance, links, and freshness state. Queries use the +compiled wiki when it is verified and return to project sources when knowledge +is missing or stale. The vault remains ordinary Obsidian Markdown so that the +user can inspect, navigate, edit, and version it without DS4Server. + +### Configuration and ownership + +- Preferences provide an enable switch and the path to one dedicated Obsidian + vault. When disabled, Dev Brain contributes neither tools nor instructions to + the agent prompt. +- A vault must be an existing directory containing `.obsidian`. DS4Server may + change only its declared wiki files and must never modify Obsidian settings, + attachments, trash, hidden files, or unrelated user notes. +- The existing configuration store persists the setting. Changing the vault or + enable state invalidates the current Dev Brain context and derived index. + +### Vault contract + +The vault is self-describing and contains: + +- `purpose.md`, maintained with the user, defining the wiki's scope, priorities, + and recurring questions; +- `schema.md`, defining page types, naming and linking conventions, provenance, + freshness states, and the rules for compilation, querying, and validation; +- `index.md`, a concise catalog of wiki pages with one-line descriptions; +- `log.md`, an append-only record of material wiki updates and their source + revisions; and +- topic pages grouped under `projects/`, `subsystems/`, `concepts/`, + `decisions/`, `invariants/`, and `workflows/` as the content requires. + +Every managed topic page has YAML frontmatter containing its page type, project +identity, `verified`, `stale`, or `needs-review` status, verification time, and +source records. A source record identifies the registered project, a +repo-relative path, an optional symbol, and the exact evidence version: the Git +revision for a clean worktree and a content hash for the files actually read. +The body uses normal Obsidian links and embeds to connect related knowledge. +Pages model useful concepts rather than mirroring every source file. + +### Compilation and maintenance + +Initial compilation examines high-signal project material such as manifests, +documentation, schemas, entry points, public interfaces, and tests, then writes +the smallest set of durable topic pages that answers the purpose of the wiki. +It does not attempt to summarize every file. + +Refresh compares each page's recorded evidence with the current project state. +Changed source paths and hashes select the dependent pages to re-read and +revalidate. The agent may update, split, merge, or retire those pages and keeps +`index.md` and `log.md` consistent with the result. Knowledge crystallized from +a development session must still cite project evidence; unsupported conclusions +are marked `needs-review` rather than presented as fact. + +Wiki changes are prepared as one candidate batch in a temporary location. The +complete candidate is validated before managed files are replaced. A failed or +ambiguous update leaves the last valid vault untouched, and publication must +not expose a partially updated wiki. + +### Parsing, indexing, and querying + +Use `turbovault-parser` to parse Obsidian-flavored Markdown, including YAML +frontmatter, headings, wikilinks, embeds, tags, and source positions without +treating link-like text inside code as graph edges. Keep the Markdown files as +the source of truth and build a disposable SQLite index using the database +support already present in DS4Server: + +- page path, title, type, status, headings, tags, and searchable body text; +- resolved wikilink and embed edges; +- project-source-to-page provenance edges; and +- an FTS5 index for ranked full-text search. + +The index is rebuilt when a vault is enabled and refreshed from file metadata +and content hashes before use. Querying starts with `index.md` or FTS, follows +relevant links and backlinks, and returns both wiki pages and their project +evidence. Stale or unverified pages are labeled and excluded from authoritative +answers until the underlying sources have been checked. + +The first implementation deliberately uses no filesystem watcher, embeddings, +vector database, or separate search service. Explicit refresh plus SQLite FTS5 +is sufficient until vault size or measured retrieval quality proves otherwise. + +### Validation + +Validation prevents the wiki from silently becoming a confident but obsolete +source: + +- structural validation checks frontmatter, page types, required fields, and + the managed path boundary; +- graph validation checks that links resolve, page identities are unique, and + `index.md` matches the managed pages; +- provenance validation checks that every cited project and source exists and + that paths remain inside the registered project; +- freshness validation checks recorded revisions and hashes against the current + files; and +- semantic validation has the agent re-check affected claims against changed + code, documentation, and tests before marking a page verified. + +Semantic validation cannot prove every statement correct, so uncertainty must +remain visible as `needs-review`. Agents may rely on `verified` pages, may use +`stale` pages only as leads, and must not turn unsupported wiki text into project +truth. + +### Agent integration and acceptance + +Dev Brain is a separate wiki maintenance and retrieval capability, not generic +filesystem access outside the active project. Its operations expose search and +read over the derived index plus validated batch publication of managed wiki +content. Detailed operating instructions live in `schema.md`; the system prompt +only announces Dev Brain when enabled and directs the agent to that contract. + +The implementation is complete when tests cover vault path confinement, +Obsidian parsing and index rebuilding, detection of source-hash drift, broken +links and invalid provenance, exclusion of stale pages from authoritative +queries, all-or-nothing publication, and prompt gating when Dev Brain is +disabled.