chore: updated plan for dev brain

PLAN.md

@@ -4,12 +4,128 @@ Only unfinished implementation work belongs here. DwarfStar remains the
 behavioral oracle for model execution, token processing, context accounting,
 KV-cache behavior, the HTTP API, and the built-in agent loop.
 
-## 1. Add Dev Brain support
+## Dev Brain: a project-backed LLM Wiki
 
-- Let the user select one Obsidian vault and give the local agent bounded
+Dev Brain implements the LLM Wiki idea described by Andrej Karpathy. The
-  Markdown memory_search, memory_read, memory_create, and memory_append
+registered projects remain the authoritative, changing source material. A
-  operations outside project folders.
+user-selected Obsidian vault is the derived, human-readable body of knowledge
-- the dev brain uses simple markdown lines to separate multiple memories
+that the agent maintains from those projects. Its purpose is to compile durable
-- check LLM wiki ideas online to see how obdsidian should be used by the LLM
+knowledge about architecture, behavior, decisions, invariants, workflows, and
-  as a memory storage
+relationships once, keep it aligned as the projects change, and make it cheap
-- dev brain can be enabled/disabled in preferences to reduce system prompt
+for later agents to retrieve the relevant context.
+
+This is not a general Markdown memory store. The vault contains coherent topic
+pages with explicit provenance, links, and freshness state. Queries use the
+compiled wiki when it is verified and return to project sources when knowledge
+is missing or stale. The vault remains ordinary Obsidian Markdown so that the
+user can inspect, navigate, edit, and version it without DS4Server.
+
+### Configuration and ownership
+
+- Preferences provide an enable switch and the path to one dedicated Obsidian
+  vault. When disabled, Dev Brain contributes neither tools nor instructions to
+  the agent prompt.
+- A vault must be an existing directory containing `.obsidian`. DS4Server may
+  change only its declared wiki files and must never modify Obsidian settings,
+  attachments, trash, hidden files, or unrelated user notes.
+- The existing configuration store persists the setting. Changing the vault or
+  enable state invalidates the current Dev Brain context and derived index.
+
+### Vault contract
+
+The vault is self-describing and contains:
+
+- `purpose.md`, maintained with the user, defining the wiki's scope, priorities,
+  and recurring questions;
+- `schema.md`, defining page types, naming and linking conventions, provenance,
+  freshness states, and the rules for compilation, querying, and validation;
+- `index.md`, a concise catalog of wiki pages with one-line descriptions;
+- `log.md`, an append-only record of material wiki updates and their source
+  revisions; and
+- topic pages grouped under `projects/`, `subsystems/`, `concepts/`,
+  `decisions/`, `invariants/`, and `workflows/` as the content requires.
+
+Every managed topic page has YAML frontmatter containing its page type, project
+identity, `verified`, `stale`, or `needs-review` status, verification time, and
+source records. A source record identifies the registered project, a
+repo-relative path, an optional symbol, and the exact evidence version: the Git
+revision for a clean worktree and a content hash for the files actually read.
+The body uses normal Obsidian links and embeds to connect related knowledge.
+Pages model useful concepts rather than mirroring every source file.
+
+### Compilation and maintenance
+
+Initial compilation examines high-signal project material such as manifests,
+documentation, schemas, entry points, public interfaces, and tests, then writes
+the smallest set of durable topic pages that answers the purpose of the wiki.
+It does not attempt to summarize every file.
+
+Refresh compares each page's recorded evidence with the current project state.
+Changed source paths and hashes select the dependent pages to re-read and
+revalidate. The agent may update, split, merge, or retire those pages and keeps
+`index.md` and `log.md` consistent with the result. Knowledge crystallized from
+a development session must still cite project evidence; unsupported conclusions
+are marked `needs-review` rather than presented as fact.
+
+Wiki changes are prepared as one candidate batch in a temporary location. The
+complete candidate is validated before managed files are replaced. A failed or
+ambiguous update leaves the last valid vault untouched, and publication must
+not expose a partially updated wiki.
+
+### Parsing, indexing, and querying
+
+Use `turbovault-parser` to parse Obsidian-flavored Markdown, including YAML
+frontmatter, headings, wikilinks, embeds, tags, and source positions without
+treating link-like text inside code as graph edges. Keep the Markdown files as
+the source of truth and build a disposable SQLite index using the database
+support already present in DS4Server:
+
+- page path, title, type, status, headings, tags, and searchable body text;
+- resolved wikilink and embed edges;
+- project-source-to-page provenance edges; and
+- an FTS5 index for ranked full-text search.
+
+The index is rebuilt when a vault is enabled and refreshed from file metadata
+and content hashes before use. Querying starts with `index.md` or FTS, follows
+relevant links and backlinks, and returns both wiki pages and their project
+evidence. Stale or unverified pages are labeled and excluded from authoritative
+answers until the underlying sources have been checked.
+
+The first implementation deliberately uses no filesystem watcher, embeddings,
+vector database, or separate search service. Explicit refresh plus SQLite FTS5
+is sufficient until vault size or measured retrieval quality proves otherwise.
+
+### Validation
+
+Validation prevents the wiki from silently becoming a confident but obsolete
+source:
+
+- structural validation checks frontmatter, page types, required fields, and
+  the managed path boundary;
+- graph validation checks that links resolve, page identities are unique, and
+  `index.md` matches the managed pages;
+- provenance validation checks that every cited project and source exists and
+  that paths remain inside the registered project;
+- freshness validation checks recorded revisions and hashes against the current
+  files; and
+- semantic validation has the agent re-check affected claims against changed
+  code, documentation, and tests before marking a page verified.
+
+Semantic validation cannot prove every statement correct, so uncertainty must
+remain visible as `needs-review`. Agents may rely on `verified` pages, may use
+`stale` pages only as leads, and must not turn unsupported wiki text into project
+truth.
+
+### Agent integration and acceptance
+
+Dev Brain is a separate wiki maintenance and retrieval capability, not generic
+filesystem access outside the active project. Its operations expose search and
+read over the derived index plus validated batch publication of managed wiki
+content. Detailed operating instructions live in `schema.md`; the system prompt
+only announces Dev Brain when enabled and directs the agent to that contract.
+
+The implementation is complete when tests cover vault path confinement,
+Obsidian parsing and index rebuilding, detection of source-hash drift, broken
+links and invalid provenance, exclusion of stale pages from authoritative
+queries, all-or-nothing publication, and prompt gating when Dev Brain is
+disabled.