DS4Server/PLAN.md

DS4Server implementation plan

Only unfinished implementation work belongs here. DwarfStar remains the behavioral oracle for model execution, token processing, context accounting, KV-cache behavior, the HTTP API, and the built-in agent loop.

Dev Brain: a project-backed LLM Wiki

Dev Brain implements the LLM Wiki idea described by Andrej Karpathy. The registered projects remain the authoritative, changing source material. A user-selected Obsidian vault is the derived, human-readable body of knowledge that the agent maintains from those projects. Its purpose is to compile durable knowledge about architecture, behavior, decisions, invariants, workflows, and relationships once, keep it aligned as the projects change, and make it cheap for later agents to retrieve the relevant context.

This is not a general Markdown memory store. The vault contains coherent topic pages with explicit provenance, links, and freshness state. Queries use the compiled wiki when it is verified and return to project sources when knowledge is missing or stale. The vault remains ordinary Obsidian Markdown so that the user can inspect, navigate, edit, and version it without DS4Server.

Configuration and ownership

• Preferences provide an enable switch and the path to one dedicated Obsidian vault. When disabled, Dev Brain contributes neither tools nor instructions to the agent prompt.
• A vault must be an existing directory containing .obsidian. DS4Server may change only its declared wiki files and must never modify Obsidian settings, attachments, trash, hidden files, or unrelated user notes.
• The existing configuration store persists the setting. Changing the vault or enable state invalidates the current Dev Brain context and derived index.

Vault contract

The vault is self-describing and contains:

• purpose.md, maintained with the user, defining the wiki's scope, priorities, and recurring questions;
• schema.md, defining page types, naming and linking conventions, provenance, freshness states, and the rules for compilation, querying, and validation;
• index.md, a concise catalog of wiki pages with one-line descriptions;
• log.md, an append-only record of material wiki updates and their source revisions; and
• topic pages grouped under projects/, subsystems/, concepts/, decisions/, invariants/, and workflows/ as the content requires.

Every managed topic page has YAML frontmatter containing its page type, project identity, verified, stale, or needs-review status, verification time, and source records. A source record identifies the registered project, a repo-relative path, an optional symbol, and the exact evidence version: the Git revision for a clean worktree and a content hash for the files actually read. The body uses normal Obsidian links and embeds to connect related knowledge. Pages model useful concepts rather than mirroring every source file.

Compilation and maintenance

Initial compilation examines high-signal project material such as manifests, documentation, schemas, entry points, public interfaces, and tests, then writes the smallest set of durable topic pages that answers the purpose of the wiki. It does not attempt to summarize every file.

Refresh compares each page's recorded evidence with the current project state. Changed source paths and hashes select the dependent pages to re-read and revalidate. The agent may update, split, merge, or retire those pages and keeps index.md and log.md consistent with the result. Knowledge crystallized from a development session must still cite project evidence; unsupported conclusions are marked needs-review rather than presented as fact.

Wiki changes are prepared as one candidate batch in a temporary location. The complete candidate is validated before managed files are replaced. A failed or ambiguous update leaves the last valid vault untouched, and publication must not expose a partially updated wiki.

Parsing, indexing, and querying

Use turbovault-parser to parse Obsidian-flavored Markdown, including YAML frontmatter, headings, wikilinks, embeds, tags, and source positions without treating link-like text inside code as graph edges. Keep the Markdown files as the source of truth and build a disposable SQLite index using the database support already present in DS4Server:

• page path, title, type, status, headings, tags, and searchable body text;
• resolved wikilink and embed edges;
• project-source-to-page provenance edges; and
• an FTS5 index for ranked full-text search.

The index is rebuilt when a vault is enabled and refreshed from file metadata and content hashes before use. Querying starts with index.md or FTS, follows relevant links and backlinks, and returns both wiki pages and their project evidence. Stale or unverified pages are labeled and excluded from authoritative answers until the underlying sources have been checked.

The first implementation deliberately uses no filesystem watcher, embeddings, vector database, or separate search service. Explicit refresh plus SQLite FTS5 is sufficient until vault size or measured retrieval quality proves otherwise.

Validation

Validation prevents the wiki from silently becoming a confident but obsolete source:

• structural validation checks frontmatter, page types, required fields, and the managed path boundary;
• graph validation checks that links resolve, page identities are unique, and index.md matches the managed pages;
• provenance validation checks that every cited project and source exists and that paths remain inside the registered project;
• freshness validation checks recorded revisions and hashes against the current files; and
• semantic validation has the agent re-check affected claims against changed code, documentation, and tests before marking a page verified.

Semantic validation cannot prove every statement correct, so uncertainty must remain visible as needs-review. Agents may rely on verified pages, may use stale pages only as leads, and must not turn unsupported wiki text into project truth.

Agent integration and acceptance

Dev Brain is a separate wiki maintenance and retrieval capability, not generic filesystem access outside the active project. Its operations expose search and read over the derived index plus validated batch publication of managed wiki content. Detailed operating instructions live in schema.md; the system prompt only announces Dev Brain when enabled and directs the agent to that contract.

The implementation is complete when tests cover vault path confinement, Obsidian parsing and index rebuilding, detection of source-hash drift, broken links and invalid provenance, exclusion of stale pages from authoritative queries, all-or-nothing publication, and prompt gating when Dev Brain is disabled.