feat: first cut at dev brain

This commit is contained in:
Georg Bauer
2026-07-27 21:16:10 +02:00
parent 56be87d288
commit cfc4fce8d7
14 changed files with 2119 additions and 141 deletions

126
PLAN.md
View File

@@ -3,129 +3,3 @@
Only unfinished implementation work belongs here. DwarfStar remains the
behavioral oracle for model execution, token processing, context accounting,
KV-cache behavior, the HTTP API, and the built-in agent loop.
## Dev Brain: a project-backed LLM Wiki
Dev Brain implements the LLM Wiki idea described by Andrej Karpathy. The
registered projects remain the authoritative, changing source material. A
user-selected Obsidian vault is the derived, human-readable body of knowledge
that the agent maintains from those projects. Its purpose is to compile durable
knowledge about architecture, behavior, decisions, invariants, workflows, and
relationships once, keep it aligned as the projects change, and make it cheap
for later agents to retrieve the relevant context.
This is not a general Markdown memory store. The vault contains coherent topic
pages with explicit provenance, links, and freshness state. Queries use the
compiled wiki when it is verified and return to project sources when knowledge
is missing or stale. The vault remains ordinary Obsidian Markdown so that the
user can inspect, navigate, edit, and version it without DS4Server.
### Configuration and ownership
- Preferences provide an enable switch and the path to one dedicated Obsidian
vault. When disabled, Dev Brain contributes neither tools nor instructions to
the agent prompt.
- A vault must be an existing directory containing `.obsidian`. DS4Server may
change only its declared wiki files and must never modify Obsidian settings,
attachments, trash, hidden files, or unrelated user notes.
- The existing configuration store persists the setting. Changing the vault or
enable state invalidates the current Dev Brain context and derived index.
### Vault contract
The vault is self-describing and contains:
- `purpose.md`, maintained with the user, defining the wiki's scope, priorities,
and recurring questions;
- `schema.md`, defining page types, naming and linking conventions, provenance,
freshness states, and the rules for compilation, querying, and validation;
- `index.md`, a concise catalog of wiki pages with one-line descriptions;
- `log.md`, an append-only record of material wiki updates and their source
revisions; and
- topic pages grouped under `projects/`, `subsystems/`, `concepts/`,
`decisions/`, `invariants/`, and `workflows/` as the content requires.
Every managed topic page has YAML frontmatter containing its page type, project
identity, `verified`, `stale`, or `needs-review` status, verification time, and
source records. A source record identifies the registered project, a
repo-relative path, an optional symbol, and the exact evidence version: the Git
revision for a clean worktree and a content hash for the files actually read.
The body uses normal Obsidian links and embeds to connect related knowledge.
Pages model useful concepts rather than mirroring every source file.
### Compilation and maintenance
Initial compilation examines high-signal project material such as manifests,
documentation, schemas, entry points, public interfaces, and tests, then writes
the smallest set of durable topic pages that answers the purpose of the wiki.
It does not attempt to summarize every file.
Refresh compares each page's recorded evidence with the current project state.
Changed source paths and hashes select the dependent pages to re-read and
revalidate. The agent may update, split, merge, or retire those pages and keeps
`index.md` and `log.md` consistent with the result. Knowledge crystallized from
a development session must still cite project evidence; unsupported conclusions
are marked `needs-review` rather than presented as fact.
Wiki changes are prepared as one candidate batch in a temporary location. The
complete candidate is validated before managed files are replaced. A failed or
ambiguous update leaves the last valid vault untouched, and publication must
not expose a partially updated wiki.
### Parsing, indexing, and querying
Use `turbovault-parser` to parse Obsidian-flavored Markdown, including YAML
frontmatter, headings, wikilinks, embeds, tags, and source positions without
treating link-like text inside code as graph edges. Keep the Markdown files as
the source of truth and build a disposable SQLite index using the database
support already present in DS4Server:
- page path, title, type, status, headings, tags, and searchable body text;
- resolved wikilink and embed edges;
- project-source-to-page provenance edges; and
- an FTS5 index for ranked full-text search.
The index is rebuilt when a vault is enabled and refreshed from file metadata
and content hashes before use. Querying starts with `index.md` or FTS, follows
relevant links and backlinks, and returns both wiki pages and their project
evidence. Stale or unverified pages are labeled and excluded from authoritative
answers until the underlying sources have been checked.
The first implementation deliberately uses no filesystem watcher, embeddings,
vector database, or separate search service. Explicit refresh plus SQLite FTS5
is sufficient until vault size or measured retrieval quality proves otherwise.
### Validation
Validation prevents the wiki from silently becoming a confident but obsolete
source:
- structural validation checks frontmatter, page types, required fields, and
the managed path boundary;
- graph validation checks that links resolve, page identities are unique, and
`index.md` matches the managed pages;
- provenance validation checks that every cited project and source exists and
that paths remain inside the registered project;
- freshness validation checks recorded revisions and hashes against the current
files; and
- semantic validation has the agent re-check affected claims against changed
code, documentation, and tests before marking a page verified.
Semantic validation cannot prove every statement correct, so uncertainty must
remain visible as `needs-review`. Agents may rely on `verified` pages, may use
`stale` pages only as leads, and must not turn unsupported wiki text into project
truth.
### Agent integration and acceptance
Dev Brain is a separate wiki maintenance and retrieval capability, not generic
filesystem access outside the active project. Its operations expose search and
read over the derived index plus validated batch publication of managed wiki
content. Detailed operating instructions live in `schema.md`; the system prompt
only announces Dev Brain when enabled and directs the agent to that contract.
The implementation is complete when tests cover vault path confinement,
Obsidian parsing and index rebuilding, detection of source-hash drift, broken
links and invalid provenance, exclusion of stale pages from authoritative
queries, all-or-nothing publication, and prompt gating when Dev Brain is
disabled.