From d88aaf111f8cdb597d1f9a50f628bb888ba96e01 Mon Sep 17 00:00:00 2001 From: Chili Palmer Date: Sat, 18 Jul 2026 21:50:27 +0200 Subject: [PATCH] chore: cleaned up documents about plan and status --- AGENTS.md | 1 + README.md | 40 ++ RUST_COMPATIBILITY_MATRIX_TEMPLATE.md | 98 --- RUST_EXECUTION_BACKLOG.md | 456 ------------ RUST_PLAN.md | 120 ---- RUST_PLAN_CORE.md | 984 +++----------------------- RUST_PLAN_EXTENSION.md | 291 +++----- SPECIFICATION_INDEX.md | 7 +- docs/COMPATIBILITY_INVENTORY.md | 106 --- 9 files changed, 229 insertions(+), 1874 deletions(-) create mode 100644 README.md delete mode 100644 RUST_COMPATIBILITY_MATRIX_TEMPLATE.md delete mode 100644 RUST_EXECUTION_BACKLOG.md delete mode 100644 RUST_PLAN.md delete mode 100644 docs/COMPATIBILITY_INVENTORY.md diff --git a/AGENTS.md b/AGENTS.md index ec7364e..1087b38 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,6 +25,7 @@ Invariants and behaviours in the allium spec should be covered by unit tests of ## Important facts +- update `README.md` whenever a user-visible feature is added, removed, or materially changed; keep it a compact overview with relevant pointers - published posts don't have body in the database, the body content is only in the file - functionality you implement have to be tied to UI - UI you implement has to be tied to functionality diff --git a/README.md b/README.md new file mode 100644 index 0000000..ca4d375 --- /dev/null +++ b/README.md @@ -0,0 +1,40 @@ +# RuDS + +RuDS is a native Rust blogging desktop application and the successor to bDS2. It manages local projects from authoring through preview, static-site generation, integrity checks, and publishing while preserving the existing bDS filesystem and SQLite formats. + +The project is under active development. Core blogging workflows are broadly available; remaining core work and optional extensions are tracked separately. + +## Available Features + +- Native Iced desktop workspace with localized menus, tabs, sidebars, dialogs, tasks, and embedded Wry previews. +- Post and translation authoring with draft/published lifecycle, metadata, tags, categories, links, and media. +- Media import, thumbnails, metadata translations, filters, validation, and post assignment. +- Template and Lua script management using a custom Ropey/Syntect/Cosmic Text editor. +- SQLite and filesystem persistence with frontmatter, sidecars, rebuild, metadata diff/repair, and FTS5 search. +- Markdown/Liquid rendering with native macros, multilingual routes, feeds, sitemap, Pagefind, and incremental site generation. +- Local preview in the app or system browser. +- Optional one-shot AI translation, description, analysis, taxonomy, and language-detection operations using online or local OpenAI-compatible endpoints with airplane-mode gating. +- SSH-agent-based SCP or rsync publishing. +- Site, media, and translation validation plus Blogmark capture and Lua transforms. + +RuDS uses no JavaScript application runtime and loads no CSS or JavaScript from CDNs. The preview is served by the Rust application and displayed by the operating-system webview. + +## Repository Map + +- `crates/bds-core` — data, engines, rendering, AI, publishing, and Lua +- `crates/bds-editor` — reusable syntax-highlighting editor +- `crates/bds-ui` — desktop application and platform integration +- `crates/bds-cli` — planned automation CLI +- `specs` — authoritative Allium behavior specifications +- `fixtures` — compatibility projects and generated-site fixtures +- `locales` — UI and native-menu translations + +## Plans and References + +- [Core plan and current status](RUST_PLAN_CORE.md) +- [Extension plan and current status](RUST_PLAN_EXTENSION.md) +- [Specification index](SPECIFICATION_INDEX.md) +- [UI style guide](docs/UI_STYLE_GUIDE.md) +- `../bDS2` — reference implementation when an Allium contract is ambiguous + +Contributor workflow and project invariants are documented in [AGENTS.md](AGENTS.md). diff --git a/RUST_COMPATIBILITY_MATRIX_TEMPLATE.md b/RUST_COMPATIBILITY_MATRIX_TEMPLATE.md deleted file mode 100644 index 673fc07..0000000 --- a/RUST_COMPATIBILITY_MATRIX_TEMPLATE.md +++ /dev/null @@ -1,98 +0,0 @@ -# bDS Rust Rewrite — Compatibility Matrix Template - -## Purpose - -This document is the required inventory of persisted fields and compatibility-sensitive behaviors. Fill it before engine implementation starts, then keep it current as the rewrite proceeds. - -One row per persisted field or compatibility-sensitive artifact. - -## How To Use - -1. Start with posts, translations, media, templates, project metadata, menus, and generated outputs. -2. Record the baseline (bDS2, `../bDS2/`) behavior first; the allium specs in `specs/` are the contract. -3. Record the intended Rust behavior only if it is identical or explicitly approved as a divergence. -4. Link every row to tests or golden fixtures once available. - -## Field Matrix - -| Domain | Entity | Field / Artifact | Type | Current Source Of Truth | Persisted In | Read Path | Write Path | Rebuild From Files | Metadata Diff | Publish Impact | Generation Impact | Rust Status | Fixture / Test | Notes | -|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| -| posts | Post | id | string | DB + frontmatter | DB, frontmatter | | | | | | | not-started | | | -| posts | Post | title | string | | | | | | | | | not-started | | | -| posts | Post | slug | string | | | | | | | | | not-started | | | -| posts | Post | excerpt | string? | | | | | | | | | not-started | | | -| posts | Post | content body | markdown | published body in file; draft body per current app rules | DB?, markdown file | | | | | | | not-started | | | -| posts | Post | status | enum | | | | | | | | | not-started | | | -| posts | Post | author | string? | | | | | | | | | not-started | | | -| posts | Post | language | string? | | | | | | | | | not-started | | | -| posts | Post | do_not_translate | bool | | | | | | | | | not-started | | | -| posts | Post | tags | list | | | | | | | | | not-started | | | -| posts | Post | categories | list | | | | | | | | | not-started | | | -| posts | Post | template_slug | string? | | | | | | | | | not-started | | | -| posts | Post | checksum | string? | | | | | | | | | not-started | | | -| translations | PostTranslation | translation_for | string | | | | | | | | | not-started | | | -| media | Media | alt | string? | | | | | | | | | not-started | | | -| media | Media | caption | string? | | | | | | | | | not-started | | | -| media | MediaTranslation | translationFor | string | | DB | | | | | | | not-started | | | -| media | MediaTranslation | language | string | | DB | | | | | | | not-started | | | -| media | MediaTranslation | title | string? | | DB | | | | | | | not-started | | | -| media | MediaTranslation | alt | string? | | DB | | | | | | | not-started | | | -| media | MediaTranslation | caption | string? | | DB | | | | | | | not-started | | | -| posts | PostLink | sourcePostId | string | | DB | | | | | | | not-started | | | -| posts | PostLink | targetPostId | string | | DB | | | | | | | not-started | | | -| posts | PostLink | linkText | string? | | DB | | | | | | | not-started | | | -| templates | Template | file content | liquid | | | | | | | | | not-started | | | -| scripts | Script | runtime | runtime | Python in current app, Lua in Rust app | app behavior | | | n/a | n/a | yes | yes | approved-divergence | | Explicit break | -| scripts | BlogmarkTransform | transform chain | runtime | Python in current app | app behavior | | | n/a | n/a | no | no | deferred | | Deferred to extension Bucket H | -| menus | MenuDocument | source document | opml/xml | | | | | | | | | not-started | | | -| generation | SiteOutput | route path | path | | generated files | | | n/a | n/a | yes | yes | not-started | | | -| generation | Feed | rss/atom output | xml | | generated files | | | n/a | n/a | yes | yes | not-started | | | -| generation | Sitemap | sitemap output | xml | | generated files | | | n/a | n/a | yes | yes | not-started | | | -| generation | GeneratedFileHash | content hash | string | DB | DB | | | n/a | n/a | yes | yes | not-started | | Skip-unchanged-writes optimization | -| search | PostsFts | FTS5 index | virtual table | DB | DB (FTS5) | | | rebuild | n/a | no | no | not-started | | | -| search | MediaFts | FTS5 index | virtual table | DB | DB (FTS5) | | | rebuild | n/a | no | no | not-started | | | -| search | ClientSearch | search index | binary | generated files | generated files | | | n/a | n/a | yes | yes | not-started | | Determine in M0: integrate via `pagefind` crate library API | -| media | Thumbnail | generated thumbnail | image file | filesystem | filesystem | | | regenerate | n/a | yes | yes | not-started | | `image` crate; verify size/format matches current `sharp` output | -| macros | BuiltInMacro | gallery | Rust-native | app code | n/a | | | n/a | n/a | no | yes | not-started | | Was JS in TS app; Rust in Rust app | -| macros | BuiltInMacro | youtube | Rust-native | app code | n/a | | | n/a | n/a | no | yes | not-started | | Was JS in TS app; Rust in Rust app | -| macros | BuiltInMacro | vimeo | Rust-native | app code | n/a | | | n/a | n/a | no | yes | not-started | | Was JS in TS app; Rust in Rust app | -| macros | BuiltInMacro | photo_archive | Rust-native | app code | n/a | | | n/a | n/a | no | yes | not-started | | Was JS in TS app; Rust in Rust app | -| macros | BuiltInMacro | tag_cloud | Rust-native | app code | n/a | | | n/a | n/a | no | yes | not-started | | Was JS in TS app; Rust in Rust app | - -## Behavior Matrix - -Use this for compatibility-sensitive behaviors that are not a single persisted field. - -| Area | Behavior | Current App Rule | Rust Target Rule | Allowed Divergence | Validation Method | Status | Notes | -|---|---|---|---|---|---|---|---| -| posts | published body storage | body not stored in DB when published | identical | no | fixture DB + file assertions | not-started | | -| localization | UI locale | follows OS locale | identical | no | manual + integration test | not-started | | -| localization | render language | follows project setting | identical | no | preview/generation tests | not-started | | -| menus | native menu routing | current app uses native menus | native menus via muda (cross-platform: NSMenu, Win32, GTK) | implementation changes, behavior identical | menu integration tests | not-started | | -| scripts | user Python scripts | supported in current app | unsupported in Rust app | yes | migration note + runtime detection tests | approved-divergence | | -| rendering | built-in macros | JS server-side (gallery, youtube, vimeo, photo_archive, tag_cloud) | Rust-native in bds-core/render | implementation language changes, output must match | golden-generated-site comparisons | not-started | These are NOT Python macros | -| rendering | Liquid feature subset | liquidjs 10.25 (full spec available) | Rust Liquid (scoped to used subset) | implementation may differ for unused features | template compatibility suite | not-started | Only ~35% of spec used by default templates | -| slugs | slug generation | `transliteration` npm package | `deunicode` Rust crate | possible edge-case differences | slug corpus tests in M0 fixtures | not-started | Verify against real content | -| editor | content editing | Markdown editor + rendered preview | syntax-highlighting Markdown editor (bds-editor: ropey + syntect + cosmic-text) + rendered preview | no | editor + preview integration tests | not-started | | -| preview | asset sourcing | local assets only | identical | no | HTML assertions | not-started | | -| runtime | JS dependency | Electron (Chromium + Node.js) | no JavaScript anywhere — pure Rust + native APIs | yes (intentional) | build verification: no JS in dependency tree | approved-divergence | Supply-chain security constraint | -| runtime | async executor | Node.js event loop | tokio | yes (internal) | n/a | approved-divergence | Used for preview server, publish, file watching | -| media | thumbnail generation | `sharp` (libvips) | `image` crate | output dimensions and format must match | golden-file comparison of thumbnails | not-started | Fallback to `libvips-rs` if quality/perf insufficient | -| generation | client-side search | determine in M0 | `pagefind` crate library API (`PagefindIndex`) | output must match | golden-generated-site comparisons | not-started | Rust library dep, no CLI binary | -| generation | parallel rendering | single-threaded in current app | `rayon` for parallel page rendering | yes (faster, same output) | golden-generated-site comparisons | approved-divergence | Output must be identical regardless of parallelism | -| ai | one-shot AI operations | not in current app | `reqwest` against configurable OpenAI-compatible endpoint | yes (new feature) | mocked endpoint tests | approved-divergence | Translation, alt text, title suggestion. Entirely optional — app works without endpoint configured | - -## Status Values - -- `not-started` -- `in-progress` -- `verified` -- `approved-divergence` -- `blocked` - -## Sign-Off Checklist - -- every persisted field used by core is represented -- every approved divergence is explicit -- every row links to a fixture or test before release -- metadata diff, rebuild, and publish implications are filled in for all relevant rows diff --git a/RUST_EXECUTION_BACKLOG.md b/RUST_EXECUTION_BACKLOG.md deleted file mode 100644 index c3bdf6a..0000000 --- a/RUST_EXECUTION_BACKLOG.md +++ /dev/null @@ -1,456 +0,0 @@ -# bDS Rust Rewrite — Execution Backlog - -## Purpose - -This document converts the core and extension plans into execution order. It is organized by milestone first and by crate second. - -Rules: - -1. Do not pull work forward from a later milestone unless it directly unblocks the current milestone. -2. Keep tasks vertically sliced where possible: engine + UI + tests. -3. Compatibility tasks outrank polish tasks. -4. The allium specs in `specs/` (synced from `../bDS2/specs/` on 2026-07-18) are the behaviour contract. When a task references a spec file, implement what the spec says and cover its invariants with unit tests. - -## Status (2026-07-18) - -- M0–M2: complete (items marked DONE below). -- M3: implemented (all views and the editor exist in `crates/bds-ui/src/views/`); items below are not individually ticked — verify against tests before re-doing anything. -- M4: in progress per git history ("M4 closing" commits). `bds-core/src/render/` exists (generation, macros, markdown, page renderer, routes, site, template lookup). `bds-core/src/ai/` does **not** exist yet — the one-shot AI client is unstarted. -- M5: not started (`bds-core/src/scripting/` is an empty stub; no PublishEngine). -- M6 (below): new — code-vs-spec gaps introduced by the bDS2 spec sync. Fold these in as the milestones they touch come up; none of them should wait for "later" if the code they touch is being written now. - -## Milestone M0: Compatibility Baseline - -### `bds-core` - -- ~~create workspace and crate boundaries (bds-core, bds-editor, bds-ui, bds-cli)~~ **DONE** -- ~~add typed SQLite connection and query layer via `diesel`~~ **DONE** -- ~~add embedded migration loader via `diesel_migrations`~~ **DONE** -- ~~define initial shared model modules with `serde` derives~~ **DONE** -- ~~add checksum (`sha2`) and slug (`deunicode`) utilities~~ **DONE** -- ~~establish error handling conventions: `thiserror` for bds-core, `anyhow` for bds-ui/bds-cli~~ **DONE** -- ~~add `tokio` runtime as workspace dependency (used by bds-ui and bds-cli, not directly by bds-core engine code)~~ **DONE** - -### `bds-editor` - -- ~~create crate with ropey, syntect, cosmic-text dependencies~~ **DONE** -- ~~implement basic rope buffer wrapper with edit operations~~ **DONE** -- ~~implement syntect integration for markdown highlighting~~ **DONE** -- ~~implement cosmic-text layout for rendering highlighted text~~ **DONE** (measured monospace font metrics via cosmic-text FontSystem/Buffer; OnceLock-cached MonoMetrics replaces hardcoded constants) -- ~~implement basic cursor model (position, move, place by click)~~ **DONE** (up/down/left/right/home/end + click placement) -- ~~implement basic text insertion and deletion~~ **DONE** (insert, backspace, delete forward, enter) -- ~~implement Iced custom widget that composes buffer + highlight + layout + cursor~~ **DONE** (CodeEditor widget with gutter, text, cursor rendering + keyboard/mouse events) -- ~~implement basic vertical scrolling with viewport-aware rendering~~ **DONE** (scroll_by, ensure_cursor_visible, mouse wheel, viewport-clipped rendering) -- ~~verify IME composition events work through winit (early risk check)~~ **DONE** (verified: winit/Iced delivers composed chars via Key::Character; pre-edit display deferred to M3) - -### `bds-ui` - -- ~~create app entry point with Iced `Application` impl~~ **DONE** -- ~~wire Iced window creation~~ **DONE** -- ~~wire muda menu bar with skeleton App/File/Edit/View/Window/Help menus~~ **DONE** (stub) -- ~~wire macOS lifecycle shim via objc2 (application:openFile:, application:openURLs:) behind cfg gate~~ **DONE** (stub) -- ~~wire muda `MenuEvent` receiver as Iced `Subscription`~~ **DONE** (platform/menu.rs menu_subscription + app.rs subscription) - -### `fixtures` and `docs` - -- ~~collect representative fixture projects from current app~~ **DONE** (fixtures/compatibility-projects/rfc1437-sample/) -- ~~capture golden generated output for those fixtures~~ **DONE** (fixtures/golden-generated-sites/rfc1437-sample/ — subset from live TypeScript-generated site: 3 post pages × 2 languages, structural files, assets, feeds, sitemap, category page) -- ~~create initial compatibility inventory from the matrix template (must include `mediaTranslations`, `postLinks`, FTS5 tables)~~ **DONE** (docs/COMPATIBILITY_INVENTORY.md) -- ~~create Liquid feature inventory from default templates~~ **DONE** (docs/LIQUID_FEATURE_INVENTORY.md) -- ~~determine whether current app uses Pagefind or another client-side search index — if Pagefind, plan for `pagefind` crate library integration (not CLI binary)~~ **DONE** (confirmed: TypeScript app uses pagefind_extended CLI; Rust will use pagefind crate library API) -- ~~create slug compatibility test suite comparing `deunicode` vs `transliteration` output on fixture content~~ **DONE** -- ~~create Iced architecture patterns document (message design, subscription model, custom widget patterns)~~ **DONE** (docs/ICED_ARCHITECTURE_PATTERNS.md) - -### Validation - -- ~~DB readability tests (all tables including `mediaTranslations`, `postLinks`, FTS5, and AI/catalog tables)~~ **DONE** (59 unit tests + 29 fixture integration tests) -- ~~app launch smoke test (Iced window + muda menus)~~ **DONE** (tests/app_smoke.rs — type-level tests; muda requires main thread so full launch tested via `cargo run`) -- ~~bds-editor PoC test: renders highlighted markdown, accepts keyboard input, cursor moves~~ **DONE** (tests/editor_poc.rs — 11 integration tests covering highlight + input + cursor + scroll) -- ~~fixture harness test~~ **DONE** (tests/fixture_readability.rs) -- ~~slug compatibility tests~~ **DONE** - -## Milestone M1: Data Fidelity - -### `bds-core/db` - -- ~~verify schema compatibility against existing projects~~ **DONE** -- ~~add FTS5 virtual tables (`posts_fts`, `media_fts`) for in-app full-text search~~ **DONE** (multi-column schema: title/excerpt/content/tags/categories with per-language snowball stemming, 24 languages) -- ~~verify `mediaTranslations` table read/write~~ **DONE** -- ~~verify `postLinks` table read/write~~ **DONE** - -### `bds-core/engine` - -- ~~implement `ProjectEngine`~~ **DONE** -- ~~implement `PostEngine`~~ **DONE** -- ~~implement `MediaEngine`~~ **DONE** -- ~~implement `PostMediaEngine`~~ **DONE** -- ~~implement `TagEngine`~~ **DONE** -- ~~implement `MetaEngine`~~ **DONE** -- ~~implement metadata diff flow (posts, media, scripts, templates)~~ **DONE** -- ~~implement rebuild-from-filesystem flow~~ **DONE** - -### `bds-core/util` - -- ~~frontmatter parser/writer via `serde_yaml`~~ **DONE** -- ~~translation file parser/writer~~ **DONE** -- ~~sidecar parser/writer~~ **DONE** -- ~~content hashing support via `sha2`~~ **DONE** -- ~~thumbnail generation via `image` crate (resize, WEBP encoding)~~ **DONE** -- ~~recursive directory traversal via `walkdir` (for rebuild-from-filesystem)~~ **DONE** - -### Validation - -- ~~round-trip persistence tests~~ **DONE** (38 m1_validation integration tests) -- ~~rebuild tests~~ **DONE** -- ~~metadata diff tests~~ **DONE** (full coverage matrix: 13 post fields, 6 media, 4 template, 5 script) -- ~~golden-file comparisons for written files~~ **DONE** - -## Milestone M2: Native Workspace - -### `bds-ui/platform` - -- ~~muda menu bar: full menu tree with accelerators~~ **DONE** (platform/menu.rs with muda MenuBar, menu_subscription, accelerators) -- ~~menu enable/disable validation synced to app state~~ **DONE** (sync_menu_state() evaluates has_project/has_tab/offline_mode after every state change) -- ~~menu event routing as `Message` variants~~ **DONE** (MenuEvent → Message mapping in app.rs subscription) -- ~~rfd integration for file/folder open and save dialogs~~ **DONE** (platform/dialog.rs with i18n-aware pick_folder/pick_media_files) -- ~~macOS lifecycle shim: file-open and URL-open events forwarded as `Message` variants~~ **DONE** (objc2 BdsAppDelegate with application:openFile: and application:openURLs: → mpsc channel → Message::FileOpenRequested/UrlOpenRequested) - -### `bds-ui/app` - -- ~~root `Message` enum and `update()` dispatcher~~ **DONE** (Message enum with menu, lifecycle, sidebar, tab, editor variants + update dispatcher) -- ~~app state model~~ **DONE** (AppState with project, sidebar, editor, output panel state; sidebar post list with 500-post limit) -- ~~task surface model~~ **DONE** (engine/task.rs TaskManager with concurrency limit, FIFO queue, progress reporting, cancel support) - -### `bds-ui/views` - -- ~~workspace layout~~ **DONE** (three-panel layout: sidebar + editor + optional right panel) -- ~~sidebar~~ **DONE** (post list with search, status filter, category/tag counts) -- ~~activity bar~~ **DONE** (VS Code-style 50px bar with SVG icons for Posts/Pages/Media/Scripts/Templates/Tags/Chat/Import/Git/Settings) -- ~~tab bar~~ **DONE** (editor tab bar with open/close/switch) -- ~~status bar~~ **DONE** (bottom bar with project name, task progress) -- ~~project selector~~ **DONE** (project open dialog and recent projects) -- ~~toast notifications~~ **DONE** (overlay toast stack with auto-dismiss, 4 severity levels, i18n messages; replaces output-only notifications) - -### Validation - -- ~~menu event → `Message` routing integration tests~~ **DONE** (tests/menu_routing.rs — 4 tests: i18n key prefix, all-locale coverage, action count, no duplicates) -- ~~keyboard shortcut integration tests~~ **DONE** (tests/m2_validation.rs — 15 accelerator-bound actions verified) -- ~~rfd dialog invocation tests~~ **DONE** (tests/m2_validation.rs — dialog i18n keys in all 5 locales) -- ~~fixture project open flow test~~ **DONE** (tests/project_flow.rs — 6 tests: create, switch, delete, directory, meta files) -- ~~toast notification tests~~ **DONE** (tests/m2_validation.rs — toast id monotonicity, levels, expiry, message preservation, i18n keys) -- ~~menu enable/disable rule tests~~ **DONE** (tests/m2_validation.rs — project-gated 11 items, tab-gated 4 items, offline-gated 2 items) - -## Milestone M3: Authoring - -### `bds-editor` (maturation) - -- full cursor movement (arrows, word, line, page, home/end) -- selection (shift+movement, click-and-drag, double-click word select) -- system clipboard integration (copy/cut/paste) -- undo/redo with edit grouping -- line numbers gutter -- incremental syntax rehighlighting on edits -- IME input handling for CJK and other input methods -- configurable soft wrap -- additional syntax grammars: Liquid/HTML, Lua, YAML, JSON - -### `bds-core/engine` - -- expose editor-safe save APIs -- expose publish/unpublish/discard flows -- expose template validation APIs -- expose script validation APIs - -### `bds-ui/views` - -- sidebar post filtering: text search box, status filter, tag/category filter dropdowns, language filter, year/month selectors, date range picker — wired to existing `PostSearchFilters` / `search_posts_filtered()` in bds-core. The 500 post limit applies AFTER filtering, not before. -- dashboard -- post editor (bds-editor with markdown + YAML frontmatter highlighting) -- translation editor (bds-editor) -- media browser -- media editor -- tags view -- settings view -- templates view and editor (bds-editor with Liquid/HTML highlighting) -- scripts view and editor (bds-editor with Lua highlighting) - -### `bds-ui/components` - -- inputs -- modals -- reusable list rows and panels - -### Validation - -- entity create/edit/save flows -- template validation flow -- script validation flow -- editor widget tests: cursor movement, selection, undo/redo, clipboard, IME - -## Milestone M4: Rendering Parity - -### `bds-core/render` - -- markdown render pipeline via `pulldown-cmark` -- Liquid render pipeline via `liquid` crate (scoped to the feature subset documented in the Liquid inventory: `if`/`elsif`/`else`, `for`, `assign`, `render`, whitespace stripping, plus filters: `default`, `escape`, `url_encode`, `append`; `.size` is property access on arrays, not a pipe filter) -- custom Liquid filter: `i18n` (translation lookup by key and language) -- custom Liquid filter: `markdown` (markdown-to-HTML with macro expansion, URL rewriting, media path canonicalization) -- built-in macro renderers: `gallery`, `youtube`, `vimeo`, `photo_archive`, `tag_cloud` (native Rust, not Lua) -- template resolution rules -- URL rewriting -- RSS/Atom feed generation via `quick-xml` -- sitemap generation via `quick-xml` -- generated file hash tracking (`generatedFileHashes` table for skip-unchanged-writes) -- Pagefind search index generation via `pagefind` crate library API (`PagefindIndex::add_html_file()` + `get_files()`) — if required for parity (determine in M0 inventory) - -### `bds-core/ai` - -- one-shot AI client: `reqwest` + `serde_json` against OpenAI-compatible Chat Completions endpoint -- two-endpoint configuration: online endpoint (URL, API key, model) + airplane mode endpoint (URL, model) -- AI translate post operation (title, excerpt, content to target language) -- AI translate media metadata operation (title, alt, caption to target language) -- AI image description / alt text generation operation -- AI post analysis operation (title, excerpt, slug suggestion) -- AI taxonomy analysis operation (tag, category suggestions) -- AI language detection operation -- API key storage via OS keychain (macOS Keychain, Windows DPAPI, Linux libsecret) -- airplane mode gating: block online endpoint, use airplane endpoint or show toast -- error handling: surface failures as user-visible feedback, never silent - -### `bds-core/engine` - -- `TemplateEngine` -- `PageRenderer` (parallelize rendering via `rayon`) -- `BlogGenerationEngine` -- `PreviewServer` (localhost HTTP via `axum` on `tokio`) -- `SearchIndexEngine` only if required for parity - -### `bds-ui/views` - -- internal Markdown/Preview switch backed by Wry, plus external Open in Browser command -- generation progress display -- render errors and diagnostics -- AI operation triggers in post editor (analysis, taxonomy), translation editor (translate), and media editor (alt text, translate) -- AI endpoint configuration in settings view (online + airplane mode endpoints) - -### Validation - -- golden generated-site comparisons -- preview route coverage -- template compatibility suite -- one-shot AI client tests (mocked endpoint: all 6 operations) -- AI endpoint configuration persistence tests -- AI airplane mode gating tests - -## Milestone M5: Operate And Ship - -### `bds-core/engine` - -- `PublishEngine` (SSH/SCP via `ssh2`, rsync via child process) -- validation and integrity services -- filesystem watcher via `notify` (if needed for detecting external changes to open content) - -### `bds-core/scripting` - -- `ScriptEngine` -- `LuaRuntime` via `mlua` (Lua 5.4, vendored) — user-authored scripts only; built-in macros are native Rust in `bds-core/render` -- `LuaApi` (expose post, media, tag, and project data to Lua scripts via `mlua::UserData` trait) -- user-authored Lua macro execution at render time -- user-authored transform and utility script execution -- scripting docs generator - -### `bds-ui/views` - -- publish workflow screens -- publish progress and failure surfaces -- script docs access points if needed for core usability - -### Validation - -- publish end-to-end tests -- Lua API bridge tests -- built-in macro compatibility tests -- docs sync tests - -## Milestone M6: bDS2 Spec Parity - -Code-vs-spec gaps from the 2026-07-18 spec sync. Each task names its authoritative spec. Schema changes go through generated Diesel migrations, and every new metadata field must be wired into publishing, metadata-diff, and rebuild-from-filesystem together. - -### `bds-core/util` - -- slug uniqueness: unbounded numeric suffix `{slug}-2, -3, …` — remove the 999 cap and timestamp fallback in `util/slug.rs` (post.allium) - -### `bds-core/db` - -- migration: add `cache_read_tokens` and `cache_write_tokens` (nullable integers) to `ai_usage` (schema.allium) -- migration: rename `npm` → `package_ref` and align `provider_npm` naming (schema.allium) -- media FTS search: accept structured `filters` alongside the query (search.allium `SearchMediaRequested(query, filters)`) - -### `bds-core/engine` - -- project open: discover `data_path` from the location of `meta/project.json`; never persist it in project.json; keep a machine-local project registry under the OS app-data dir (project.allium `DiscoverProjectDataPath`, `DataPathNotPersistedInProjectJson`) -- verify public/private data placement matches project.allium (`PublicContentLivesInProjectFolder`, `PrivateArtifactsLiveInOsAppDir`): DB, embeddings index, model cache, registry, and window state in the OS app-data dir; everything user-owned in the project folder -- menu: file-only model (no DB table), `HomeAlwaysFirst` normalization, `SyncMenuFromFilesystem` flow (menu.allium) -- media: optional import metadata (title, alt, caption, author, language, tags), `ReplaceMediaFile` flow, batch-import linked-image processing (media.allium) -- metadata diff: compare only translation-specific fields for translation files; per-item repair direction; `embedding` entity type (metadata_diff.allium) -- translations: `ReopenPublishedTranslation` on edit; `TranslationFilesCarryFullMetadata` (translation.allium) -- auto-translation pipeline: `ScheduleAutoTranslation`, `AutoTranslatePost`, `AutoTranslateMediaCascade`, `FillMissingTranslations` — gated by configured AI endpoint and airplane mode, skips `doNotTranslate`, fills only missing languages (translation.allium; depends on M4 AI client) -- task manager: allow cancelling `pending` (queued) tasks, not only running ones (task.allium) - -### `bds-core/render` (fold into M4) - -- implement rendering.allium as the shared render-assigns contract for preview and generation; content language only, never UI locale -- expose `slugify` as a custom Liquid filter alongside `i18n` and `markdown` (rendering.allium) -- generation task structure: task group with one task per section (Site Core, Single Posts, Category/Tag/Date Archives) + final Build Search Index task; stream one progress message per written page — "Generated /url (n/total)" or "Rewrote …" for validation apply (generation.allium) -- enforce `GenerationPublishedOnly` and the specified language variant selection (generation.allium) - -### `bds-core/ai` (fold into M4) - -- honour ai.allium invariants in the one-shot client and settings: endpoint add/remove rules, advisory model catalog with conditional refresh, provider detection, vision capability gate, airplane-mode model swap -- record cache token usage in `ai_usage` (schema.allium) - -### `bds-core/scripting` (fold into M5) - -- configurable `script_extension` (default "lua"): file path `scripts/{slug}.{extension}`; entrypoint default "render" for macros, "main" otherwise (script.allium, frontmatter.allium) -- sandboxed Lua runtime: no filesystem mutation, process control, or package loading; host capabilities only via explicit `bds.*` API (script.allium `SandboxedExecution`, `ExplicitHostCapabilities`) -- enforce config limits: macro timeout 10s; transform toasts max 5/script, 20 total, 300 chars; blogmark title ≤ 200, URL ≤ 2048 (script.allium) - -### `bds-ui` - -- settings: technology section (semantic similarity toggle), data section (rebuild buttons: posts/media/scripts/templates/links/thumbnails/embedding + Open Data Folder), agent integrations (Claude Code + GitHub Copilot; others disabled with "not supported yet" note), image import concurrency (1–8, default 4), default editor mode, settings in a separate tab (editor_settings.allium) -- media search filter UI wired to the filtered media search -- auto-translation and repair-direction UI hooks for the engine flows above - -### Validation - -- unit tests for every invariant and rule named above, per AGENTS.md spec-coverage requirement -- re-run `allium check specs/*.allium` after any spec edit - -## Extension Backlog - -### Bucket A: Git And Validation - -#### `bds-core` - -- `GitEngine` via `git2` (shell out for LFS operations) -- richer site validation service support - -#### `bds-ui` - -- Git sidebar -- diff view -- richer metadata diff UI - -### Bucket B: Import - -#### `bds-core` - -- WXR parser -- import analysis -- import execution -- import definitions - -#### `bds-ui` - -- import wizard -- import progress and result views - -### Bucket C: AI Chat And Tool Use - -#### `bds-core` - -- streaming client extension (SSE parsing on top of core `reqwest` client) -- tool execution framework (tool definitions, call routing, result formatting) -- multi-turn conversation management - -#### `bds-ui` - -- chat sidebar -- chat panel -- provider settings UI (extends core AI endpoint configuration) - -### Bucket D: Embeddings And Duplicates - -#### `bds-core` - -- embedding engine via `ort` (ONNX Runtime) -- vector index via `usearch` (HNSW) -- duplicate detection logic - -#### `bds-ui` - -- semantic search UI -- duplicates view - -### Bucket E: Translation QA And Docs UX - -#### `bds-core` - -- translation validation engine - -#### `bds-ui` - -- translation validation view -- documentation browser - -### Bucket F: Menu Editing And Deep Links - -#### `bds-core` - -- menu editing services -- deep-link flow support beyond core shell hooks - -#### `bds-ui` - -- menu editor UI - -### Bucket G: MCP And Automation - -#### `bds-cli` - -- CLI command set per cli.allium: `rebuild | repair | render | upload | push | pull | post | media | gallery | config | project | lua` — same database, settings, and cache as the GUI; no listeners, no window; logging redirected to the rotating log file -- MCP server surface per mcp.allium (incl. translation tools, stats/media resources, proposal status) -- CLI-to-app synchronization via the domain event bus (events.allium): entity mutations broadcast `(entity, entity_id, action)` on one topic shape covering in-app and external writes - -#### `bds-core` - -- domain event bus (events.allium): broadcast after every successful mutation of posts, media, tags, templates, scripts; `SettingsChangedEvent` for global settings - -### Bucket K: Headless Server Mode (server.allium) - -#### `bds-core` / `bds-cli` - -- boot modes desktop | server | tui resolved from an env var at start -- headless server: no window; SSH transport for remote TUI/GUI clients -- SSH key material generated on first boot in the private app-data dir (host key + empty authorized_keys, mode 600) — never in the repo or project folder - -### Bucket L: Terminal UI (tui.allium) - -#### new crate `bds-tui` (or feature in `bds-cli`) - -- TUI as a second renderer over the same shared UI core: sidebar views (posts/media/templates/scripts/tags/settings/git), sidebar/editor focus model, vim-style + arrow navigation skipping section headers -- post editor drives the same workflow as the GUI: canonical-language edits update the post, other languages update translations; publish routes through the same pipeline -- subscribes to the domain event bus (Bucket G) for live updates - -### Bucket H: Blogmark And Transforms - -#### `bds-core` - -- `BlogmarkTransformService` -- transform script chain execution - -#### `bds-ui` - -- external content capture workflow - -### Bucket J: A2UI Surfaces - -#### `bds-ui` - -- A2UI component renderer -- A2UI surface manager -- AI assistant dynamic surface integration - -## Exit Rule - -Do not start extension buckets until Milestones M5 and M6 are complete and the Rust app is already a credible replacement for the baseline authoring and publishing workflow. diff --git a/RUST_PLAN.md b/RUST_PLAN.md deleted file mode 100644 index 365344f..0000000 --- a/RUST_PLAN.md +++ /dev/null @@ -1,120 +0,0 @@ -# bDS Rust Rewrite Plan - -bDS is a blogging desktop system. The behavioural baseline is **bDS2**, an Elixir (Phoenix LiveView) implementation living in: - -../bDS2/ - -This project (RuDS) rewrites it in Rust. For any implementation detail, look into ../bDS2/ — the Elixir code there is "the truth" about the current implementation. The old TypeScript app at ../bDS is historical and no longer the reference. - -## Spec Baseline - -The authoritative behaviour contract is the allium spec set in `specs/`, synced wholesale from `../bDS2/specs/` on 2026-07-18 (46 files, all pass `allium check`). Rules: - -- When spec and Rust code disagree, the spec wins; when the spec is ambiguous, the bDS2 Elixir code wins. -- Five specs are new with the bDS2 sync and have **no Rust implementation yet**: `rendering.allium` (core scope — render assigns/filters/macros shared by preview and generation), `cli.allium`, `events.allium`, `server.allium`, `tui.allium` (all extension scope). -- The concrete code-vs-spec gaps are tracked in RUST_EXECUTION_BACKLOG.md under "Milestone M6: bDS2 Spec Parity". -- When re-syncing specs from bDS2, re-run `allium check specs/*.allium` and refresh the M6 gap list. - -This plan is split into multiple documents: - -- [RUST_PLAN_CORE.md](RUST_PLAN_CORE.md) — the shipping core. This must already be a fully functioning blogging app: create, edit, preview, generate, and publish content using the exact same on-disk and generated formats as the current app. -- [RUST_PLAN_EXTENSION.md](RUST_PLAN_EXTENSION.md) — parity work and advanced tooling that can land after the core app is usable. -- [RUST_EXECUTION_BACKLOG.md](RUST_EXECUTION_BACKLOG.md) — implementation backlog grouped by milestone and crate. -- [RUST_COMPATIBILITY_MATRIX_TEMPLATE.md](RUST_COMPATIBILITY_MATRIX_TEMPLATE.md) — template for the required persistence and compatibility inventory. - -## Non-Negotiable Constraints - -1. **Content compatibility is exact.** Existing SQLite data, markdown/frontmatter, translation files, media sidecars, templates, menu documents, generated HTML structure, feeds, and sitemaps must remain readable and writable by the Rust app. -2. **No JavaScript application runtime.** No npm, Node.js, Electron, or remotely loaded application assets. The required in-editor and style preview panels use Wry to display the localhost-only Rust preview server in the operating system webview; the separate Open in Browser command opens the same preview in the system browser. Pagefind is integrated as a Rust library dependency (`pagefind` crate), not as an external binary. -3. **Script compatibility is intentionally broken.** Python/Pyodide is removed. Rust bDS is a green-field app and uses Lua for user-authored scripting. Existing Python scripts are not carried forward as compatible runtime artifacts. Built-in macros (gallery, youtube, vimeo, photo_archive, tag_cloud) are re-implemented as native Rust, not routed through Lua. -4. **Core release ships as a native desktop app.** Primary target is macOS, but the UI stack (Iced + muda + rfd) is cross-platform from day one. Native menus, key handling, open-file/deep-link handling, and platform command routing are part of core scope, not follow-up polish. -5. **Template editing is core scope.** Template rendering without template management UI is not sufficient. -6. **Script API docs remain mandatory.** The runtime changes to Lua, but the app still ships and generates proper scripting API documentation. -7. **Split localization is mandatory.** UI locale follows the OS. Rendered and previewed content language follows project settings. - -## UI Technology Stack - -The application uses the following UI and platform integration stack: - -| Layer | Technology | Purpose | -|---|---|---| -| UI framework | **Iced** (Elm architecture) | Application layout, views, widgets, event loop | -| Text editing | **ropey** + **syntect** + **cosmic-text** | Custom syntax-highlighting editor widget for markdown, Liquid templates, and Lua scripts | -| Native menus | **muda** | Cross-platform native menu bar (NSMenu on macOS, Win32 menus on Windows, GTK/dbus on Linux) | -| File dialogs | **rfd** | Cross-platform native file/folder dialogs (NSOpenPanel/NSSavePanel on macOS, equivalents elsewhere) | -| Internal preview | **wry** | Embedded post and style preview panels backed only by the localhost Rust preview server | -| Platform lifecycle | **objc2** (macOS only, cfg-gated) | Thin shim for `application:openFile:`, `application:openURLs:`, and other `NSApplicationDelegate` hooks | - -### Why this stack - -- **Iced** is a published crate with versioned releases, proper documentation, and a stable API. It uses wgpu for GPU-accelerated rendering and follows the Elm architecture (Message → update → view). -- **muda** and **rfd** render through real platform APIs (NSMenu, NSOpenPanel, etc.) with zero fidelity loss versus hand-rolled platform code, while providing cross-platform support from day one. -- **wry** preserves the baseline app's internal Markdown/Preview workflow using the operating system webview. It is a presentation surface for the loopback preview server, not the application runtime; external-browser preview remains available separately. -- **ropey + syntect + cosmic-text** gives full control over the editor experience: rope-based efficient text storage, Sublime Text syntax grammars (markdown, Liquid, Lua), and proper font shaping and layout via the same engine used by cosmic-DE. -- The only platform-specific code is a small (~50 line) lifecycle shim for macOS app delegate hooks, conditionally compiled via `cfg(target_os = "macos")`. Linux and Windows equivalents are bounded and isolated to the same module. - -## Workspace Shape - -```text -bds-rust/ -├── Cargo.toml -├── crates/ -│ ├── bds-core/ # engines, models, rendering, publishing -│ ├── bds-editor/ # custom Iced editor widget (ropey + syntect + cosmic-text) -│ ├── bds-ui/ # Iced application, views, platform integration (muda, rfd) -│ └── bds-cli/ # later, optional automation surface -├── migrations/ -├── locales/ -├── assets/ -└── docs/ - └── scripting/ # generated Lua API docs + guides -``` - -### Crate responsibilities - -- **bds-core**: all engines, models, persistence, rendering, publishing — zero UI dependencies. -- **bds-editor**: reusable Iced custom widget for syntax-highlighting text editing. Depends on ropey, syntect, cosmic-text, and iced. Does not depend on bds-core. Can be extracted as a standalone crate. -- **bds-ui**: application shell, Iced views and components, embedded localhost preview (wry), message routing, platform integration (muda for menus, rfd for dialogs, objc2 shim for macOS lifecycle). Depends on bds-core and bds-editor. -- **bds-cli**: headless automation surface. Depends on bds-core only. - -## Distribution Characteristics - -- **Single application binary.** No Electron or Node.js runtime. Lua and SQLite are compiled into the binary; internal preview uses the operating system webview supplied by the target platform. -- **Binary size:** ~15–25 MB. -- **Memory usage / startup:** no BEAM VM or bundled browser engine — one native application process plus the operating system webview used while preview is visible. -- **Platform prerequisites:** macOS uses the system WebKit runtime. Windows/Linux packaging must account for Wry's platform webview prerequisites without adding an application-managed JavaScript runtime. - -## Split Rationale - -The old single-file plan mixed three categories of work: - -- work required to ship a real replacement for the current app -- work required for eventual feature parity -- work that is valuable but not on the critical path to replacing the baseline app - -The new split keeps the shipping path narrow. Core is only the work needed to replace the current app for everyday authoring and publishing. Extensions carry the rest. - -## Release Gates - -The Rust rewrite is not considered successful until the core release can do all of the following with existing bDS project data: - -1. Open a real project created by the baseline app. -2. Create and edit posts, translations, media, tags, templates, and settings. -3. Preview drafts and published content both in the post editor's internal preview panel and in the external system browser. -4. Generate a complete site whose output matches the current app modulo approved normalization differences. -5. Publish the generated output to a remote target. -6. Rebuild the database from files and run metadata diff without losing information. -7. Operate as a native desktop application with native menus and command handling. - -## Verification Baseline - -Both plan documents assume the same verification baseline: - -1. Fixture projects exported from the current app are the compatibility corpus. -2. Golden-file tests compare Rust-written files against baseline-written files. -3. Golden-output tests compare generated sites between the current app and the Rust app. -4. No feature is complete until both UI behavior and underlying engine behavior are covered. - -## Repository Strategy - -This repository (RuDS) is the Rust rewrite, kept separate from the baseline. `../bDS2` remains the stable reference implementation and fixture source; compatibility fixtures are pulled from it on a controlled cadence and generated output is compared here. diff --git a/RUST_PLAN_CORE.md b/RUST_PLAN_CORE.md index 17cfbbc..5f4f9ed 100644 --- a/RUST_PLAN_CORE.md +++ b/RUST_PLAN_CORE.md @@ -1,927 +1,155 @@ -# bDS Rust Rewrite — Core Plan +# RuDS Core Plan ## Goal -Ship a native desktop Rust application (macOS primary, cross-platform ready) that fully replaces the baseline app (bDS2, Elixir, at `../bDS2/`) for the primary blogging workflow: - -1. open an existing project -2. create and edit content -3. preview content locally -4. generate the site -5. publish the site - -Core is not a toy MVP. Core must already be a production-capable replacement for the current app's content pipeline. +RuDS is the native Rust replacement for bDS2. Core covers the complete everyday workflow: open an existing project, author content, preview and generate the site, publish it, and maintain database/filesystem integrity. -## Core Planning Rules +The behavioural contract is `specs/*.allium`. When a spec is ambiguous, `../bDS2` is the reference implementation. [RUST_PLAN_EXTENSION.md](RUST_PLAN_EXTENSION.md) contains optional and advanced surfaces. -1. Critical path first. Do not start advanced parity work while the authoring and publishing path is incomplete. -2. Compatibility before optimization. Matching data and output behavior matters more than early performance work. -3. Engine and UI land together at milestone boundaries. No long-lived backend-only branches of product functionality. -4. No new persistence formats during core. -5. Every milestone ends with a runnable app, not just passing unit tests. +Status in this document describes the current source code as of 2026-07-18. It deliberately does not track build runs, test runs, release gates, or implementation history. -## Explicit Compatibility Contract +## Non-Negotiable Constraints -### Must stay identical +- Preserve existing SQLite, Markdown/frontmatter, translation, media sidecar, template, menu, and generated-site formats. +- Published post bodies live in files, not the database. +- Use a native Rust desktop stack: Iced, muda, rfd, and Wry. No JavaScript application runtime or remote assets. +- Preview and generated output share the same Rust rendering pipeline. +- Python scripting is intentionally replaced by Lua; built-in macros remain native Rust. +- UI locale follows the operating system; rendered content language follows project settings. +- Online AI is optional and blocked in airplane mode; local models may remain available. +- Metadata writes, publishing, metadata diff, and rebuild must share one field mapping. +- User-visible functionality requires a UI entry point, and UI controls require working functionality. +- macOS commands must be wired through native menus and application lifecycle handling. -- SQLite schema semantics and data readability -- post markdown files and frontmatter layout -- translation file naming and frontmatter layout -- media sidecars and thumbnail conventions -- media translation records -- post-link tracking records -- template file formats and lookup rules -- menu document format (OPML at `meta/menu.opml`) — both read and write -- generated routes, feeds, sitemaps, and local preview behavior -- metadata JSON file layout: `meta/project.json`, `meta/categories.json`, `meta/category-meta.json`, `meta/publishing.json` -- metadata diff and rebuild-from-filesystem behavior -- full-text search behavior (SQLite FTS5 virtual tables for posts and media, with Snowball stemmers for 24 languages via custom tokenizer) -- slug generation behavior (verify `deunicode` output matches current `transliteration` output for the project's content corpus) +## Workspace -### May change intentionally +| Crate | Responsibility | +|---|---| +| `bds-core` | Models, SQLite, filesystem formats, engines, rendering, generation, AI, publishing, and Lua | +| `bds-editor` | Reusable Ropey/Syntect/Cosmic Text editor widget | +| `bds-ui` | Iced application, native menus/dialogs, platform lifecycle, and embedded preview | +| `bds-cli` | Extension-only headless automation surface; currently a stub | -- app implementation language and architecture -- editor technology -- scripting runtime: Python out, Lua in -- internal process model: no Electron main/renderer split -- UI framework: Iced replaces Electron/web UI -- platform integration: muda/rfd replace Electron's built-in menu and dialog APIs - -### Compatibility truth - -User content and project data are compatible. -User-authored Python scripts are not compatible and are treated as legacy artifacts. - -## bDS2 Spec Sync — Contract Updates (2026-07-18) - -The specs in `specs/` were synced wholesale from `../bDS2/specs/`. The following contracts changed or were made explicit relative to what this plan and the existing Rust code assumed. Each item names the spec file that is authoritative; the executable gap list lives in RUST_EXECUTION_BACKLOG.md M6. - -### Data placement (project.allium) - -- `public_dir` = the project folder (the directory containing `meta/project.json`): all user-owned, portable, webserver-bound content — posts, media + thumbnails, templates/, scripts/, meta/, tags.json, menu.opml, and generated `html/` output. -- `private_dir` = the OS per-user app-data dir (`~/Library/Application Support/bds`, `$XDG_CONFIG_HOME/bds`, `%APPDATA%\bds`): SQLite database, per-project embeddings index, model cache, project registry, window/UI state. Never inside the repo or the project folder. -- `data_path` is **discovered** from the on-disk location of `meta/project.json` and never persisted inside project.json — projects must stay movable (invariant `DataPathNotPersistedInProjectJson`). -- Creating a project does not seed `public_dir/templates` with bundled defaults (`ProjectTemplatesDirectoryReservedForUserTemplates`). - -### Behaviour deltas vs. old plan assumptions - -- **Slug uniqueness** (post.allium): unbounded numeric suffix `{slug}-2, -3, …`. The old `-999` cap and timestamp fallback are gone. -- **Menu** (menu.allium): file-only model — `meta/menu.opml` is the sole store, no DB table. Normalization guarantees home is always the first item (`HomeAlwaysFirst`); `UpdateMenu` strips incoming home entries and prepends one. A `SyncMenuFromFilesystem` flow reloads it from disk. -- **Scripts** (script.allium, schema.allium): file path is `scripts/{slug}.{extension}` with configurable `script_extension` (default `"lua"`); entrypoint default is `"render"` for macros, `"main"` otherwise. Execution guarantees: sandboxed runtime (no filesystem mutation, process control, or package loading) with host capabilities exposed only through an explicit `bds.*` API. Config limits: macro timeout 10s, transform toast limits (5/script, 20 total, 300 chars), blogmark title ≤200 / URL ≤2048. -- **Schema** (schema.allium): `ai_usage` gains `cache_read_tokens` / `cache_write_tokens` (nullable); the legacy `npm` / `provider_npm` columns are the generic `package_ref` / provider package reference. Statuses are named enums (PostStatus, PostTranslationStatus, TemplateStatus, ScriptStatus). -- **Media** (media.allium): import accepts optional caller-supplied metadata (title, alt, caption, author, language, tags); a `ReplaceMediaFile` flow swaps the underlying file for an existing media record; batch import processes linked images. Media search takes `filters`, not just a query (search.allium). -- **Search** (search.allium): FTS5 tables use per-field stemmed columns (title/excerpt/content/tags/categories); translations are stemmed with their own language stemmer and appended per-field. Languages without a Snowball algorithm pass through unstemmed. (Already implemented in `bds-core/db/fts.rs`.) -- **Generation** (generation.allium): only published content generates (`GenerationPublishedOnly`); language variant selection is specified; generation/validation run as a task group — one task per section (Site Core, Single Posts, Category/Tag/Date Archives) plus a final Build Search Index task — streaming one progress message per written page. -- **Rendering** (rendering.allium — NEW, core scope): the render-assigns contract shared by preview and generation. Rendering language is the content language, never the UI locale. Custom Liquid filters are `i18n`, `markdown`, and `slugify` (slugify was not in the old plan's filter list). -- **Metadata diff** (metadata_diff.allium): translation files carry only translation-specific metadata — shared status/timestamps come from the canonical post and must not be reported missing; repair supports a direction per item; `embedding` is a diffable entity type. -- **Translations** (translation.allium): translation files carry full metadata (`TranslationFilesCarryFullMetadata`); editing a published translation reopens it to draft (`ReopenPublishedTranslation`); an auto-translation pipeline (`ScheduleAutoTranslation`, `AutoTranslatePost`, `AutoTranslateMediaCascade`, `FillMissingTranslations`) fills missing languages, skips `doNotTranslate`, and is gated by the configured AI endpoint and airplane mode. -- **AI** (ai.allium): endpoint add/remove rules, advisory model catalog with conditional refresh, provider detection, vision capability gating, chat context truncation, airplane-mode model swap. One-shot operations remain core; chat remains extension. -- **Settings UI** (editor_settings.allium): settings gains a technology section (semantic similarity toggle; scripting runtime is fixed, no switch exposed), a data section (rebuild buttons for posts/media/scripts/templates/links/thumbnails/embedding + Open Data Folder), agent integrations (Claude Code and GitHub Copilot supported; others render disabled with a "not supported yet" note), image import concurrency (1–8, default 4), and default editor mode (markdown | preview). Settings opens in a separate tab. -- **Tasks** (task.allium): `pending → cancelled` is a legal transition (cancel queued tasks, not only running ones). -- **Frontmatter** (frontmatter.allium): timestamps are Unix **milliseconds**; keys are camelCase; `doNotTranslate` only written when true, `templateSlug`/`publishedAt` only when present. (Already implemented in `bds-core/util/frontmatter.rs`.) - -### New subsystems specified by bDS2 (extension scope) - -- `cli.allium` — workspace CLI (`rebuild | repair | render | upload | push | pull | post | media | gallery | config | project | tui | lua`) sharing the same database and settings as the GUI, no listeners/window. -- `events.allium` — domain event bus: every entity mutation broadcasts `(entity, entity_id, action)` on the same topic/payload shape as the CLI sync watcher, so one subscription covers in-app and external changes. -- `server.allium` — headless server mode with SSH transport (boot modes desktop | server | tui via env var; SSH key material in the private app-data dir). -- `tui.allium` — terminal UI as a second renderer over the same shared UI core and post-editor workflow as the GUI. - -These map to extension Buckets G (CLI + events) and K/L (server, TUI) in RUST_PLAN_EXTENSION.md. - -## Architecture Decisions - -- **Single process Rust app**: no Electron, no IPC boundary. -- **Iced (Elm architecture) for UI**: Message-driven update cycle. All user interactions produce `Message` variants; state mutations happen in a centralized `update()` function; views are pure functions that return element trees. -- **muda for native menus**: cross-platform native menu bar from day one. NSMenu on macOS, Win32 menus on Windows, GTK/dbus on Linux. Key equivalents and menu validation handled through muda's API. -- **rfd for file dialogs**: cross-platform native open/save/folder dialogs. NSOpenPanel/NSSavePanel on macOS, equivalents elsewhere. -- **objc2 for macOS lifecycle shim** (cfg-gated): thin bridge for `application:openFile:`, `application:openURLs:`, and other `NSApplicationDelegate` hooks that muda/rfd do not cover. ~50 lines of platform code. -- **Custom editor widget (bds-editor crate)**: syntax-highlighting markdown/Liquid/Lua editor built on ropey (rope buffer), syntect (syntax highlighting), and cosmic-text (font shaping and text layout). This is the highest-risk custom component and gets a proof-of-concept in Wave 0. -- **Diesel + embedded migrations**: typed SQLite queries and generated schema, with migrations managed by `diesel_migrations`. Backend-only SQL is confined to connection setup and FTS5 operations. -- **tokio as the async runtime**: preview server (axum), SSH publishing, file watching, and rfd async dialogs all require an async executor. tokio is the standard choice and is used workspace-wide. Synchronous engine code in bds-core does not use tokio directly — it remains callable from both async (bds-ui) and sync (bds-cli) contexts. -- **Markdown editor with live preview**: bds-editor provides syntax-highlighted Markdown editing, paired with the rendered preview required by the baseline workflow. -- **Lua for user-authored scripting**: `mlua` with Lua 5.4. Only user-authored macros, transforms, and utility scripts run through Lua. Built-in macros are native Rust — see the macro architecture section below. -- **One-shot AI operations are core scope**: six operations use a simple OpenAI-compatible HTTP client (`reqwest` + `serde_json`): (1) translate post, (2) translate media metadata, (3) image description / alt text, (4) post analysis (title + excerpt + slug suggestion), (5) taxonomy analysis (tag + category suggestions), (6) language detection. Two configurable endpoints: one for online use, one for airplane/offline mode (local model). These are fire-and-forget request/response calls — no streaming, no tool use, no chat history. The AI chat UI, streaming responses, and tool execution remain in Extension Bucket C. - -## Iced Architecture Patterns - -The Iced Elm architecture imposes a specific application structure: - -### Message routing - -All user interactions, menu events, file dialog results, platform lifecycle events, and async task completions are expressed as variants of a root `Message` enum. The application's `update()` method matches on these variants and mutates state accordingly. This replaces the "command dispatcher" pattern seen in imperative UI frameworks. - -### View composition - -Views are pure functions: `fn view(&self) -> Element`. They read application state and return a widget tree. Views never mutate state directly. This makes UI code inherently testable — you can assert on the element tree without rendering. - -### Command and subscription model - -Side effects (file I/O, network, timers, platform events) are expressed as `Command` or `Task` values returned from `update()`. Menu events from muda arrive via a `Subscription` that polls `MenuEvent::receiver()`. Platform lifecycle events from the objc2 shim arrive via a similar channel-based subscription. - -### State model - -The UI state must track at minimum: - -- active project -- open tabs and selected tab -- selected entities -- dirty editors (with undo/redo state in bds-editor) -- task progress -- `ui_locale` -- project render settings including `content_language` -- menu state (enabled/disabled/checked items synced to app state) - -## Workspace Structure - -```text -bds-rust/ -├── Cargo.toml -├── crates/ -│ ├── bds-core/ -│ │ ├── src/ -│ │ │ ├── db/ -│ │ │ ├── engine/ -│ │ │ ├── model/ -│ │ │ ├── render/ -│ │ │ ├── scripting/ -│ │ │ ├── i18n/ -│ │ │ └── util/ -│ ├── bds-editor/ -│ │ ├── src/ -│ │ │ ├── buffer.rs # ropey rope wrapper, edit operations -│ │ │ ├── highlight.rs # syntect integration, incremental rehighlight -│ │ │ ├── layout.rs # cosmic-text buffer/shaping/layout -│ │ │ ├── cursor.rs # cursor model, selection, multi-cursor -│ │ │ ├── history.rs # undo/redo operation log -│ │ │ ├── input.rs # key event → buffer mutation mapping -│ │ │ ├── widget.rs # Iced custom widget implementation -│ │ │ └── lib.rs -│ ├── bds-ui/ -│ │ ├── src/ -│ │ │ ├── app.rs # Iced Application impl, root Message enum, update() -│ │ │ ├── platform/ -│ │ │ │ ├── mod.rs -│ │ │ │ ├── macos.rs # objc2 lifecycle shim (cfg-gated) -│ │ │ │ └── menu.rs # muda menu construction and event routing -│ │ │ ├── views/ -│ │ │ ├── components/ -│ │ │ └── i18n/ -│ └── bds-cli/ -├── migrations/ -├── fixtures/ -│ ├── compatibility-projects/ -│ └── golden-generated-sites/ -└── docs/ - └── scripting/ -``` - -## Core Feature Scope - -### Included in core - -- project open/create/select -- posts and translations -- media import and metadata editing -- tags and categories -- template editing and validation -- settings and publishing preferences -- live preview and full static generation -- publish via SSH and rsync -- metadata diff and rebuild-from-filesystem -- native menus and shortcuts via muda (cross-platform) -- native file dialogs via rfd (cross-platform) -- macOS lifecycle hooks via objc2 shim (open-file, URL-open) -- syntax-highlighting editor for markdown, Liquid templates, and Lua scripts via bds-editor -- built-in macros implemented natively in Rust (gallery, youtube, vimeo, photo_archive, tag_cloud) -- Lua runtime for user-authored macros, transforms, and utility scripts -- generated Lua scripting API docs -- FTS5 virtual tables for in-app post and media search -- one-shot AI operations: translate post, translate media, image alt text, post analysis, taxonomy analysis, language detection (two configurable OpenAI-compatible endpoints: online + airplane mode, via `reqwest`) - -### Deferred to extensions - -- AI chat UI, streaming responses, and tool execution (Bucket C) -- Git UI and history tools -- WordPress import wizard -- embeddings and duplicate detection -- translation validation reports -- menu editor UI -- documentation browser UI -- MCP and remote automation surfaces -- Blogmark transform service (external content capture pipeline) -- A2UI server-driven UI surfaces - -## Cross-Cutting Requirements - -### Native platform shell - -Core includes: - -- native menu bar via muda (App, File, Edit, View, Window, Help menus) -- standard key equivalents and accelerators via muda -- menu enable/disable validation synced to application state -- menu actions routed as `Message` variants into the Iced update cycle -- file-open and folder-open dialogs via rfd -- macOS: open-file from Finder and URL scheme handling via objc2 lifecycle shim -- proper window lifecycle and recent-project tracking - -### Split localization - -Two independent localization domains are required: +## Current Core Status -- `ui_locale`: detected from the OS and used for menus, dialogs, toasts, and workspace chrome -- `content_language`: read from project settings and used for rendering, preview, feeds, sitemaps, and generation - -No design in core may collapse those into a single "current language" field. - -### Editor widget requirements - -The bds-editor crate is a custom Iced widget that provides syntax-highlighting text editing. It must support: - -- **Buffer**: ropey `Rope` for O(log n) edits and line indexing -- **Syntax highlighting**: syntect `HighlightLines` with incremental rehighlighting on edits. Grammars needed: Markdown, Liquid (HTML), Lua, YAML (frontmatter), JSON -- **Text layout**: cosmic-text `Buffer` for font shaping, glyph positioning, and line layout -- **Cursor**: position tracking, selection (shift+movement), click-to-place, click-and-drag selection -- **Input handling**: standard key bindings (arrows, home/end, page up/down, word movement with option/ctrl), text insertion, delete/backspace, tab/indent -- **Undo/redo**: operation log tracking edit groups, triggered by standard shortcuts -- **Line numbers**: gutter with line numbers, synchronized scroll -- **Scroll**: vertical and horizontal scrolling, viewport-aware rendering (only lay out visible lines) -- **Soft wrap**: configurable per editor instance -- **IME support**: proper handling of composition events from winit for CJK and other input methods — test early +### Project, persistence, and integrity — Mostly done -The widget emits Iced `Message` variants for content changes, cursor movement, and save requests. The parent view owns the buffer state and passes it to the widget. +Available: -### Metadata parity matrix +- Existing project discovery and opening, with the database stored in private OS application data. +- Diesel-backed SQLite schema and migrations. +- Posts, translations, media, media translations, post-media links, tags, templates, scripts, settings, menus, and publishing settings. +- Draft/published lifecycle with canonical content placement. +- YAML frontmatter, translation files, media sidecars, template/script files, and project metadata. +- FTS5 post and media search with language-aware stemming. +- Database rebuild from filesystem and directional metadata diff/repair. +- Structured site, media, and translation validation. +- Unbounded unique slug allocation. -Before any engine work begins, create a compatibility inventory for every persisted field across: +Open: -- database columns -- markdown frontmatter -- translation frontmatter -- media sidecars -- project metadata files -- template files -- generated output metadata +- Watch project files for external changes and reconcile open editors, preview, and database state. +- Make the script file extension project-configurable instead of hard-coding `.lua`. -For each field record: +### Native desktop shell — Done -- source of truth -- persisted locations -- read path -- write path -- rebuild behavior -- metadata diff behavior -- publish behavior +Available: -This matrix is a release artifact, not a temporary note. +- Iced workspace with activity bar, sidebar, tabs, status bar, task/output panel, toasts, and modal flows. +- Native muda menus, localized labels, accelerators, and state-dependent command enablement. +- Native file/folder dialogs and recent-project handling. +- macOS open-file and URL lifecycle plumbing. +- Localized UI separate from project content language. -### Liquid template feature subset +Open: -The current default templates use a small subset of the Liquid specification. Before implementing the Liquid render pipeline, inventory the exact features used. The current templates (12 files: 3 main templates, 5 macro templates, 4 partials in `src/main/engine/templates/`) use only: +- No known large core block. New menu commands must continue to be wired through both localization layers and the native intercept. -- **Tags:** `if`/`elsif`/`else`, `for`, `assign`, `render` (partials), whitespace stripping (`{%- -%}`) -- **Filters (built-in):** `default`, `escape`, `url_encode`, `append` -- **Filters (custom, must be re-implemented in Rust):** `i18n` (translation lookup by key and language), `markdown` (markdown-to-HTML with macro expansion, URL rewriting, and media path canonicalization) -- **Operators and access:** `==`, `>`, `or`, `and`, bare truthiness, `blank` (nil/empty). Property access via dot notation (`object.property`), `.size` on arrays (this is property access, **not** a pipe filter), bracket notation for map lookups (`map[key]`). -- **Not used:** `unless`, `case/when`, `capture`, `layout`, `include`, `comment`, `raw`, `date`, `truncate`, `split`, `join`, `where`, `group_by`, `map`, `sort`, `reverse`, `cycle`, `tablerow`, `increment`/`decrement`, `size` (as a pipe filter), and most other standard filters +### Authoring and editor — Mostly done -This means the Rust Liquid implementation only needs roughly 35% of the full specification. Use `liquid-rust` or a minimal custom engine scoped to just the features above. User templates may use additional features, but parity with the current default templates is the release gate. +Available: -### Built-in macro architecture +- Dashboard and editors for posts, translations, media, tags, templates, scripts, and settings. +- Post create, edit, duplicate, publish, unpublish, discard, and delete flows. +- Media import, replacement, metadata editing, translations, thumbnails, filters, and post assignment. +- Template and Lua script creation, editing, validation, publication, and deletion. +- Rope-based editing with syntax highlighting, selection, clipboard, undo/redo, word/line/page movement, line numbers, soft wrapping, mouse selection, and committed IME input. +- Automatic translation flows with airplane-mode gating and media translation propagation. -The current app has two distinct macro systems that must not be conflated: +Open: -1. **Built-in macros** (`gallery`, `youtube`, `vimeo`, `photo_archive`, `tag_cloud`) — implemented in the host language of the baseline app, rendered server-side during generation, each with a corresponding `.liquid` macro template. These are **not** user scripts and never were. -2. **User-authored script macros** — Lua in bDS2 and in the Rust app (the old TypeScript app used Python/Pyodide; those scripts are legacy artifacts). See script.allium for the sandbox and `bds.*` host API contract. +- Replace the post-links placeholder with the specified functional linked-post view. +- Add the specified batch workflow for images linked from post content. -In the Rust app: +### Rendering, preview, and generation — Mostly done -- Built-in macros become **native Rust functions** in `bds-core/render`. They are part of Wave 4 (rendering parity), not Wave 6 (Lua scripting). Rendering cannot produce correct output without them. -- User-authored macros run through the **Lua runtime** (`mlua`). This is Wave 6 scope. During Wave 4, unknown macro placeholders should produce a visible "unsupported macro" marker rather than silently dropping content. +Available: -Macro invocation syntax in content: `[[macro_name param1="value1" param2="value2"]]` +- Shared Markdown and Liquid rendering for preview and generated output. +- Template lookup, routes, language variants, published snapshots, custom assigns, and URL rewriting. +- Native gallery, YouTube, Vimeo, photo archive, and tag cloud macros. +- User-authored Lua macro invocation during rendering. +- Localhost-only Axum preview server, draft routes, embedded Wry preview, and external-browser preview. +- Complete site generation with pages, archives, feeds, sitemap, static assets, changed-file tracking, parallel page rendering, and Pagefind output. +- OPML/menu document loading and normalized Home-first menu output. -### Slug generation compatibility +Open: -The Rust app uses `deunicode` for Unicode-to-ASCII conversion in slugs; it must match the established bDS transliteration behaviour (especially German umlauts). The slug compatibility test suite in M0 fixtures covers this. Uniqueness per post.allium: try the base slug, then `{slug}-2`, `{slug}-3`, … with an **unbounded** numeric suffix (no 999 cap, no timestamp fallback). +- Represent generation as the specified group of section tasks followed by a final search-index task, rather than one coarse application task. +- Keep closing concrete output differences found against bDS2; approved normalization differences belong in this document when discovered. -### Two search systems +### One-shot AI — Mostly done -The app has two independent search systems — do not conflate them: +Available: -1. **FTS5 (in-app search)**: SQLite FTS5 virtual tables (`posts_fts`, `media_fts`) power the desktop app's search UI. Text is Snowball-stemmed in application code before indexing and querying. FTS5 virtual-table and `MATCH` operations form the explicitly isolated raw-SQL backend boundary; ordinary filtering uses Diesel's typed query builder. This is Wave 1 scope. +- Configurable online and airplane-mode OpenAI-compatible endpoints. +- Secure API-key storage through the operating-system keychain. +- Model catalog discovery and model selection. +- Post translation, media translation, image alt text, post analysis, taxonomy analysis, and language detection. +- Explicit offline gating and user-visible errors. -2. **Pagefind (generated site search)**: a client-side search index bundled with the generated static site. Pagefind indexes the generated HTML files and produces JavaScript/WASM artifacts that power search on the published website. This is Wave 4 scope, added to the generation pipeline via the `pagefind` crate's Rust library API. +Open: -### Client-side search index (Pagefind) +- Persist actual input, output, cache-read, and cache-write token usage where the schema provides those fields. -The baseline app generates a Pagefind search index as part of site generation, and the Rust app must produce the same artifact. Pagefind publishes a Rust library (`pagefind` crate) with a programmatic API (`pagefind::api::PagefindIndex`). The generation pipeline feeds rendered HTML to `PagefindIndex::add_html_file()` and writes the resulting index files via `PagefindIndex::get_files()`. No external binary, no npm — this is a pure Rust library dependency, fully compatible with the no-JavaScript constraint. +Interactive chat, tools, agents, and MCP belong to the extension plan. -Determine during the compatibility inventory (M0) whether Pagefind or another client-side search solution is used. If Pagefind: add it to the generation pipeline in Wave 4 via the `pagefind` crate. If a different solution: document it and plan accordingly. +### Publishing — Mostly done -### Image processing +Available: -Media import requires thumbnail generation and potentially format conversion (e.g., to WEBP). The Rust choice is the `image` crate for decoding/encoding and basic transforms (resize, crop). If WEBP encoding performance or advanced processing (EXIF handling, ICC profiles) proves insufficient with `image` alone, `libvips` Rust bindings (`libvips-rs`) are the fallback. Start with `image` — it covers the common cases and has no system dependencies. +- SCP and rsync publishing through system commands. +- SSH-agent-only authentication with no password prompts. +- Separate HTML, thumbnail, and media targets. +- `.meta` exclusion, changed-file skipping for SCP, progress reporting, cancellation, and UI commands. -## Planned Crate Registry +Open: -All crate choices for core scope, organized by subsystem. This prevents ad-hoc technology decisions during implementation. +- Run the three upload targets as parallel tasks instead of processing them sequentially. +- Integrate external-file watching with publish/integrity workflows. -### Foundation (Wave 0 onward) +### Lua scripting — Partly done -| Crate | Purpose | Notes | -|---|---|---| -| `diesel` (sqlite) | Typed SQLite database access | Query builder and generated schema | -| `diesel_migrations` | Embedded migration management | Runs generated Diesel migrations at startup | -| `libsqlite3-sys` (bundled) | SQLite native library | Compiles SQLite from source with FTS5 | -| `uuid` (v4) | Entity identifiers | | -| `serde` + `serde_json` | Serialization/deserialization | Used everywhere | -| `serde_yaml` | YAML frontmatter parsing/writing | Posts, translations, media sidecars | -| `chrono` | Date/time handling | | -| `sha2` | Content hashing, checksums | | -| `deunicode` | Unicode-to-ASCII for slug generation | Verify against `transliteration` output | -| `thiserror` | Typed error definitions in library crates | bds-core, bds-editor | -| `anyhow` | Ergonomic error handling in application crates | bds-ui, bds-cli | -| `tokio` | Async runtime | Preview server, publish, file watching, async dialogs | -| `rust-stemmers` | Snowball stemming for FTS5 | Custom FTS5 tokenizer for 24-language stemmed search | +Available: -### UI Framework (Wave 0 onward) +- Sandboxed vendored Lua 5.4 runtime with cancellation and execution limits. +- Application log, progress, and toast functions. +- User macro and transform execution, including Blogmark transforms. +- Lua script persistence, rebuild, metadata diff, validation, and editor support. -| Crate | Purpose | Notes | -|---|---|---| -| `iced` (wgpu, advanced, image) | Application framework | Elm architecture, GPU-accelerated via wgpu/Metal | -| `muda` | Cross-platform native menu bar | NSMenu / Win32 / GTK | -| `rfd` | Cross-platform native file dialogs | NSOpenPanel / Win32 / GTK | +Open: -### Editor Widget (Wave 0 onward, bds-editor crate) +- Expose the specified `bds.*` API for post, media, tag, project, and other script-visible data. +- Generate and bundle the Lua API reference, canonical type reference, and macro/transform/utility examples under `docs/scripting/`. +- Add a documentation-sync check tied to the exposed API. -| Crate | Purpose | Notes | -|---|---|---| -| `ropey` | Rope buffer for text storage | O(log n) edits, line indexing | -| `syntect` | Syntax highlighting | Sublime Text grammars: Markdown, Liquid, Lua, YAML, JSON | -| `cosmic-text` | Font shaping, text layout | Same engine as cosmic-DE | +## Remaining Core Blocks -### Platform Lifecycle (Wave 0 onward, macOS cfg-gated) +1. Complete the Lua host API and scripting documentation. +2. Add filesystem change watching and reconciliation. +3. Parallelize publishing targets. +4. Finish the linked-post and linked-image authoring workflows. +5. Add generation section-task grouping and AI token accounting. +6. Remove the hard-coded Lua script extension. -| Crate | Purpose | Notes | -|---|---|---| -| `objc2` | Objective-C runtime bindings | | -| `objc2-foundation` | Foundation framework types | | -| `objc2-app-kit` | AppKit framework (NSApplicationDelegate hooks) | ~50 lines of shim code | - -### Data Layer (Wave 1) - -| Crate | Purpose | Notes | -|---|---|---| -| `serde_yaml` | Frontmatter read/write | Already listed in foundation | -| `notify` | Filesystem watching | Detect external file changes affecting open editors | -| `image` | Thumbnail generation, format conversion | Start here; libvips-rs as fallback if needed | -| `walkdir` | Recursive directory traversal | Rebuild-from-filesystem, media import | - -### Rendering and Generation (Wave 4) - -| Crate | Purpose | Notes | -|---|---|---| -| `pulldown-cmark` | Markdown → HTML | Fast, CommonMark-compliant | -| `liquid` | Liquid template rendering | Scoped to the ~35% feature subset actually used | -| `quick-xml` | RSS/Atom feeds, sitemaps | Also handles OPML menu documents | -| `rayon` | Parallel site generation | Parallelize page rendering across CPU cores | -| `axum` | Preview HTTP server | Lightweight, tokio-based | - -### Publishing (Wave 5) - -| Crate | Purpose | Notes | -|---|---|---| -| `ssh2` | SSH/SCP file upload | For publish-via-SSH targets. Authentication via SSH agent (`SSH_AUTH_SOCK`) only — no password auth, no interactive prompts. | -| (shell out) | rsync invocation | rsync ships with macOS/Linux; invoke as child process | - -### Client-Side Search (Wave 4) - -| Crate | Purpose | Notes | -|---|---|---| -| `pagefind` | Search index generation | Rust library API: `PagefindIndex::add_html_file()` + `get_files()`. No CLI binary needed. | - -### AI — One-Shot Operations (Wave 4–5) - -| Crate | Purpose | Notes | -|---|---|---| -| `reqwest` | HTTP client for AI endpoints | OpenAI-compatible Chat Completions API. Two endpoints: online + airplane mode (local). Used for translation, alt text, taxonomy, post analysis, language detection. | - -### Scripting (Wave 6) - -| Crate | Purpose | Notes | -|---|---|---| -| `mlua` (lua54) | Lua 5.4 embedding | User-authored macros, transforms, utility scripts. Use `vendored` feature to compile Lua from source. | - -### Extension-Only Crates (not used in core) - -| Crate | Purpose | Bucket | -|---|---|---| -| `git2` | Git operations | A (Git + Validation) | -| `ort` | ONNX inference for embeddings | D (Embeddings) | -| `usearch` | HNSW vector index | D (Embeddings) | - -## Critical Path - -The hard sequence is: - -1. compatibility inventory and fixtures -2. exact-read and exact-write data layer -3. native platform shell and command system (muda menus, rfd dialogs, Iced message routing) -4. editor widget MVP (ropey + syntect + cosmic-text proof of concept) -5. editors for posts, media, templates, scripts, and settings -6. preview and generation parity -7. one-shot AI operations (translate post, translate media, image alt text, post analysis, taxonomy analysis, language detection) -8. publish and integrity tooling -9. Lua built-ins plus generated script API docs - -Anything outside that path is noise until the previous step is stable. - -## Core Milestones - -### Milestone M0: Compatibility Baseline - -- fixture projects checked in -- golden harness working -- schema readable -- native empty app launches with Iced window and muda menu bar -- bds-editor proof-of-concept renders a markdown file with syntax highlighting - -### Milestone M1: Data Fidelity - -- posts, translations, media, tags, and settings round-trip -- rebuild works -- metadata diff works -- no format drift on fixture writes - -### Milestone M2: Native Workspace - -- projects open in a real app window -- native menus and shortcuts work via muda -- sidebar, tabs, and message routing work -- file dialogs work via rfd - -### Milestone M3: Authoring - -- post, translation, media, template, and script editing works using bds-editor -- errors surface in the UI -- all required authoring entities are reachable from the workspace - -### Milestone M4: Rendering Parity - -- preview is trustworthy -- generation matches golden output -- route, feed, and sitemap parity is acceptable -- one-shot AI operations work (all 6 operations) with two configurable OpenAI-compatible endpoints (online + airplane mode) - -### Milestone M5: Operate And Ship - -- publish works end to end -- integrity checks surface actionable issues -- Lua built-ins and scripting docs are complete enough for users - -Core release happens only after M5. - -## Wave 0: Foundation And Compatibility Inventory - -**Goal:** Bootstrapped workspace, exact compatibility target defined, golden fixtures checked in, and editor widget risk retired. - -### Deliverables - -- Cargo workspace for `bds-core`, `bds-editor`, `bds-ui`, `bds-cli` -- SQLite connection layer, migrations loader, and base models -- compatibility inventory document covering all persisted fields (including `mediaTranslations`, `postLinks`, and FTS5 virtual tables) -- Liquid feature inventory documenting exactly which tags, filters, and patterns the default templates use -- slug compatibility test suite comparing `deunicode` output against `transliteration` output for fixture content -- fixture projects copied from the current app -- golden-file harness for file writes and generation output comparisons -- empty native app window: Iced window with muda-driven menu bar (App/File/Edit/View/Window/Help) -- **bds-editor proof-of-concept**: custom Iced widget rendering a markdown file with syntax highlighting via ropey + syntect + cosmic-text. Must demonstrate: text display with highlighting, cursor placement, text insertion, basic scrolling. This retires the highest-risk component early. -- Iced architecture patterns document covering: message design conventions, subscription model for menu events and platform hooks, custom widget patterns for bds-editor - -### Dependencies - -```toml -# Core -diesel = { version = "2.3", features = ["sqlite", "returning_clauses_for_sqlite_3_35"] } -diesel_migrations = "2.3" -libsqlite3-sys = { version = "0.37", features = ["bundled"] } -uuid = { version = "1", features = ["v4"] } -serde = { version = "1", features = ["derive"] } -serde_json = "1" -serde_yaml = "0.9" -chrono = "0.4" -sha2 = "0.10" -deunicode = "1" -thiserror = "2" -anyhow = "1" -tokio = { version = "1", features = ["full"] } -walkdir = "2" - -# UI framework -iced = { version = "0.13", features = ["wgpu", "advanced", "image"] } - -# Editor widget -cosmic-text = "0.12" -ropey = "1" -syntect = "5" - -# Platform integration (cross-platform) -muda = "0.15" -rfd = "0.15" - -# Platform lifecycle (macOS only) -[target.'cfg(target_os = "macos")'.dependencies] -objc2 = "0.6" -objc2-foundation = "0.3" -objc2-app-kit = "0.3" -``` - -Not all of these are needed in Wave 0 — this is the full foundation set. Wave-specific additions (image, pulldown-cmark, liquid, axum, rayon, quick-xml, pagefind, reqwest, ssh2, mlua, notify) are added when their wave begins. See the Planned Crate Registry for the complete list. - -### Tests - -- open real bDS databases and verify read access across all tables (including `mediaTranslations`, `postLinks`, FTS5 tables, and AI/catalog tables that must not cause errors even though they are not used in core) -- verify empty Rust app opens as a native window with working menu bar -- verify bds-editor PoC renders highlighted text and accepts keyboard input -- verify fixture loader and golden harness run in CI -- slug compatibility tests pass for all fixture content - -### Done when - -- workspace builds on macOS (and ideally on Linux for CI verification) -- fixture projects load -- compatibility corpus exists -- Liquid feature inventory is complete -- bds-editor PoC works -- Iced patterns doc is available -- native app shell with menus exists -- milestone M0 acceptance review passes - -## Wave 1: Core Data Layer And Filesystem Contract - -**Goal:** Exact-read and exact-write support for the current project data model. - -### Key crates introduced - -- `serde_yaml` for frontmatter parsing/writing -- `image` for thumbnail generation and format conversion during media import -- `walkdir` for recursive directory traversal during rebuild-from-filesystem -- `notify` for filesystem change detection (may be deferred to Wave 5 if not needed until publish) - -### Engines - -- `ProjectEngine` -- `PostEngine` -- `MediaEngine` -- `PostMediaEngine` -- `TagEngine` -- `MetaEngine` -- `TaskManager` — max 3 concurrent tasks, FIFO queue, 250ms progress throttle, cancellation support -- `MetadataDiffEngine` -- `RebuildEngine` or equivalent rebuild services - -### File and DB responsibilities - -- draft vs published post lifecycle -- translation files with canonical linkage -- media sidecars and thumbnails (thumbnail generation via `image` crate) -- tag/category sync -- checksum and content-hash tracking -- rebuild database from filesystem -- metadata diff against filesystem - -### Required behavior - -- published posts continue storing body content in files, not in DB -- every field persisted by the current app remains persisted in the Rust app in the same places -- metadata writes, metadata diff, rebuild, and publish all use the same field mapping - -### Tests - -- round-trip tests for posts, translations, media, templates, and project metadata -- rebuild tests using fixture projects -- metadata diff tests proving no false negatives for known field changes -- file-by-file golden comparisons against baseline output - -### Done when - -- Rust can read and write existing project data without format drift -- rebuild restores the DB from files accurately -- metadata diff catches every field covered by the matrix -- milestone M1 acceptance review passes - -## Wave 2: Native Platform Workspace Shell - -**Goal:** Native-feeling desktop application shell with real menus and command plumbing. - -### UI scope - -- workspace window (Iced application) -- activity bar -- sidebar -- tab bar -- status bar -- task/output panel -- project selector -- message routing layer (root `Message` enum with command dispatch in `update()`) - -### Platform scope - -- App, File, Edit, View, Window, Help menus via muda -- accelerators and key equivalents via muda -- menu enable/disable synced to app state (e.g., Save disabled when no dirty editor) -- menu events received via `Subscription` polling `MenuEvent::receiver()` -- file/folder dialogs via rfd -- macOS: open-file from Finder and URL-open handled by objc2 lifecycle shim, forwarded as `Message` variants -- recent project tracking (app-managed list; no platform API dependency) - -### State model - -The Iced application state must track at minimum: - -- active project -- open tabs -- selected entities -- dirty editors -- task progress -- `ui_locale` -- project render settings including `content_language` - -### Tests - -- tab lifecycle -- message dispatch from both menu events and keyboard entry points -- app-shell integration tests for opening a fixture project -- rfd dialog invocation tests (mocked where needed) - -### Done when - -- the app can be navigated by menu or keyboard alone -- native menu labels are localized from the UI locale -- shell behavior feels like a native desktop app -- milestone M2 acceptance review passes - -## Wave 3: Content Editing UI - -**Goal:** Full editing UI for the content types required by the publishing workflow. - -### Views - -- dashboard -- post editor (using bds-editor with markdown + YAML frontmatter highlighting) -- translation editor (using bds-editor) -- media browser and media editor -- tags view -- settings view -- templates view and template editor (using bds-editor with Liquid/HTML highlighting) -- scripts view and script editor (using bds-editor with Lua highlighting) -- lightweight linked-media and post-links views where needed for parity - -### Required capabilities - -- create, edit, duplicate, publish, unpublish, discard, and delete posts -- edit title, slug, excerpt, tags, categories, language, author, template assignment -- import media (via rfd file dialog) and edit title, alt, caption, author, tags, language -- create and edit templates with syntax validation -- create and edit Lua scripts with syntax validation -- expose errors and conflicts through dialogs and task panel output -- undo/redo in all editor instances via bds-editor history - -### Editor widget maturation - -By Wave 3, bds-editor must support the full feature set documented in the editor widget requirements section: - -- all cursor movement patterns (arrows, word, line, page, home/end) -- selection (shift+movement, click-and-drag, double-click word select) -- copy/cut/paste via system clipboard -- undo/redo with edit grouping -- line numbers gutter -- incremental syntax rehighlighting on edits -- IME input for non-Latin scripts -- configurable soft wrap - -### Tests - -- integration tests from sidebar selection to persisted save -- template edit and validation tests -- script save and validation tests -- editor widget tests for cursor movement, selection, undo/redo, and IME composition - -### Done when - -- every content type needed by preview/generation/publish has an editor -- template management is reachable and usable from the UI -- scripts are manageable from the UI even if advanced tooling is deferred -- bds-editor handles real-world editing scenarios reliably -- milestone M3 acceptance review passes - -## Wave 4: Rendering, Preview, And Generation - -**Goal:** Reproduce the current site's rendering pipeline and local preview behavior. - -### Engines - -- `TemplateEngine` -- `PageRenderer` -- `BlogGenerationEngine` -- `PreviewServer` (axum, localhost-only HTTP server) -- `SearchIndexEngine` if required for parity of generated sites - -### Key crates introduced - -- `pulldown-cmark` for Markdown → HTML -- `liquid` for Liquid template rendering (scoped to the feature subset) -- `quick-xml` for RSS/Atom feeds, sitemaps, and OPML menu document reading -- `rayon` for parallel page rendering during generation -- `axum` for the preview HTTP server (runs on tokio) -- `pagefind` for client-side search index generation (Rust library API, not CLI binary) -- `reqwest` for one-shot AI operations (all 6 operations: translate post/media, image alt text, post analysis, taxonomy analysis, language detection) against two configurable OpenAI-compatible endpoints (online + airplane mode) - -### Rendering pipeline - -1. load published posts, translations, menus, templates, and project settings -2. resolve templates using the same lookup rules as the current app -3. render markdown to HTML via `pulldown-cmark` -4. expand built-in macros (gallery, youtube, vimeo, photo_archive, tag_cloud) natively in Rust — these do not go through the Lua runtime -5. for user-authored Lua macros: delegate to the Lua runtime if available, otherwise emit an "unsupported macro" placeholder -6. apply Liquid templates with custom `i18n` and `markdown` filters -7. rewrite URLs and media references -8. generate archives, feeds (via `quick-xml`), sitemaps, and search artifacts required for parity -9. feed rendered HTML pages to Pagefind via `pagefind::api::PagefindIndex::add_html_file()`, then write the search index files via `get_files()` — if client-side search index is required for parity -10. write only changed outputs when safe to do so (track output file content hashes in a `generatedFileHashes` table, matching the current app's skip-unchanged-writes behavior) -11. parallelize page rendering via `rayon` where safe (each page render is independent) - -### Preview rules - -- preview server binds to `127.0.0.1:4123` (localhost only, fixed port) -- preview and generated HTML use local assets only -- preview must support drafts and language-prefixed routes -- each post editor can switch between Markdown and an internal Wry preview panel backed by the draft preview route -- Open in Browser starts or reuses the same preview server and opens its URL in the external system browser; it does not replace the internal panel -- rendered language is controlled by project settings, not by UI locale - -### One-shot AI operations - -Wave 4 introduces the one-shot AI client in `bds-core`. This is a minimal `reqwest`-based HTTP client that sends single request/response calls to an OpenAI-compatible Chat Completions endpoint. No streaming, no tool use, no chat history. - -**Operations (all 6 are core scope):** - -- **Translate post**: translate title, excerpt, and content to a target language. Used from the translation editor. -- **Translate media metadata**: translate title, alt, and caption to a target language. Used from the media editor. -- **Image description / alt text**: generate alt text and caption for a media item. Used from the media editor. -- **Post analysis**: suggest title, excerpt, and slug from post content. Used from the post editor. -- **Taxonomy analysis**: suggest tags and categories for a post. Used from the post editor. -- **Language detection**: detect the language of a text fragment. Used internally by other operations. - -**Configuration:** - -- **Online endpoint**: URL, API key, model name — for cloud AI providers (OpenAI, Anthropic-via-proxy, etc.) -- **Airplane mode endpoint**: URL, model name — for local models (Ollama, LM Studio, etc.) — no API key needed -- When airplane mode is active, only the airplane mode endpoint is used. Cloud endpoint calls are blocked. -- Default: no endpoints configured — AI features are opt-in. -- API keys stored securely via OS keychain (macOS Keychain, Windows DPAPI, Linux libsecret), never in project files. - -**Constraints:** - -- AI operations are entirely optional. The app is fully functional without any AI endpoint configured. -- When no endpoint is configured, AI-related UI elements are disabled or hidden. -- When airplane mode is active, the online endpoint is disabled. If no airplane mode endpoint is configured, AI is unavailable with a user-visible toast. -- Error responses produce user-visible feedback (toast or inline error), never silent failures. -- Request/response payloads use the OpenAI Chat Completions wire format. - -### Tests - -- fixture-based generation comparisons against current app output -- preview route tests for posts, drafts, assets, media, and language routes -- UI tests for switching a post between Markdown and internal preview, plus the external Open in Browser command -- template compatibility tests using current `.liquid` templates - -### Done when - -- the same project can be generated by both apps with matching output -- internal and external preview are accurate enough to be trusted as authoring tools -- milestone M4 acceptance review passes - -## Wave 5: Publishing And Operational Integrity - -**Goal:** Publishing and integrity tooling needed to operate the app in production. - -### Key crates introduced - -- `ssh2` for SSH/SCP file upload -- rsync invoked as child process (ships with macOS/Linux) - -### Engines and services - -- `PublishEngine` -- `SiteValidationDiffService` or equivalent core validation service -- `NotificationWatcher` if needed for filesystem coherence in core workflows - -### Required capabilities - -- upload generated site via SCP or rsync -- upload media and thumbnails correctly -- SSH authentication via SSH agent (`SSH_AUTH_SOCK`) only — no password auth, no interactive prompts -- three upload targets (html, thumbnails, media) run as parallel tasks -- exclude `.meta` sidecar files from media uploads -- surface publish progress and failures in the UI -- detect and surface external file changes that affect open editors or preview accuracy - -### Tests - -- publish tests against mocked remote targets -- validation tests on generated sites -- watcher tests for externally modified content files - -### Done when - -- publishing works end to end from the Rust app -- operator-visible integrity issues are surfaced before or during publish -- milestone M5 operational review is unblocked pending Wave 6 completion - -## Wave 6: Lua Runtime And Scripting Docs - -**Goal:** Deliver user-authored scripting via Lua and complete scripting documentation. - -Built-in macros (gallery, youtube, vimeo, photo_archive, tag_cloud) are already implemented as native Rust in Wave 4. Wave 6 covers only the user-extensible scripting layer. - -### Engines - -- `ScriptEngine` -- `LuaRuntime` -- `LuaApi` - -### Required scope - -- user-authored Lua macros invoked at render time via `[[macro_name]]` syntax -- user-authored transforms (post processing pipelines) -- user-authored utility scripts -- Lua API bridge exposing post, media, tag, and project data to scripts - -### Documentation requirements - -- generated Lua API documentation from source annotations or schema definitions -- canonical data structure reference for script-visible types -- examples for macro, transform, and utility scripts -- docs bundled with the app and exported as markdown files in `docs/scripting/` - -### Tests - -- Lua execution tests -- API bridge tests -- macro compatibility tests for built-in macros -- docs sync tests proving generated docs match exposed API - -### Done when - -- Lua scripting covers the built-ins needed by current templates and render flows -- scripting docs are complete enough for third-party script authors -- milestone M5 acceptance review passes - -## Core Dependency Graph - -```text -Wave 0 Foundation + Editor PoC - ↓ -Wave 1 Data + Filesystem Contract - ↓ -Wave 2 Native Platform Shell (Iced + muda + rfd) - ↓ -Wave 3 Content Editing UI (bds-editor maturation) - ↓ -Wave 4 Rendering + Preview + Generation - ↓ -Wave 5 Publishing + Operational Integrity - ↓ -Wave 6 Lua Runtime + Scripting Docs -``` - -Wave 4 depends on built-in macros (native Rust, not Lua). Wave 6 can start earlier for Lua runtime bootstrapping, but core release is not complete until user-authored scripting and docs are done. - -## Core Release Checklist - -Core ships only when all of the following are true: - -1. An existing bDS project opens without manual migration. -2. Posts, translations, media, templates, and settings can be edited in-app. -3. Preview and generation match the current app closely enough to pass golden tests. -4. Publishing works against supported remote targets. -5. Metadata diff and rebuild are accurate. -6. The app is a native desktop app with native menus and shortcuts. -7. Lua replaces Python and has proper generated API documentation. - -## Cross-Platform Notes - -The core stack (Iced + muda + rfd) is cross-platform. The only platform-specific code is the macOS lifecycle shim in `bds-ui/src/platform/macos.rs`. To ship on Linux or Windows: - -- Linux: no lifecycle shim needed (file open arrives as CLI args; URL handling via XDG). Iced, muda, and rfd work natively. -- Windows: similar lifecycle shim for file associations and URL protocol handling via Windows APIs. Iced, muda, and rfd work natively. - -Cross-platform packaging is not core scope, but the architecture does not accumulate macOS-only technical debt that would block it. - -## Supporting Docs - -- [RUST_EXECUTION_BACKLOG.md](RUST_EXECUTION_BACKLOG.md) -- [RUST_COMPATIBILITY_MATRIX_TEMPLATE.md](RUST_COMPATIBILITY_MATRIX_TEMPLATE.md) +Core is feature-complete when these blocks are closed and the implementation continues to satisfy the Allium and bDS2 compatibility contracts. diff --git a/RUST_PLAN_EXTENSION.md b/RUST_PLAN_EXTENSION.md index 60d6ac0..2a3acee 100644 --- a/RUST_PLAN_EXTENSION.md +++ b/RUST_PLAN_EXTENSION.md @@ -1,263 +1,132 @@ -# bDS Rust Rewrite — Extension Plan +# RuDS Extension Plan ## Goal -Deliver the rest of current-app parity and the advanced tooling that is valuable, but not required to ship a production-capable Rust replacement for everyday authoring and publishing. +Extensions add migration, collaboration, automation, alternate clients, and advanced discovery on top of the core blogging workflow. They must reuse core engines and formats, remain reachable through real UI or automation surfaces, and preserve airplane-mode and localization rules. -Extensions begin only after the core plan is already usable end to end. +Status describes the current source code as of 2026-07-18. Core one-shot AI, publishing, rendering, and integrity work is tracked in [RUST_PLAN_CORE.md](RUST_PLAN_CORE.md). -## Extension Principles +## Current Extension Status -1. No extension may break the core compatibility contract. -2. Extensions must reuse core models, engines, and persistence rules rather than invent parallel formats. -3. UI features must still be tied to underlying functionality; no placeholder shells. -4. AI features remain gated by offline mode and must prefer local models or provide explicit user feedback when unavailable. One-shot AI operations (6 operations: translate post/media, image alt text, post analysis, taxonomy analysis, language detection) are part of core with two configurable OpenAI-compatible endpoints (online + airplane mode), and respect the same offline gating. -5. Extensions use the same Iced + muda + rfd platform stack as core. No additional UI frameworks. +### Git and richer validation — Partly done -## Extension Buckets +Done: -### Bucket A: Git And Validation Tooling +- Site, media, translation, and metadata-diff engines and UI views. -#### Scope +Open: -- `GitEngine` via `git2` crate (shell out for LFS operations — no LFS library binding) -- Git sidebar -- diff view -- commit, fetch, pull, push -- richer site validation views -- richer metadata diff UI +- `GitEngine`, repository status, history, diff view, commit, fetch, pull, push, and LFS command integration. +- Replace the Git sidebar and Git log placeholders with working flows. -#### Why extension +### WordPress import — Open -Helpful for operators, but not required to create, preview, generate, and publish content. +Open: -#### Done when +- WXR parsing, import analysis, recoverable execution, saved import definitions, and import UI. +- The existing media importer is core functionality and does not implement this workflow. -- users can inspect repo state and diffs from within the app -- Git actions work reliably enough to replace the current Git tooling +### Conversational AI and agent tools — Open -### Bucket B: Import And Migration Tooling +Open: -#### Scope +- Conversation persistence and chat UI. +- Streaming OpenAI-compatible responses and tool-call parsing. +- Tools over posts, media, templates, search, and other core engines. +- Agent integrations such as Claude Code and Copilot where required by the specs. +- Replace the current Chat placeholders with the working feature. -- WXR parser -- import analysis -- import execution -- saved import definitions +Core endpoint settings, offline gating, key storage, model discovery, and six one-shot operations are already implemented. -#### Why extension +### Embeddings, semantic search, and duplicates — Open -Important onboarding feature, but not required to operate existing bDS projects. +Existing source contains schema placeholders only. -#### Done when +Open: -- WordPress import flows are usable and recoverable -- import results match the current app's expectations closely enough for fixture-based tests +- Embedding generation and persistence. +- Vector index and semantic search. +- Duplicate detection, dismissed-pair handling, metadata integrity, and UI. +- Replace the Find Duplicates placeholder. -### Bucket C: AI Chat And Tool Use +### Translation QA and documentation UX — Partly done -#### Scope +Done: -- chat UI (sidebar panel with conversation history) -- streaming responses via `reqwest` (SSE / chunked transfer) -- tool use against local engines (post lookup, media search, template info, etc.) -- multi-turn conversation management -- model and credential settings UI (extends the core AI endpoint configuration) +- Translation validation engine and report view. -One-shot AI operations (translation, image alt text, title suggestion) are already in core scope. This bucket adds the interactive, conversational AI layer on top. +Open: -The AI client extends the core `reqwest` + `serde_json` client with streaming support and tool-call parsing. Works with any OpenAI-compatible endpoint: OpenAI, Anthropic-via-proxy, local Ollama, etc. +- In-app project documentation browser. +- Browsable Lua API documentation and examples. +- Replace Documentation and API Documentation placeholders. -#### Hard constraints +The generated Lua documentation itself is a core requirement. -- offline mode gates all automatic AI work -- cloud providers are disabled when offline mode is enabled -- local providers remain usable when allowed -- unavailable operations produce explicit user-visible feedback +### Menu editor and deep links — Partly done -#### Done when +Done: -- AI chat is useful for content-related queries and actions without weakening the app's offline guarantees -- Streaming and tool use work reliably with at least one OpenAI-compatible provider +- Menu file parsing/rendering, Home-first normalization, macOS URL plumbing, and Blogmark deep-link parsing. -### Bucket D: Search, Embeddings, And Duplicate Detection +Open: -#### Scope +- OPML/menu editor UI. +- Remaining application deep-link parity flows. +- Replace the Menu Editor placeholder. -- ONNX embeddings via `ort` (ONNX Runtime Rust bindings) -- HNSW vector index via `usearch` -- semantic search UI -- duplicate detection UI +### CLI, MCP, and domain events — Open -#### Why extension +Open: -Improves discovery and cleanup, but not required for core publishing flows. +- Implement `bds-cli`; its current binary is only a stub. +- Commands from `cli.allium` and `cli_sync.allium` using the same project, database, engines, and settings as the desktop app. +- MCP tools/resources and proposal-based writes from `mcp.allium`. +- Domain event bus from `events.allium` for desktop, CLI, TUI, and future remote clients. +- Replace the MCP settings placeholder. -#### Done when +### Blogmark and transform pipeline — Mostly done -- near-duplicate detection and semantic search are trustworthy on real projects +Done: -### Bucket E: Translation QA And Documentation UX +- Blogmark deep-link parsing, content capture, post import, transform selection, and Lua transform execution. -#### Scope +Open: -- translation validation engine and report view -- in-app documentation browser -- richer scripting docs browser and examples +- Complete the dedicated user-facing capture/review workflow where the current implicit flow does not cover the bDS2 behavior. +- Close remaining transform-chain differences found against the specs and bDS2. -#### Why extension +### Headless server — Open -Operationally valuable, but the core release can ship with generated markdown docs and without dedicated browsing surfaces. +Open: -#### Done when +- Desktop/server/TUI boot-mode selection. +- Headless engine host and SSH transport. +- Private host-key and authorized-key management. +- Desktop connection flow for remote projects. -- translation integrity issues are discoverable before publish -- docs are comfortably browsable in-app +### Terminal UI — Open -### Bucket F: Menu Editing And Deep Links +Open: -#### Scope +- Terminal renderer over shared application workflows. +- Sidebar/editor navigation, editing, publishing, and live domain-event updates. +- Remote operation through the headless server. -- menu editor UI for OPML/menu documents -- deep-link protocol handling beyond core app-open behaviors +### A2UI surfaces — Open -#### Why extension +Open: -Core must read menu documents for rendering compatibility, but editing them can follow once the main authoring path is stable. +- A2UI component renderer, surface state/data flow, and AI-assistant integration. -#### Done when +## Suggested Order -- users can inspect and edit menus from the Rust app -- deep links cover parity flows from the current app +1. Git workflow and WordPress import. +2. CLI, MCP, and domain events. +3. Conversational AI and agent tools. +4. Embeddings and duplicate detection. +5. Documentation and menu UX. +6. Headless server and TUI. +7. A2UI after conversational AI exists. -### Bucket G: MCP And Automation Surfaces - -#### Scope - -- workspace CLI per `cli.allium`: `rebuild | repair | render | upload | push | pull | post | media | gallery | config | project | lua` — boots the same repo, settings, and database as the desktop app, with no listeners and no window; logging to the rotating log file -- MCP server per `mcp.allium` (translation tools, stats and media resources, proposal workflow) -- domain event bus per `events.allium`: every successful entity mutation (posts, media, tags, templates, scripts) broadcasts `(entity, entity_id, action)`; global settings changes broadcast a settings event. One topic/payload shape covers in-app mutations and external CLI writes, replacing the old `db_notifications` polling design. The bus also feeds the TUI (Bucket L) and any future multi-client sync. - -#### Why extension - -Useful ecosystem surface, not required to replace the desktop app itself. - -#### Done when - -- automation consumers can drive the Rust app safely and consistently -- CLI changes are detected and surfaced to the running desktop app via the event bus - -### Bucket H: Blogmark And Transform Pipeline - -#### Scope - -- `BlogmarkTransformService` -- external content capture (bookmarklet) workflow -- transform script execution chain -- integration with Lua transform scripts - -#### Why extension - -Secondary content-capture workflow. Not required for core authoring with existing projects. - -#### Done when - -- external content can be captured and transformed into posts via the Blogmark pipeline -- transform scripts execute reliably - -### Bucket K: Headless Server Mode - -#### Scope - -Per `server.allium`: - -- boot modes `desktop | server | tui` resolved from an environment variable at application start -- headless server: full engine, no window, SSH transport for remote TUI/GUI clients -- SSH key material (host key + authorized_keys, mode 600) generated on first boot into the private OS app-data dir — never the repo or the project folder -- GUI "connect to server" flow - -#### Why extension - -Client/server split is new bDS2 capability, not needed to replace the desktop app for local use. - -#### Done when - -- the app can run headless and accept an SSH-transported client session against real project data - -### Bucket L: Terminal UI - -#### Scope - -Per `tui.allium` — a second renderer over the same shared UI core: - -- sidebar views (posts, media, templates, scripts, tags, settings, git) with sidebar/editor focus model and vim-style + arrow navigation that skips section headers -- post editor driving the same workflow as the GUI: canonical-language edits update the post, other languages update translations, publish routes through the same pipeline -- live updates via the domain event bus (Bucket G) - -#### Why extension - -Second renderer; requires the shared UI core and (for remote use) Bucket K. - -#### Done when - -- everyday authoring (browse, edit, publish) works in a terminal against the same project data as the GUI - -### Bucket J: A2UI Server-Driven Surfaces - -#### Scope - -- A2UI component renderer (layout, input, display, chart, etc.) -- A2UI surface manager for bidirectional data flow -- integration with AI assistant outputs - -#### Why extension - -Tightly coupled to the AI feature set (Bucket C). Not required until AI features are active. - -#### Done when - -- AI-generated dynamic UI surfaces render correctly in the app - -## Suggested Extension Ordering - -```text -Bucket A Git + Validation - ↓ -Bucket B Import - ↓ -Bucket C AI - ↓ -Bucket D Embeddings + Duplicates - ↓ -Bucket E Translation QA + Docs UX - ↓ -Bucket F Menu Editing + Deep Links - ↓ -Bucket G MCP + Automation - ↓ -Bucket H Blogmark + Transforms - ↓ -Bucket K Headless Server Mode - ↓ -Bucket L Terminal UI (after Buckets G + K) - ↓ -Bucket J A2UI Surfaces (after Bucket C) -``` - -The ordering is pragmatic, not mandatory. Git and validation are the closest to operational parity, so they should land first after core. - -## Extension Verification - -Every extension still inherits the core verification baseline plus extension-specific tests: - -- Git fixtures for repo state and diff rendering -- WXR fixtures for import -- mocked SSE and provider fixtures for AI -- embedding fixtures for semantic search and duplicate detection -- translation fixture projects with intentional integrity failures -- OPML fixtures for menu editing - -## Out Of Scope For Now - -- cross-platform packaging polish beyond what the core and extension work naturally require -- feature work that introduces new persistence formats before full parity is reached +The order may change when an extension directly unlocks a concrete user workflow; it must not create a parallel data model or bypass core engines. diff --git a/SPECIFICATION_INDEX.md b/SPECIFICATION_INDEX.md index fc84e7a..cd2cb34 100644 --- a/SPECIFICATION_INDEX.md +++ b/SPECIFICATION_INDEX.md @@ -207,11 +207,8 @@ Diese Datei dient als Index zu allen Allium-Spezifikationen und Schema-Inventari | Datei | Beschreibung | |-------|--------------| -| `RUST_PLAN.md` | Haupt-Plan mit Non-Negotiable Constraints | -| `RUST_PLAN_CORE.md` | Detaillierter Core-Plan (886 Zeilen) | -| `RUST_PLAN_EXTENSION.md` | Extension Buckets A-J | -| `RUST_EXECUTION_BACKLOG.md` | Milestone-basierte Aufgabenliste | -| `RUST_COMPATIBILITY_MATRIX_TEMPLATE.md` | Vorlage für Metadata-Inventory | +| `RUST_PLAN_CORE.md` | Core-Umfang mit aktuellem Implementierungsstand und offenen Punkten | +| `RUST_PLAN_EXTENSION.md` | Extensions mit aktuellem Implementierungsstand und offenen Punkten | ## Spezifikations-Abdeckungs-Analyse diff --git a/docs/COMPATIBILITY_INVENTORY.md b/docs/COMPATIBILITY_INVENTORY.md deleted file mode 100644 index 8a903fc..0000000 --- a/docs/COMPATIBILITY_INVENTORY.md +++ /dev/null @@ -1,106 +0,0 @@ -# bDS Compatibility Inventory: TypeScript vs Rust - -Tracks feature parity between the TypeScript bDS app and the Rust rewrite (RuDS). - -Legend: `[x]` implemented/verified, `[ ]` not yet implemented - ---- - -## Database Schema - -- [x] projects table -- [x] posts table (with published_* legacy columns) -- [x] post_translations table -- [x] media table -- [x] media_translations table (mediaTranslations) -- [x] tags table -- [x] templates table (with kind/status defaults matching TypeScript) -- [x] scripts table (with kind/status defaults matching TypeScript) -- [x] post_links table (postLinks) -- [x] post_media table -- [x] settings table -- [x] generated_file_hashes table -- [x] db_notifications table -- [x] chat_conversations table -- [x] chat_messages table -- [x] ai_providers table -- [x] ai_models table -- [x] ai_model_modalities table -- [x] ai_catalog_meta table -- [x] embedding_keys table -- [x] dismissed_duplicate_pairs table -- [x] import_definitions table -- [x] All 10 unique indexes -- [x] FTS5 virtual tables (posts_fts, media_fts) -- present in fixture DB, runtime creation deferred to M1 -- [x] Default values match TypeScript (status, kind, enabled, version, entrypoint) - -## File Formats - -- [ ] Post frontmatter (YAML) read/write -- [ ] Translation file format (posts/YYYY/MM/slug.lang.md) -- [ ] Media sidecar (.meta) read/write -- [ ] Template frontmatter read/write -- [ ] Script frontmatter read/write -- [ ] meta/tags.json -- [ ] meta/project.json -- [ ] meta/categories.json -- [ ] meta/category-meta.json -- [ ] meta/publishing.json -- [ ] meta/menu.opml - -## Slug Generation - -- [x] Basic transliteration (Unicode to ASCII, lowercase, hyphens, trim) -- [x] German umlauts: ae/oe/ue/ss/Ae/Oe/Ue (matches TypeScript transliteration npm) -- [x] Uniqueness: base, then {slug}-2..999, then {slug}-{timestamp} - -## Content Location - -- [x] Published posts have NULL content in DB, body in filesystem .md -- [x] Draft posts have content in DB -- [x] Published translations have NULL content in DB -- [x] Published templates have NULL content in DB -- [x] Published scripts have NULL content in DB - -## Rendering - -- [ ] Liquid template rendering (subset: if/elsif/else, for, assign, render, whitespace stripping) -- [ ] Liquid filters: escape, url_encode, default, append, i18n, markdown -- [ ] Markdown rendering via pulldown-cmark -- [ ] Built-in macros: gallery, youtube, vimeo, photo_archive, tag_cloud -- [ ] RSS/Atom feed generation -- [ ] Sitemap generation -- [ ] Generated file hash tracking (incremental) -- [ ] Pagefind search index generation (via pagefind crate library API) - -## Search - -- [ ] FTS5 post indexing with Snowball stemmers (24 languages) -- [ ] FTS5 media indexing -- [ ] Cross-language stemming - -## Publishing - -- [ ] SSH/SCP upload -- [ ] Rsync upload -- [ ] Three parallel upload targets (html, thumbnails, media) -- [ ] .meta files excluded from upload - -## AI Integration - -- [ ] Two-endpoint model (online + airplane) -- [ ] One-shot operations (translate, analyze, etc.) -- [ ] Chat with tool use -- [ ] Model catalog refresh -- [ ] Secure key storage (OS keychain) - -## Thumbnails - -- [ ] Small (150px), Medium (400px), Large (800px), AI (448x448 JPEG) -- [ ] WEBP output - -## MCP Server - -- [ ] HTTP + stdio transports -- [ ] Read-only tools -- [ ] Proposal-based write tools