diff --git a/BDS_SEMANTIC_SIMILARITY.md b/BDS_SEMANTIC_SIMILARITY.md index 82508c5..ab319d2 100644 --- a/BDS_SEMANTIC_SIMILARITY.md +++ b/BDS_SEMANTIC_SIMILARITY.md @@ -2,6 +2,8 @@ Surface thematically related posts as an impulse — "Have I written something similar?" — inspired by Luhmann's Zettelkasten. Cross-domain connections across 10k+ posts over 20 years are the point, not a flaw. The algorithm finds the surface. The human finds the depth. +**Status: Not yet implemented.** No packages installed, no engine, no IPC, no UI integration. + --- ## Integration Point @@ -10,7 +12,23 @@ Surface thematically related posts as an impulse — "Have I written something s When the search field is empty (`query.length < 2`), instead of showing "type at least 2 characters", show 3–5 semantically similar posts to the currently edited post. These are default suggestions — "posts you might want to link to." -Requires threading `currentPostId` from `Editor.tsx` → `InsertModal` (currently only passes `currentPostTags` / `currentPostCategories`). +The InsertModal currently accepts these props: +```ts +interface InsertModalProps { + mode: InsertMode; + onInsertLink: (url: string, text?: string) => void; + onInsertImage: (url: string, alt: string, mediaId?: string) => void; + onClose: () => void; + initialText?: string; + currentPostTags?: string[]; + currentPostCategories?: string[]; + // currentPostId is NOT yet threaded through — needs to be added +} +``` + +`currentPostId` must be added to this interface and threaded from `Editor.tsx`. + +Note: The InsertModal now also has a **"Create post" option** — when `query.length >= 2` and no exact title match exists, it shows a `+` button to create a new post with that title and inherit the current post's tags/categories. This is the only UI addition since the original spec; it doesn't conflict with the semantic similarity integration. --- @@ -21,6 +39,8 @@ Requires threading `currentPostId` from `Editor.tsx` → `InsertModal` (currentl | Embeddings | Hugging Face Transformers.js | `@huggingface/transformers` | ONNX, local, no API key | | Vector index | USearch | `usearch` | HNSW, native C++ via N-API, prebuilt binaries | +Neither package is installed yet. + **Embedding model:** `multilingual-e5-small` — 384 dimensions, 512-token context, ~470 MB on disk, ~200–300 MB RAM, ~100ms/post inference. Natively multilingual (100+ languages incl. DE/EN) — critical for a mixed-language blog. `all-MiniLM-L6-v2` (~90 MB) was considered but is EN-trained with weak DE transfer; not suitable for nuanced cross-language similarity. **Why USearch over alternatives:** @@ -52,6 +72,8 @@ The `bigint → postId` key mapping lives in a Drizzle table (`embedding_keys`), ### Engine: `EmbeddingEngine` (`src/main/engine/EmbeddingEngine.ts`) +File does not exist yet. Create it. + Responsibilities: - Load/save USearch index + key map on startup/shutdown - Embed post content via `@huggingface/transformers` @@ -72,17 +94,60 @@ class EmbeddingEngine { } ``` -### Project switching +### EngineBundle (`src/main/engine/EngineBundle.ts`) -The app supports multiple projects. On project switch (`setProjectContext`), the engine must save and unload the current index, then load (or create) the index for the new project. Each project has its own `embeddings.usearch` file and `embedding_keys` table rows. +Add `embeddingEngine: EmbeddingEngine` to the `EngineBundle` interface. The current bundle contains: +`postEngine`, `mediaEngine`, `scriptEngine`, `templateEngine`, `metaEngine`, `menuEngine`, `tagEngine`, `postMediaEngine`, `projectEngine`, `gitEngine`, `gitApiAdapter`, `blogGenerationEngine`, `publishEngine`, `metadataDiffEngine`, `taskManager`, `blogmarkTransformService`, `mcpServer`, `blogmarkPythonWorkerRuntime`, `pythonMacroWorkerRuntime`, `publishApiAdapter`, `appApiAdapter`. -### IPC +### IPC layer (`src/main/ipc/`) + +The IPC layer now has five files: +- `handlers.ts` — main handler file (posts, media, project, meta, tags, templates, scripts, blog generation, publishing, preview, site validation, import, settings, model catalog, tasks, notifications) +- `chatHandlers.ts` — AI chat streaming & tool use +- `blogHandlers.ts` — blog generation & publishing +- `publishHandlers.ts` — publishing +- `metadataDiffHandlers.ts` — metadata diff +- `index.ts` — module exports + +Add the embedding IPC handlers to `handlers.ts` (they're small; no need for a new file): ``` embeddings:findSimilar(postId: string, k?: number) → SimilarPost[] embeddings:getProgress() → { indexed: number; total: number } +embeddings:suggestTags(postId: string, excludeTags: string[]) → TagSuggestion[] +embeddings:findDuplicates(threshold?: number) → DuplicatePair[] +embeddings:dismissPair(postIdA: string, postIdB: string) → void ``` +### Database: `embedding_keys` table + +Add to `src/main/database/schema.ts`. The current schema has: `projects`, `posts`, `media`, `settings`, `generatedFileHashes`, `postLinks`, `postMedia`, `tags`, `chatConversations`, `chatMessages`, `importDefinitions`, `scripts`, `templates`, `dbNotifications`, `modelCatalogProviders`, `modelCatalog`, `modelCatalogModalities`, `modelCatalogMeta`. + +New tables: +```ts +export const embeddingKeys = sqliteTable('embedding_keys', { + label: integer('label', { mode: 'bigint' }).primaryKey(), // USearch bigint key + postId: text('post_id').notNull(), + projectId: text('project_id').notNull(), +}); + +export const dismissedDuplicatePairs = sqliteTable('dismissed_duplicate_pairs', { + id: text('id').primaryKey(), + projectId: text('project_id').notNull(), + postIdA: text('post_id_a').notNull(), + postIdB: text('post_id_b').notNull(), + dismissedAt: integer('dismissed_at', { mode: 'timestamp' }).notNull(), +}, (table) => ({ + pairIdx: uniqueIndex('dismissed_pairs_idx').on(table.projectId, table.postIdA, table.postIdB), +})); +``` + +Create a Drizzle migration (`db:generate` / `db:migrate`) after adding the tables. + +### Project switching + +The app supports multiple projects. On project switch (`setProjectContext`), the engine must save and unload the current index, then load (or create) the index for the new project. Each project has its own `embeddings.usearch` file and `embedding_keys` table rows (filtered by `projectId`). + ### Embedding content Embed the raw markdown body of each post (title + content). Markdown's lightweight markup (headers, links, emphasis) adds minimal noise and preserves semantic structure well enough for transformer models. No stripping needed. @@ -97,16 +162,21 @@ This keeps the index simple (one vector per post, one lookup per query) while pr ### Hook into existing post lifecycle -Post create/update/delete events already exist in `PostEngine`. On post content change → call `embeddingEngine.embedPost()`. On delete → call `embeddingEngine.removePost()`. +`PostEngine` emits these events (confirmed in current codebase): +- `postCreated` — on create and on import +- `postUpdated` — on update, publish, revert +- `postDeleted` — on delete +- `databaseRebuilt` — emitted after `reconcileFromDisk()` (e.g., git sync replaces entire DB) +- `rebuildStarted` — emitted just before `databaseRebuilt` -Also listen for `databaseRebuilt` — emitted after `reconcileFromDisk()` (e.g., git sync). This replaces the entire DB, so individual post events don't fire. On `databaseRebuilt` → trigger a full reindex. +On post content change → call `embeddingEngine.embedPost()`. On delete → call `embeddingEngine.removePost()`. On `databaseRebuilt` → trigger a full reindex. Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation). During bulk indexing, batch-save every N posts (e.g., 100) instead of after each one — avoids 10k full file rewrites. ### Initial indexing (10k+ posts) - ~100ms per post × 10k = **~17 minutes** one-time background job -- Must run as a low-priority background task after app startup +- Must run as a low-priority background task after app startup (use `TaskManager` for queuing) - Emit progress events so UI can show "Indexing 3,421 / 10,247…" - On git sync to new machine, file watchers fire for all posts → triggers full reindex automatically - Model download (~470 MB) on first run — needs progress indicator or opt-in preference @@ -122,22 +192,171 @@ Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation) 2. Show results in the same result list format, with a subtle header like "Related posts" 3. Clicking a suggestion works identically to a search result — inserts the link -**When `query.length >= 2`:** existing search behavior, unchanged. +**When `query.length >= 2`:** existing search behavior, unchanged. (This includes the "Create post" option for link mode.) **Fallback:** if embeddings aren't ready (indexing in progress, feature disabled), show the existing "type at least 2 characters" message. --- +### TagInput (post tag editing) + +`TagInput` (`src/renderer/components/TagInput/TagInput.tsx`) already shows a suggestions dropdown driven by text input. Add a second suggestion source: tags inferred from semantically similar posts. + +Add an optional `postId?: string` prop. `PostEditor` already has `postId` and renders `TagInput`, so threading is a one-liner. + +**When `inputValue.length === 0` and the input is focused and `postId` is set:** +1. Call `embeddings:suggestTags(postId, currentTags)` once on focus (cache the result for the session) +2. Show a "Suggested" section at the top of the dropdown, above the regular tag list +3. Clicking a suggested tag adds it identically to any other tag + +**When `inputValue.length > 0`:** existing text-filter behavior, suggested section hidden. + +**Fallback:** if embeddings aren't ready or `postId` is absent, the dropdown behaves exactly as today — no visible change. + +**Algorithm** (in `EmbeddingEngine.suggestTags`): +1. Find top-10 similar posts via `findSimilar(postId, 10)` +2. Collect tags from each neighbour, weighted by similarity score +3. Sum weights per tag (tag appearing in 3 posts at 0.9 similarity scores higher than tag in 1 post at 0.95) +4. Filter out tags the current post already has +5. Return top 5 by weighted score + +```ts +interface TagSuggestion { + name: string; + score: number; // weighted frequency, for ranking only — not shown in UI +} + +async suggestTags(postId: string, excludeTags: string[]): Promise +``` + +New IPC endpoint (add to `handlers.ts`): +``` +embeddings:suggestTags(postId: string, excludeTags: string[]) → TagSuggestion[] +``` + +No new DB table needed — this is a pure read from the existing index. + +--- + +## Duplication Analysis + +A periodic audit tool to surface posts that are so semantically similar they might be unintentional duplicates — the same topic written twice years apart, a post and its draft that both got published, a cross-post that was forgotten. The goal is human review and action, not automated deletion. + +### Algorithm + +For each indexed post, query the index for its top-k nearest neighbours (k=20). Filter pairs where cosine similarity exceeds a threshold (default: 0.92). Deduplicate symmetric pairs (A→B = B→A). Sort descending by similarity. + +This is O(n) queries against the HNSW index — fast even at 10k posts (~20ms total). Run on demand, not continuously. + +```ts +interface DuplicatePair { + postA: { id: string; title: string; slug: string; publishedAt?: Date }; + postB: { id: string; title: string; slug: string; publishedAt?: Date }; + similarity: number; // cosine similarity, 0–1 +} +``` + +New engine method: +```ts +async findDuplicates(threshold?: number): Promise +// threshold default: 0.92. Lower = more results, more noise. Higher = only near-identical posts. +``` + +### Dismissed pairs + +When the user reviews a pair and decides it's intentional (e.g., two posts on the same topic that are meaningfully different), they can dismiss it. Store dismissed pairs in a new DB table so they don't reappear: + +```ts +export const dismissedDuplicatePairs = sqliteTable('dismissed_duplicate_pairs', { + id: text('id').primaryKey(), + projectId: text('project_id').notNull(), + postIdA: text('post_id_a').notNull(), + postIdB: text('post_id_b').notNull(), + dismissedAt: integer('dismissed_at', { mode: 'timestamp' }).notNull(), +}, (table) => ({ + pairIdx: uniqueIndex('dismissed_pairs_idx').on(table.projectId, table.postIdA, table.postIdB), +})); +``` + +`findDuplicates` filters out any pair where both (A, B) and (B, A) appear in `dismissed_duplicate_pairs`. + +### New IPC endpoints + +Add to `handlers.ts`: +``` +embeddings:findDuplicates(threshold?: number) → DuplicatePair[] +embeddings:dismissPair(postIdA: string, postIdB: string) → void +``` + +### UI placement + +The duplication analysis is a periodic audit task — the same category as `validateSite` and `metadataDiff`, both of which already live in the **Blog menu** in `src/main/shared/menuCommands.ts`. Add `findDuplicates` there, next to `validateSite`. + +Required changes to `menuCommands.ts`: +- Add `'findDuplicates'` to `AppMenuAction` union +- Add menu item to the Blog group next to `validateSite`: `{ label: 'menu.item.findDuplicates', action: 'findDuplicates' }` +- Add to `APP_MENU_ACTION_EVENT_MAP`: `findDuplicates: 'menu:findDuplicates'` + +The renderer listens for `menu:findDuplicates` and opens the duplicates tab (same pattern as `menu:validateSite` → `SiteValidationView`). + +Results open as a **dedicated tab** (new tab type: `duplicates`) in the main editor area, so the user can keep it open while navigating to individual posts. Tab type should be added to the `Tab` union in `appStore`. + +**Duplicates tab layout:** +``` +┌─────────────────────────────────────────────────────────┐ +│ Potential Duplicates [Threshold: ▼ 92%] [Re-run] │ +├─────────────────────────────────────────────────────────┤ +│ 97% "My trip to Berlin" (2019-03) │ +│ "Berlin travel notes" (2023-08) [Open both] │ +│ [Dismiss] │ +├─────────────────────────────────────────────────────────┤ +│ 94% "Bullet journaling setup" (2018-11) │ +│ "How I use a bullet journal" (2021-02)[Open both] │ +│ [Dismiss] │ +└─────────────────────────────────────────────────────────┘ +``` + +- **Threshold slider** — adjustable 80–99%, results update on re-run +- **"Open both"** — opens both posts as sequential editor tabs +- **"Dismiss"** — calls `embeddings:dismissPair`, removes the row from the list +- Results show similarity %, both post titles, and published dates +- If no duplicates found at the current threshold: "No duplicates found above X% similarity" +- If index not ready: "Semantic index is still building…" with progress + +### Python API + +Add to `bds_api` and `API.md`: +```python +posts.find_duplicates(threshold=0.92) # → list of DuplicatePair +posts.dismiss_duplicate_pair(post_id_a, post_id_b) # → None +``` + +--- + +## Settings: Opt-In Preference + +The feature must be opt-in (model download + 17 min indexing is not a silent default). + +Store as a project-level metadata field via `meta:updateProjectMetadata`. Add `semanticSimilarityEnabled: boolean` to `ProjectMetadata`. When the user enables it in Project Settings, start the background indexer. When disabled, skip embedding hooks and hide the UI section. + +The model download itself (~470 MB) should show a progress indicator before the indexer starts. + +--- + ## Implementation Steps -1. **Test + implement `EmbeddingEngine`** — model loading, embed, add/remove/query against USearch index, save/load persistence -2. **Drizzle key map table** — `embedding_keys` table mapping `bigint` label → post UUID -3. **Wire into post lifecycle** — hook create/update/delete → embedding updates -4. **Background indexer** — on startup, diff indexed vs. existing posts, queue unindexed for background embedding with progress events -5. **IPC endpoints** — `findSimilar`, `getProgress` -6. **InsertModal integration** — add `currentPostId` prop, fetch similar on mount, render as default suggestions -7. **Settings** — opt-in preference to enable semantic similarity (triggers model download + initial index) -8. **I18n** — all new UI strings through locale files +1. **Spike USearch packaging** — verify prebuilt binaries exist for all target Electron ABIs before committing. Fall back to `vectra` if they don't. +2. **Test + implement `EmbeddingEngine`** — model loading, embed, add/remove/query against USearch index, save/load persistence +3. **Drizzle key map table** — add `embedding_keys` to `schema.ts`, run `db:generate` + `db:migrate` +4. **Add `semanticSimilarityEnabled` to project metadata** — `ProjectMetadata` type + `meta:updateProjectMetadata` handler + Project Settings UI toggle +5. **Wire into post lifecycle** — hook `postCreated`/`postUpdated`/`postDeleted`/`databaseRebuilt` → embedding updates (guarded by opt-in flag) +6. **Background indexer** — on startup (if enabled), diff indexed vs. existing posts, queue unindexed for background embedding via `TaskManager` with progress events +7. **IPC endpoints** — `embeddings:findSimilar`, `embeddings:getProgress`, `embeddings:findDuplicates`, `embeddings:dismissPair` in `handlers.ts` +8. **Add `embeddingEngine` to `EngineBundle`** — update `EngineBundle.ts` interface and `main.ts` construction +9. **InsertModal integration** — add `currentPostId` prop, thread from `Editor.tsx`, fetch similar on mount, render as default suggestions +10. **Duplicates tab** — add `'findDuplicates'` to `AppMenuAction` + Blog menu group + `APP_MENU_ACTION_EVENT_MAP` in `menuCommands.ts`; add `duplicates` to `Tab` union in `appStore`; implement `DuplicatesView` component wired to `menu:findDuplicates` event +11. **I18n** — all new UI strings through locale files (no hardcoded text) +12. **Python API** — add `posts.findRelated(postId, k)`, `posts.find_duplicates(threshold)`, `posts.dismiss_duplicate_pair(a, b)` to `bds_api`, regenerate `API.md` --- @@ -148,3 +367,4 @@ Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation) - Model cached in `~/.cache/huggingface/`, index in internal project directory - Total added footprint: ~520 MB on disk (onnxruntime-node ~50 MB + model ~470 MB), ~300 MB RAM at runtime for model + index - Graceful degradation: if USearch native module fails to load (unsupported platform), disable the feature silently — never crash the app +- Follow test-first mandate: write failing tests before implementing `EmbeddingEngine`