Claude/update beds documentation jdk5 y (#35)

* docs: update BDS_SEMANTIC_SIMILARITY.md for current app state

* docs: add duplication analysis feature to semantic similarity spec

* docs: add tag suggestion and duplicate check via Blog menu to similarity spec

---------

Co-authored-by: Claude <noreply@anthropic.com>

BDS_SEMANTIC_SIMILARITY.md

@@ -2,6 +2,8 @@
 
 Surface thematically related posts as an impulse — "Have I written something similar?" — inspired by Luhmann's Zettelkasten. Cross-domain connections across 10k+ posts over 20 years are the point, not a flaw. The algorithm finds the surface. The human finds the depth.
 
+**Status: Not yet implemented.** No packages installed, no engine, no IPC, no UI integration.
+
 ---
 
 ## Integration Point
@@ -10,7 +12,23 @@ Surface thematically related posts as an impulse — "Have I written something s
 
 When the search field is empty (`query.length < 2`), instead of showing "type at least 2 characters", show 3–5 semantically similar posts to the currently edited post. These are default suggestions — "posts you might want to link to."
 
-Requires threading `currentPostId` from `Editor.tsx` → `InsertModal` (currently only passes `currentPostTags` / `currentPostCategories`).
+The InsertModal currently accepts these props:
+```ts
+interface InsertModalProps {
+  mode: InsertMode;
+  onInsertLink: (url: string, text?: string) => void;
+  onInsertImage: (url: string, alt: string, mediaId?: string) => void;
+  onClose: () => void;
+  initialText?: string;
+  currentPostTags?: string[];
+  currentPostCategories?: string[];
+  // currentPostId is NOT yet threaded through — needs to be added
+}
+```
+
+`currentPostId` must be added to this interface and threaded from `Editor.tsx`.
+
+Note: The InsertModal now also has a **"Create post" option** — when `query.length >= 2` and no exact title match exists, it shows a `+` button to create a new post with that title and inherit the current post's tags/categories. This is the only UI addition since the original spec; it doesn't conflict with the semantic similarity integration.
 
 ---
 
@@ -21,6 +39,8 @@ Requires threading `currentPostId` from `Editor.tsx` → `InsertModal` (currentl
 | Embeddings | Hugging Face Transformers.js | `@huggingface/transformers` | ONNX, local, no API key |
 | Vector index | USearch | `usearch` | HNSW, native C++ via N-API, prebuilt binaries |
 
+Neither package is installed yet.
+
 **Embedding model:** `multilingual-e5-small` — 384 dimensions, 512-token context, ~470 MB on disk, ~200–300 MB RAM, ~100ms/post inference. Natively multilingual (100+ languages incl. DE/EN) — critical for a mixed-language blog. `all-MiniLM-L6-v2` (~90 MB) was considered but is EN-trained with weak DE transfer; not suitable for nuanced cross-language similarity.
 
 **Why USearch over alternatives:**
@@ -52,6 +72,8 @@ The `bigint → postId` key mapping lives in a Drizzle table (`embedding_keys`),
 
 ### Engine: `EmbeddingEngine` (`src/main/engine/EmbeddingEngine.ts`)
 
+File does not exist yet. Create it.
+
 Responsibilities:
 - Load/save USearch index + key map on startup/shutdown
 - Embed post content via `@huggingface/transformers`
@@ -72,17 +94,60 @@ class EmbeddingEngine {
 }
 ```
 
-### Project switching
+### EngineBundle (`src/main/engine/EngineBundle.ts`)
 
-The app supports multiple projects. On project switch (`setProjectContext`), the engine must save and unload the current index, then load (or create) the index for the new project. Each project has its own `embeddings.usearch` file and `embedding_keys` table rows.
+Add `embeddingEngine: EmbeddingEngine` to the `EngineBundle` interface. The current bundle contains:
+`postEngine`, `mediaEngine`, `scriptEngine`, `templateEngine`, `metaEngine`, `menuEngine`, `tagEngine`, `postMediaEngine`, `projectEngine`, `gitEngine`, `gitApiAdapter`, `blogGenerationEngine`, `publishEngine`, `metadataDiffEngine`, `taskManager`, `blogmarkTransformService`, `mcpServer`, `blogmarkPythonWorkerRuntime`, `pythonMacroWorkerRuntime`, `publishApiAdapter`, `appApiAdapter`.
 
-### IPC
+### IPC layer (`src/main/ipc/`)
+
+The IPC layer now has five files:
+- `handlers.ts` — main handler file (posts, media, project, meta, tags, templates, scripts, blog generation, publishing, preview, site validation, import, settings, model catalog, tasks, notifications)
+- `chatHandlers.ts` — AI chat streaming & tool use
+- `blogHandlers.ts` — blog generation & publishing
+- `publishHandlers.ts` — publishing
+- `metadataDiffHandlers.ts` — metadata diff
+- `index.ts` — module exports
+
+Add the embedding IPC handlers to `handlers.ts` (they're small; no need for a new file):
 
 ```
 embeddings:findSimilar(postId: string, k?: number) → SimilarPost[]
 embeddings:getProgress() → { indexed: number; total: number }
+embeddings:suggestTags(postId: string, excludeTags: string[]) → TagSuggestion[]
+embeddings:findDuplicates(threshold?: number) → DuplicatePair[]
+embeddings:dismissPair(postIdA: string, postIdB: string) → void
 ```
 
+### Database: `embedding_keys` table
+
+Add to `src/main/database/schema.ts`. The current schema has: `projects`, `posts`, `media`, `settings`, `generatedFileHashes`, `postLinks`, `postMedia`, `tags`, `chatConversations`, `chatMessages`, `importDefinitions`, `scripts`, `templates`, `dbNotifications`, `modelCatalogProviders`, `modelCatalog`, `modelCatalogModalities`, `modelCatalogMeta`.
+
+New tables:
+```ts
+export const embeddingKeys = sqliteTable('embedding_keys', {
+  label: integer('label', { mode: 'bigint' }).primaryKey(), // USearch bigint key
+  postId: text('post_id').notNull(),
+  projectId: text('project_id').notNull(),
+});
+
+export const dismissedDuplicatePairs = sqliteTable('dismissed_duplicate_pairs', {
+  id: text('id').primaryKey(),
+  projectId: text('project_id').notNull(),
+  postIdA: text('post_id_a').notNull(),
+  postIdB: text('post_id_b').notNull(),
+  dismissedAt: integer('dismissed_at', { mode: 'timestamp' }).notNull(),
+}, (table) => ({
+  pairIdx: uniqueIndex('dismissed_pairs_idx').on(table.projectId, table.postIdA, table.postIdB),
+}));
+```
+
+Create a Drizzle migration (`db:generate` / `db:migrate`) after adding the tables.
+
+### Project switching
+
+The app supports multiple projects. On project switch (`setProjectContext`), the engine must save and unload the current index, then load (or create) the index for the new project. Each project has its own `embeddings.usearch` file and `embedding_keys` table rows (filtered by `projectId`).
+
 ### Embedding content
 
 Embed the raw markdown body of each post (title + content). Markdown's lightweight markup (headers, links, emphasis) adds minimal noise and preserves semantic structure well enough for transformer models. No stripping needed.
@@ -97,16 +162,21 @@ This keeps the index simple (one vector per post, one lookup per query) while pr
 
 ### Hook into existing post lifecycle
 
-Post create/update/delete events already exist in `PostEngine`. On post content change → call `embeddingEngine.embedPost()`. On delete → call `embeddingEngine.removePost()`.
+`PostEngine` emits these events (confirmed in current codebase):
+- `postCreated` — on create and on import
+- `postUpdated` — on update, publish, revert
+- `postDeleted` — on delete
+- `databaseRebuilt` — emitted after `reconcileFromDisk()` (e.g., git sync replaces entire DB)
+- `rebuildStarted` — emitted just before `databaseRebuilt`
 
-Also listen for `databaseRebuilt` — emitted after `reconcileFromDisk()` (e.g., git sync). This replaces the entire DB, so individual post events don't fire. On `databaseRebuilt` → trigger a full reindex.
+On post content change → call `embeddingEngine.embedPost()`. On delete → call `embeddingEngine.removePost()`. On `databaseRebuilt` → trigger a full reindex.
 
 Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation). During bulk indexing, batch-save every N posts (e.g., 100) instead of after each one — avoids 10k full file rewrites.
 
 ### Initial indexing (10k+ posts)
 
 - ~100ms per post × 10k = **~17 minutes** one-time background job
-- Must run as a low-priority background task after app startup
+- Must run as a low-priority background task after app startup (use `TaskManager` for queuing)
 - Emit progress events so UI can show "Indexing 3,421 / 10,247…"
 - On git sync to new machine, file watchers fire for all posts → triggers full reindex automatically
 - Model download (~470 MB) on first run — needs progress indicator or opt-in preference
@@ -122,22 +192,171 @@ Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation)
 2. Show results in the same result list format, with a subtle header like "Related posts"
 3. Clicking a suggestion works identically to a search result — inserts the link
 
-**When `query.length >= 2`:** existing search behavior, unchanged.
+**When `query.length >= 2`:** existing search behavior, unchanged. (This includes the "Create post" option for link mode.)
 
 **Fallback:** if embeddings aren't ready (indexing in progress, feature disabled), show the existing "type at least 2 characters" message.
 
 ---
 
+### TagInput (post tag editing)
+
+`TagInput` (`src/renderer/components/TagInput/TagInput.tsx`) already shows a suggestions dropdown driven by text input. Add a second suggestion source: tags inferred from semantically similar posts.
+
+Add an optional `postId?: string` prop. `PostEditor` already has `postId` and renders `TagInput`, so threading is a one-liner.
+
+**When `inputValue.length === 0` and the input is focused and `postId` is set:**
+1. Call `embeddings:suggestTags(postId, currentTags)` once on focus (cache the result for the session)
+2. Show a "Suggested" section at the top of the dropdown, above the regular tag list
+3. Clicking a suggested tag adds it identically to any other tag
+
+**When `inputValue.length > 0`:** existing text-filter behavior, suggested section hidden.
+
+**Fallback:** if embeddings aren't ready or `postId` is absent, the dropdown behaves exactly as today — no visible change.
+
+**Algorithm** (in `EmbeddingEngine.suggestTags`):
+1. Find top-10 similar posts via `findSimilar(postId, 10)`
+2. Collect tags from each neighbour, weighted by similarity score
+3. Sum weights per tag (tag appearing in 3 posts at 0.9 similarity scores higher than tag in 1 post at 0.95)
+4. Filter out tags the current post already has
+5. Return top 5 by weighted score
+
+```ts
+interface TagSuggestion {
+  name: string;
+  score: number;  // weighted frequency, for ranking only — not shown in UI
+}
+
+async suggestTags(postId: string, excludeTags: string[]): Promise<TagSuggestion[]>
+```
+
+New IPC endpoint (add to `handlers.ts`):
+```
+embeddings:suggestTags(postId: string, excludeTags: string[]) → TagSuggestion[]
+```
+
+No new DB table needed — this is a pure read from the existing index.
+
+---
+
+## Duplication Analysis
+
+A periodic audit tool to surface posts that are so semantically similar they might be unintentional duplicates — the same topic written twice years apart, a post and its draft that both got published, a cross-post that was forgotten. The goal is human review and action, not automated deletion.
+
+### Algorithm
+
+For each indexed post, query the index for its top-k nearest neighbours (k=20). Filter pairs where cosine similarity exceeds a threshold (default: 0.92). Deduplicate symmetric pairs (A→B = B→A). Sort descending by similarity.
+
+This is O(n) queries against the HNSW index — fast even at 10k posts (~20ms total). Run on demand, not continuously.
+
+```ts
+interface DuplicatePair {
+  postA: { id: string; title: string; slug: string; publishedAt?: Date };
+  postB: { id: string; title: string; slug: string; publishedAt?: Date };
+  similarity: number; // cosine similarity, 0–1
+}
+```
+
+New engine method:
+```ts
+async findDuplicates(threshold?: number): Promise<DuplicatePair[]>
+// threshold default: 0.92. Lower = more results, more noise. Higher = only near-identical posts.
+```
+
+### Dismissed pairs
+
+When the user reviews a pair and decides it's intentional (e.g., two posts on the same topic that are meaningfully different), they can dismiss it. Store dismissed pairs in a new DB table so they don't reappear:
+
+```ts
+export const dismissedDuplicatePairs = sqliteTable('dismissed_duplicate_pairs', {
+  id: text('id').primaryKey(),
+  projectId: text('project_id').notNull(),
+  postIdA: text('post_id_a').notNull(),
+  postIdB: text('post_id_b').notNull(),
+  dismissedAt: integer('dismissed_at', { mode: 'timestamp' }).notNull(),
+}, (table) => ({
+  pairIdx: uniqueIndex('dismissed_pairs_idx').on(table.projectId, table.postIdA, table.postIdB),
+}));
+```
+
+`findDuplicates` filters out any pair where both (A, B) and (B, A) appear in `dismissed_duplicate_pairs`.
+
+### New IPC endpoints
+
+Add to `handlers.ts`:
+```
+embeddings:findDuplicates(threshold?: number) → DuplicatePair[]
+embeddings:dismissPair(postIdA: string, postIdB: string) → void
+```
+
+### UI placement
+
+The duplication analysis is a periodic audit task — the same category as `validateSite` and `metadataDiff`, both of which already live in the **Blog menu** in `src/main/shared/menuCommands.ts`. Add `findDuplicates` there, next to `validateSite`.
+
+Required changes to `menuCommands.ts`:
+- Add `'findDuplicates'` to `AppMenuAction` union
+- Add menu item to the Blog group next to `validateSite`: `{ label: 'menu.item.findDuplicates', action: 'findDuplicates' }`
+- Add to `APP_MENU_ACTION_EVENT_MAP`: `findDuplicates: 'menu:findDuplicates'`
+
+The renderer listens for `menu:findDuplicates` and opens the duplicates tab (same pattern as `menu:validateSite` → `SiteValidationView`).
+
+Results open as a **dedicated tab** (new tab type: `duplicates`) in the main editor area, so the user can keep it open while navigating to individual posts. Tab type should be added to the `Tab` union in `appStore`.
+
+**Duplicates tab layout:**
+```
+┌─────────────────────────────────────────────────────────┐
+│ Potential Duplicates   [Threshold: ▼ 92%]  [Re-run]    │
+├─────────────────────────────────────────────────────────┤
+│ 97%  "My trip to Berlin" (2019-03)                      │
+│      "Berlin travel notes" (2023-08)       [Open both]  │
+│                                            [Dismiss]    │
+├─────────────────────────────────────────────────────────┤
+│ 94%  "Bullet journaling setup" (2018-11)               │
+│      "How I use a bullet journal" (2021-02)[Open both]  │
+│                                            [Dismiss]    │
+└─────────────────────────────────────────────────────────┘
+```
+
+- **Threshold slider** — adjustable 80–99%, results update on re-run
+- **"Open both"** — opens both posts as sequential editor tabs
+- **"Dismiss"** — calls `embeddings:dismissPair`, removes the row from the list
+- Results show similarity %, both post titles, and published dates
+- If no duplicates found at the current threshold: "No duplicates found above X% similarity"
+- If index not ready: "Semantic index is still building…" with progress
+
+### Python API
+
+Add to `bds_api` and `API.md`:
+```python
+posts.find_duplicates(threshold=0.92)  # → list of DuplicatePair
+posts.dismiss_duplicate_pair(post_id_a, post_id_b)  # → None
+```
+
+---
+
+## Settings: Opt-In Preference
+
+The feature must be opt-in (model download + 17 min indexing is not a silent default).
+
+Store as a project-level metadata field via `meta:updateProjectMetadata`. Add `semanticSimilarityEnabled: boolean` to `ProjectMetadata`. When the user enables it in Project Settings, start the background indexer. When disabled, skip embedding hooks and hide the UI section.
+
+The model download itself (~470 MB) should show a progress indicator before the indexer starts.
+
+---
+
 ## Implementation Steps
 
-1. **Test + implement `EmbeddingEngine`** — model loading, embed, add/remove/query against USearch index, save/load persistence
+1. **Spike USearch packaging** — verify prebuilt binaries exist for all target Electron ABIs before committing. Fall back to `vectra` if they don't.
-2. **Drizzle key map table** — `embedding_keys` table mapping `bigint` label → post UUID
+2. **Test + implement `EmbeddingEngine`** — model loading, embed, add/remove/query against USearch index, save/load persistence
-3. **Wire into post lifecycle** — hook create/update/delete → embedding updates
+3. **Drizzle key map table** — add `embedding_keys` to `schema.ts`, run `db:generate` + `db:migrate`
-4. **Background indexer** — on startup, diff indexed vs. existing posts, queue unindexed for background embedding with progress events
+4. **Add `semanticSimilarityEnabled` to project metadata** — `ProjectMetadata` type + `meta:updateProjectMetadata` handler + Project Settings UI toggle
-5. **IPC endpoints** — `findSimilar`, `getProgress`
+5. **Wire into post lifecycle** — hook `postCreated`/`postUpdated`/`postDeleted`/`databaseRebuilt` → embedding updates (guarded by opt-in flag)
-6. **InsertModal integration** — add `currentPostId` prop, fetch similar on mount, render as default suggestions
+6. **Background indexer** — on startup (if enabled), diff indexed vs. existing posts, queue unindexed for background embedding via `TaskManager` with progress events
-7. **Settings** — opt-in preference to enable semantic similarity (triggers model download + initial index)
+7. **IPC endpoints** — `embeddings:findSimilar`, `embeddings:getProgress`, `embeddings:findDuplicates`, `embeddings:dismissPair` in `handlers.ts`
-8. **I18n** — all new UI strings through locale files
+8. **Add `embeddingEngine` to `EngineBundle`** — update `EngineBundle.ts` interface and `main.ts` construction
+9. **InsertModal integration** — add `currentPostId` prop, thread from `Editor.tsx`, fetch similar on mount, render as default suggestions
+10. **Duplicates tab** — add `'findDuplicates'` to `AppMenuAction` + Blog menu group + `APP_MENU_ACTION_EVENT_MAP` in `menuCommands.ts`; add `duplicates` to `Tab` union in `appStore`; implement `DuplicatesView` component wired to `menu:findDuplicates` event
+11. **I18n** — all new UI strings through locale files (no hardcoded text)
+12. **Python API** — add `posts.findRelated(postId, k)`, `posts.find_duplicates(threshold)`, `posts.dismiss_duplicate_pair(a, b)` to `bds_api`, regenerate `API.md`
 
 ---
 
@@ -148,3 +367,4 @@ Save strategy: debounce `index.save()` on a timer (e.g., 5s after last mutation)
 - Model cached in `~/.cache/huggingface/`, index in internal project directory
 - Total added footprint: ~520 MB on disk (onnxruntime-node ~50 MB + model ~470 MB), ~300 MB RAM at runtime for model + index
 - Graceful degradation: if USearch native module fails to load (unsupported platform), disable the feature silently — never crash the app
+- Follow test-first mandate: write failing tests before implementing `EmbeddingEngine`