hugo/RuDS
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

Implement embeddings, semantic search, and duplicate review #23

New Issue
Closed
opened 2026-07-18 20:48:45 +00:00 by hugo · 1 comment
hugo commented 2026-07-18 20:48:45 +00:00
Owner
Copy Link

Goal

Implement project-isolated local semantic indexing and expose similar posts, tag suggestions, semantic search, and duplicate review through working UI.

Current state

  • embedding_keys and dismissed_duplicate_pairs tables exist.
  • No embedding model, vector index, lifecycle engine, or semantic query path exists.
  • Find Duplicates is a workspace placeholder.
  • specs/embedding.allium and DuplicatesSurface in specs/editor_misc.allium are normative.

Required behavior

  • Load/cache the specified real multilingual local embedding model and use native acceleration where supported.
  • Generate embeddings from the normative post text representation and store content hashes/vector cache in SQLite.
  • Maintain a project-isolated vector index with debounced persistence and stable label-to-post mapping.
  • Embed or update posts after normal post mutations, remove deleted posts, index missing content, and support full rebuild.
  • Skip unchanged content by hash and recover a missing/corrupt index from the database cache.
  • Implement semantic search, similar-post lookup, similarity computation, and tag suggestions.
  • Find duplicate pairs above the configured threshold, page results, exclude self/reversed duplicates, and persist single/batch dismissals.
  • Replace Find Duplicates with a localized review UI linked to the relevant posts.
  • Respect the project semantic-similarity setting and avoid network access at runtime after the model is available.
  • Keep metadata diff and rebuild behavior synchronized with embedding lifecycle requirements.

Implementation notes

  • Read docs/UI_STYLE_GUIDE.md before UI work.
  • Keep SQLite as the authoritative vector cache and one index per project.
  • Reuse post mutation/rebuild hooks and the shared task/event paths.
  • Follow red/green TDD; do not substitute a lexical heuristic for the specified neural model.

Acceptance criteria

  • Model load/cache, hash skip, insert/update/delete, index persistence/recovery, and project isolation are tested.
  • Semantic search and similar-post results use deterministic fixtures with threshold/order assertions.
  • Duplicate symmetry, pagination, dismiss, batch dismiss, and rebuild are tested.
  • Post CRUD and filesystem rebuild leave database keys and index labels consistent.
  • Find Duplicates is a working localized UI with no placeholder.
  • cargo fmt --all -- --check, cargo build --workspace, and cargo test --workspace pass.
## Goal Implement project-isolated local semantic indexing and expose similar posts, tag suggestions, semantic search, and duplicate review through working UI. ## Current state - embedding_keys and dismissed_duplicate_pairs tables exist. - No embedding model, vector index, lifecycle engine, or semantic query path exists. - Find Duplicates is a workspace placeholder. - specs/embedding.allium and DuplicatesSurface in specs/editor_misc.allium are normative. ## Required behavior - Load/cache the specified real multilingual local embedding model and use native acceleration where supported. - Generate embeddings from the normative post text representation and store content hashes/vector cache in SQLite. - Maintain a project-isolated vector index with debounced persistence and stable label-to-post mapping. - Embed or update posts after normal post mutations, remove deleted posts, index missing content, and support full rebuild. - Skip unchanged content by hash and recover a missing/corrupt index from the database cache. - Implement semantic search, similar-post lookup, similarity computation, and tag suggestions. - Find duplicate pairs above the configured threshold, page results, exclude self/reversed duplicates, and persist single/batch dismissals. - Replace Find Duplicates with a localized review UI linked to the relevant posts. - Respect the project semantic-similarity setting and avoid network access at runtime after the model is available. - Keep metadata diff and rebuild behavior synchronized with embedding lifecycle requirements. ## Implementation notes - Read docs/UI_STYLE_GUIDE.md before UI work. - Keep SQLite as the authoritative vector cache and one index per project. - Reuse post mutation/rebuild hooks and the shared task/event paths. - Follow red/green TDD; do not substitute a lexical heuristic for the specified neural model. ## Acceptance criteria - Model load/cache, hash skip, insert/update/delete, index persistence/recovery, and project isolation are tested. - Semantic search and similar-post results use deterministic fixtures with threshold/order assertions. - Duplicate symmetry, pagination, dismiss, batch dismiss, and rebuild are tested. - Post CRUD and filesystem rebuild leave database keys and index labels consistent. - Find Duplicates is a working localized UI with no placeholder. - cargo fmt --all -- --check, cargo build --workspace, and cargo test --workspace pass.
hugo added the enhancement label 2026-07-18 20:48:45 +00:00
hugo commented 2026-07-19 16:07:46 +00:00
Author
Owner
Copy Link

Implemented and pushed in e876426. Added the real multilingual-e5-small ONNX backend with lazy local caching, query-prefix preprocessing, 384-dimensional normalized vectors, Core ML/DirectML acceleration with CPU fallback, project-isolated USearch HNSW indexes and durable SQLite BLOB cache. Added hash-skipping lifecycle hooks, batched cancellable backfill/reindex, debounced persistence, corrupt-index recovery, deletion/project cleanup, semantic sidebar search, similar-post/link ranking, tag suggestions, duplicate detection/pagination/canonical single and 100-row batch dismissals, localized native duplicate-review UI, menu actions, scripting API, metadata diff/repair, filesystem rebuild integration, migrations, docs, and plan updates. Neutral review compared the issue, bDS2 implementation/tests, specs/embedding.allium, and DuplicatesSurface in specs/editor_misc.allium; review fixes included stable-label upserts, enable-time backfill, managed progress/cancellation, cache cleanup, rebuild ordering, true batch inserts, and native-provider feature activation. Verification: real multilingual/cache model release test passed; cargo fmt --all -- --check passed; cargo build --workspace passed; cargo test --workspace passed (core 441 passed/3 intentionally ignored plus all UI/CLI/MCP/editor/integration/doc tests); cargo clippy --workspace --all-targets -- -D warnings passed; allium check specs returned zero findings.

Implemented and pushed in e876426. Added the real multilingual-e5-small ONNX backend with lazy local caching, query-prefix preprocessing, 384-dimensional normalized vectors, Core ML/DirectML acceleration with CPU fallback, project-isolated USearch HNSW indexes and durable SQLite BLOB cache. Added hash-skipping lifecycle hooks, batched cancellable backfill/reindex, debounced persistence, corrupt-index recovery, deletion/project cleanup, semantic sidebar search, similar-post/link ranking, tag suggestions, duplicate detection/pagination/canonical single and 100-row batch dismissals, localized native duplicate-review UI, menu actions, scripting API, metadata diff/repair, filesystem rebuild integration, migrations, docs, and plan updates. Neutral review compared the issue, bDS2 implementation/tests, specs/embedding.allium, and DuplicatesSurface in specs/editor_misc.allium; review fixes included stable-label upserts, enable-time backfill, managed progress/cancellation, cache cleanup, rebuild ordering, true batch inserts, and native-provider feature activation. Verification: real multilingual/cache model release test passed; cargo fmt --all -- --check passed; cargo build --workspace passed; cargo test --workspace passed (core 441 passed/3 intentionally ignored plus all UI/CLI/MCP/editor/integration/doc tests); cargo clippy --workspace --all-targets -- -D warnings passed; allium check specs returned zero findings.
hugo closed this issue 2026-07-19 16:07:46 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No Label enhancement
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
hugo
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: hugo/RuDS#23
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?