fix: force full re-embed on explicit rebuild and degrade gracefully when embedding model is unavailable

SPECGAPS.md

@@ -24,7 +24,7 @@ Gap categories: **SC** = spec correct, fix code | **CS** = code correct, update
 | A1-12 | ~~Real Pagefind integration for search~~ | generation.allium:208 | Functional client-side search: `PagefindUI` defined in bundled `pagefind-ui.js`, fragment index records url/title/body-scoped text per page, search-runtime wires it up | **Resolved:** bundled real `PagefindUI` (fetch index, ranked full-text match, highlighted excerpts) + `pagefind-ui.css` as local assets read into `Pagefind`; index scoped to `data-pagefind-body` (unmarked pages excluded per PagefindHtmlMarking), title from `<title>`/`<h1>`; localized "No results found" label via `data-search-no-results` (de/fr/it/es); 3 unit tests added |
 | A1-13 | ~~Git sidebar shows only "Working tree" placeholder~~ | sidebar_views.allium:651-770 | `git_view/1` now builds a full `layout: "git"` view from `BDS.Git` (repository/remote_state/status/history); `SidebarComponents` renders active + not_a_repo states | **Resolved:** `git_view/1` in sidebar.ex assembles branch/upstream/ahead/behind, status files, paginated history (20/page); `render_git_sidebar` renders branch header, sync legend, fetch/pull/push/prune-lfs buttons, commit form, clickable status files (open git_diff), history entries; shell_live wires `git_commit` (closes git_diff tabs), `git_fetch`/`git_pull`/`git_push`/`git_prune_lfs`, `git_initialize`; `BDS.Git.history` enriched with author/date, `BDS.Git.set_remote/2` added; i18n for de/fr/it/es; 3 shell tests + git author/date assertions added |
 | A1-14 | ~~Embedding uses TF-IDF hash projection instead of real neural model~~ | embedding.allium:44-53, invariants RealNeuralModel/ModelCaching/VectorCacheInDb | `Backends.Neural` runs `intfloat/multilingual-e5-small` (e5 weights behind the Xenova id) via Bumblebee+EXLA | **Resolved (core):** added bumblebee/nx/exla deps; `Backends.Neural` is a lazily-loaded GenServer that builds the Bumblebee text-embedding serving on first request (`"query: "` prefix + mean pooling + L2 norm), downloads+caches the model under the app data dir (ModelCaching), and is wired into the supervision tree when configured; vectors now persisted as packed little-endian Float32 BLOB (384×4=1536 bytes) instead of JSON text (VectorCacheInDb) with migration recreating `embedding_keys.vector` as BLOB; `InApp` demoted to documented offline/test stub; test config uses the stub so the suite stays offline; spec EmbeddingModel clarified (Xenova id ↔ intfloat weights via Bumblebee); batched inference via optional `embed_many/2` backend callback (configurable `batch_size`/`sequence_length`; rebuild/index/repair embed in chunks instead of one post at a time) + `NativeAcceleratedExecution` invariant added to spec; 4 tests added (BLOB round-trip, batched-rebuild, Neural model_info/behaviour). **Deferred:** A1-14b USearch HNSW index, A1-14c Apple GPU (EMLX). |
-| A1-14b | ~~USearch HNSW ANN index + debounced persistence not implemented~~ | embedding.allium config/FindSimilar/DebouncedPersistence | `Embeddings.Index` is now an HNSW (hnswlib) ANN index with debounced persistence | **Resolved:** rewrote `Embeddings.Index` as a DB-free GenServer wrapping an hnswlib HNSW graph (cosine, M=16, efConstruction=128, efSearch=64) — O(n·log n) build, O(log n) queries, replacing the O(n²) JSON cosine snapshot; per-project in-memory index + `label→post_id` map; 5s debounced `save_index` + `.meta.json` sidecar, force-save on project switch (`set_active_project`) and shutdown (`terminate`), `forget/1` on project delete; lazy reload from disk with rebuild-from-DB self-heal on miss; `find_similar`/`find_duplicates`/`compute_similarities` rewired (no brute-force fallback); USearch has no Elixir binding so hnswlib provides the identical HNSW algorithm/params (spec reconciled); supervision + dialyzer PLT updated; tests updated for debounced/binary persistence + self-heal. |
+| A1-14b | ~~USearch HNSW ANN index + debounced persistence not implemented~~ | embedding.allium config/FindSimilar/DebouncedPersistence | `Embeddings.Index` is now an HNSW (hnswlib) ANN index with debounced persistence | **Resolved:** rewrote `Embeddings.Index` as a DB-free GenServer wrapping an hnswlib HNSW graph (cosine, M=16, efConstruction=128, efSearch=64) — O(n·log n) build, O(log n) queries, replacing the O(n²) JSON cosine snapshot; per-project in-memory index + `label→post_id` map; 5s debounced `save_index` + `.meta.json` sidecar, force-save on project switch (`set_active_project`) and shutdown (`terminate`), `forget/1` on project delete; lazy reload from disk with rebuild-from-DB self-heal on miss; `find_similar`/`find_duplicates`/`compute_similarities` rewired (no brute-force fallback); USearch has no Elixir binding so hnswlib provides the identical HNSW algorithm/params (spec reconciled); supervision + dialyzer PLT updated; tests updated for debounced/binary persistence + self-heal. Follow-up hardening: explicit rebuild now forces re-embedding regardless of content_hash (ReindexAll), and model-unavailable errors propagate cleanly (post saves degrade to unindexed + log; rebuild/index return `{:error, reason}` surfaced as a failed task with a user-facing message instead of crashing). |
 | A1-14c | Embedding model runs on CPU only; no Apple GPU acceleration | embedding.allium invariant NativeAcceleratedExecution | `Backends.Neural` uses Bumblebee+EXLA; on Apple Silicon XLA has no Metal backend so inference is native CPU (batched). Apple GPU/Neural Engine unused | Fix code: spike an EMLX (Apple MLX) Nx backend so the model executes on the Apple Silicon GPU; gate by platform/availability with EXLA-CPU fallback; verify Bumblebee serving + defn compiler compatibility and benchmark vs CPU batching |
 | A1-15 | ~~Preview vs generation content source strategy undocumented~~ | preview.allium (no invariant), generation.allium (no invariant) | Generation uses only published .md file content (`Generation.Data` snapshots set `content: nil`); preview includes published+draft posts and prefers DB content over file (`Preview.Router` queries `:published`/`:draft`, uses `editor_body`) | **Resolved:** added `PreviewDraftOverlay` invariant to preview.allium and `GenerationPublishedOnly` invariant to generation.allium; both cross-reference each other; code already correct, 3 tests added for draft-in-preview behavior |
 

lib/bds/desktop/shell_commands.ex

@@ -120,16 +120,7 @@ defmodule BDS.Desktop.ShellCommands do
       "rebuild_embedding_index",
       "Rebuild Embedding Index",
       "Embeddings",
-      fn report ->
+      fn report -> rebuild_embedding_index_work(project, report) end
-        {:ok, rebuilt_post_ids} = Embeddings.rebuild_project(project.id, on_progress: report)
-        report.(1.0, "Embedding index rebuilt")
-
-        %{
-          project_id: project.id,
-          rebuilt_post_ids: rebuilt_post_ids,
-          rebuilt_count: length(rebuilt_post_ids)
-        }
-      end
     )
   end
 
@@ -524,8 +515,14 @@ defmodule BDS.Desktop.ShellCommands do
       },
       %{
         name: "Rebuild Embedding Index",
-        work: fn report ->
+        work: fn report -> rebuild_embedding_index_work(project, report) end
-          {:ok, rebuilt_post_ids} = Embeddings.rebuild_project(project.id, on_progress: report)
+      }
+    ]
+  end
+
+  defp rebuild_embedding_index_work(project, report) do
+    case Embeddings.rebuild_project(project.id, on_progress: report) do
+      {:ok, rebuilt_post_ids} ->
         report.(1.0, "Embedding index rebuilt")
 
         %{
@@ -533,9 +530,22 @@ defmodule BDS.Desktop.ShellCommands do
           rebuilt_post_ids: rebuilt_post_ids,
           rebuilt_count: length(rebuilt_post_ids)
         }
+
+      {:error, reason} ->
+        {:error, embedding_error_message(reason)}
     end
-      }
+  end
-    ]
+
+  defp embedding_error_message(reason) do
+    detail =
+      case reason do
+        message when is_binary(message) -> message
+        {:embedding_backend_unavailable, _inner} -> "the embedding service did not start"
+        other -> inspect(other)
+      end
+
+    "Could not build the embedding index: #{detail}. The model is downloaded on first use, " <>
+      "so check your internet connection — or turn off semantic similarity in Settings."
   end
 
   defp run_rebuild_sequence(_group_id, _attrs, []), do: :ok

lib/bds/embeddings.ex

@@ -2,6 +2,7 @@ defmodule BDS.Embeddings do
   @moduledoc false
 
   import Ecto.Query
+  require Logger
 
   alias BDS.Persistence
   alias BDS.Embeddings.DismissedDuplicatePair
@@ -75,11 +76,16 @@ defmodule BDS.Embeddings do
         )
 
       existing_keys = preload_keys_by_post_id(project_id, Enum.map(posts, & &1.id))
-      rows = build_key_rows(posts, existing_keys, max_label_value(), nil)
 
+      case build_key_rows(posts, existing_keys, max_label_value(), nil, false) do
+        {:ok, rows} ->
           batch_upsert_keys(rows)
           :ok = rebuild_snapshot(project_id)
           {:ok, Enum.map(posts, & &1.id)}
+
+        {:error, _reason} = error ->
+          error
+      end
     else
       {:ok, []}
     end
@@ -106,13 +112,19 @@ defmodule BDS.Embeddings do
       )
 
       existing_keys = preload_keys_by_post_id(project_id)
-      rows = build_key_rows(posts, existing_keys, max_label_value(), on_progress)
 
+      # An explicit rebuild re-embeds every post from scratch (ReindexAll),
+      # ignoring the content_hash skip optimisation.
+      case build_key_rows(posts, existing_keys, max_label_value(), on_progress, true) do
+        {:ok, rows} ->
           batch_upsert_keys(rows)
-
           :ok = report_rebuild_phase(on_progress, 0.99, "Persisting embedding snapshot")
           :ok = rebuild_snapshot(project_id)
           {:ok, post_ids}
+
+        {:error, _reason} = error ->
+          error
+      end
     else
       {:ok, []}
     end
@@ -172,8 +184,9 @@ defmodule BDS.Embeddings do
         :ok
 
       existing_key ->
+        case embed_text(raw_text, post.language) do
+          {:ok, vector} ->
             label = existing_key_label(existing_key) || next_label()
-        {:ok, vector} = embed_text(raw_text, post.language)
 
             (existing_key || %Key{})
             |> Key.changeset(%{
@@ -190,6 +203,17 @@ defmodule BDS.Embeddings do
             end
 
             :ok
+
+          {:error, reason} ->
+            # Embedding is best-effort on post save: if the model is unavailable
+            # (e.g. offline first-use download), leave the post unindexed rather
+            # than failing the save. An explicit reindex surfaces the error.
+            Logger.warning(
+              "Embedding unavailable for post #{post.id}: #{inspect(reason)}; left unindexed"
+            )
+
+            :ok
+        end
     end
   end
 
@@ -210,11 +234,12 @@ defmodule BDS.Embeddings do
     Repo.one(from key in Key, select: max(key.label)) || 0
   end
 
-  # Builds the upsert rows for a batch of posts. Posts whose content_hash is
+  # Builds the upsert rows for a batch of posts. Unless `force?` is set, posts
-  # unchanged are skipped (ContentHashSkipsUnchanged); the rest are embedded in
+  # whose content_hash is unchanged are skipped (ContentHashSkipsUnchanged); the
-  # batches (see embed_pending/2) so model inference is not serialised one post
+  # rest are embedded in batches (see embed_pending/2) so model inference is not
-  # at a time. Labels keep their existing value or take the next free integer.
+  # serialised one post at a time. Labels keep their existing value or take the
-  defp build_key_rows(posts, existing_keys, base_label, on_progress) do
+  # next free integer. Returns `{:error, reason}` if the model is unavailable.
+  defp build_key_rows(posts, existing_keys, base_label, on_progress, force?) do
     prepared =
       Enum.map(posts, fn post ->
         raw_text = compose_embedding_source(post.title, resolve_post_body(post))
@@ -226,14 +251,20 @@ defmodule BDS.Embeddings do
           existing: existing,
           raw_text: raw_text,
           content_hash: content_hash,
-          needs_embed?: is_nil(existing) or existing.content_hash != content_hash
+          needs_embed?: force? or is_nil(existing) or existing.content_hash != content_hash
         }
       end)
 
     pending = Enum.filter(prepared, & &1.needs_embed?)
     :ok = report_rebuild_started(on_progress, length(pending), "embedding entries")
-    vectors_by_post_id = embed_pending(pending, on_progress)
 
+    case embed_pending(pending, on_progress) do
+      {:ok, vectors_by_post_id} -> {:ok, collect_rows(prepared, vectors_by_post_id, base_label)}
+      {:error, _reason} = error -> error
+    end
+  end
+
+  defp collect_rows(prepared, vectors_by_post_id, base_label) do
     {rows, _next_label} =
       Enum.reduce(prepared, {[], base_label + 1}, fn entry, {acc, next_label} ->
         if entry.needs_embed? do
@@ -258,7 +289,7 @@ defmodule BDS.Embeddings do
     rows
   end
 
-  defp embed_pending([], _on_progress), do: %{}
+  defp embed_pending([], _on_progress), do: {:ok, %{}}
 
   defp embed_pending(pending, on_progress) do
     total = length(pending)
@@ -268,12 +299,12 @@ defmodule BDS.Embeddings do
     # Group by language so the lexical stub stems consistently; the neural
     # backend is multilingual and ignores the language hint.
     |> Enum.group_by(& &1.post.language)
-    |> Enum.reduce({%{}, 0}, fn {language, group}, acc ->
+    |> Enum.reduce_while({%{}, 0}, fn {language, group}, acc ->
       group
       |> Enum.chunk_every(batch)
-      |> Enum.reduce(acc, fn chunk, {vectors, done} ->
+      |> Enum.reduce_while(acc, fn chunk, {vectors, done} ->
-        {:ok, chunk_vectors} = embed_many(Enum.map(chunk, & &1.raw_text), language)
+        case embed_many(Enum.map(chunk, & &1.raw_text), language) do
-
+          {:ok, chunk_vectors} ->
             vectors =
               chunk
               |> Enum.zip(chunk_vectors)
@@ -283,10 +314,21 @@ defmodule BDS.Embeddings do
 
             done = done + length(chunk)
             :ok = report_rebuild_progress(on_progress, done, total, "embedding entries")
-        {vectors, done}
+            {:cont, {vectors, done}}
+
+          {:error, reason} ->
+            {:halt, {:error, reason}}
+        end
       end)
+      |> case do
+        {:error, reason} -> {:halt, {:error, reason}}
+        accumulator -> {:cont, accumulator}
+      end
     end)
-    |> elem(0)
+    |> case do
+      {:error, reason} -> {:error, reason}
+      {vectors, _done} -> {:ok, vectors}
+    end
   end
 
   defp batch_upsert_keys([]), do: :ok
@@ -337,8 +379,9 @@ defmodule BDS.Embeddings do
         )
 
       existing_keys = preload_keys_by_post_id(project_id)
-      rows = build_key_rows(posts, existing_keys, max_label_value(), nil)
 
+      case build_key_rows(posts, existing_keys, max_label_value(), nil, false) do
+        {:ok, rows} ->
           batch_upsert_keys(rows)
           :ok = rebuild_snapshot(project_id)
 
@@ -346,6 +389,10 @@ defmodule BDS.Embeddings do
             Repo.all(from key in Key, where: key.project_id == ^project_id, select: key.post_id)
 
           {:ok, indexed}
+
+        {:error, _reason} = error ->
+          error
+      end
     else
       {:ok, []}
     end
@@ -677,13 +724,16 @@ defmodule BDS.Embeddings do
     if function_exported?(backend, :embed_many, 2) do
       backend.embed_many(texts, language: language)
     else
-      vectors =
+      Enum.reduce_while(texts, {:ok, []}, fn text, {:ok, acc} ->
-        Enum.map(texts, fn text ->
+        case backend.embed(text, language: language) do
-          {:ok, vector} = backend.embed(text, language: language)
+          {:ok, vector} -> {:cont, {:ok, [vector | acc]}}
-          vector
+          {:error, _reason} = error -> {:halt, error}
+        end
       end)
-
+      |> case do
-      {:ok, vectors}
+        {:ok, vectors} -> {:ok, Enum.reverse(vectors)}
+        {:error, _reason} = error -> error
+      end
     end
   end
 

lib/bds/maintenance/repair.ex

@@ -101,11 +101,18 @@ defmodule BDS.Maintenance.Repair do
           :file_to_db ->
             post_ids = Enum.map(items, &metadata_diff_item_entity_id/1)
 
-            {:ok, repaired_post_ids} = Embeddings.repair_posts(project_id, post_ids)
+            # If the embedding model is unavailable, every item is reported as
-            repaired_post_ids = MapSet.new(repaired_post_ids)
+            # failed rather than crashing the repair task.
+            repaired =
+              case Embeddings.repair_posts(project_id, post_ids) do
+                {:ok, repaired_post_ids} -> repaired_post_ids
+                {:error, _reason} -> []
+              end
+
+            repaired_set = MapSet.new(repaired)
 
             build_batch_repair_result(items, total, on_progress, fn item ->
-              MapSet.member?(repaired_post_ids, metadata_diff_item_entity_id(item))
+              MapSet.member?(repaired_set, metadata_diff_item_entity_id(item))
             end)
 
           :db_to_file ->

lib/bds/metadata.ex

@@ -1,6 +1,8 @@
 defmodule BDS.Metadata do
   @moduledoc false
 
+  require Logger
+
   alias BDS.Embeddings
   alias BDS.I18n
   alias BDS.Persistence
@@ -653,7 +655,17 @@ defmodule BDS.Metadata do
        ) do
     if previous_state.semantic_similarity_enabled != true and
          project_metadata.semantic_similarity_enabled == true do
-      {:ok, _indexed_post_ids} = Embeddings.index_unindexed(project_id)
+      # Backfill is best-effort: if the embedding model is unavailable, keep the
+      # setting enabled and log it rather than failing the metadata update.
+      case Embeddings.index_unindexed(project_id) do
+        {:ok, _indexed_post_ids} ->
+          :ok
+
+        {:error, reason} ->
+          Logger.warning(
+            "Embedding backfill skipped for project #{project_id}: #{inspect(reason)}"
+          )
+      end
     end
 
     result

priv/data/projects/aceac6d3-8f3f-4a9c-a2b4-517b59b20f44/embeddings.usearch.meta.json

@@ -0,0 +1 @@
+{"dim":384,"labels":[[3399,"e03e9895-d886-4ced-9b01-25ec5a8e1236"],[2430,"8e43a02d-7f96-44a6-91e8-1a9df4c99fe5"],[8304,"0c16060a-a7f0-4c93-9b32-b4f2a64d00cc"],[7206,"64884b0c-31f0-4e5d-9df2-227b29241355"],[5732,"38201eed-8442-4203-8f7c-df64d4b9b1fa"],[3058,"95a4b14b-15b0-49ec-866e-fa87dbf7c77f"],[8341,"9aefc460-4daa-4de8-a126-2c442237076e"],[8646,"76cca26f-d0c0-4e43-962f-f4696076ece0"],[8670,"83bebb91-f500-4a70-b58c-188c6bbe7b2a"],[5951,"20d19d2c-b5db-4409-93b8-433366bf9211"],[631,"e0884bf0-31a8-477f-984f-b9405bd9e2e4"],[9480,"36e06d4d-d298-4587-aecd-cd5870e89be7"],[6543,"c1ca31f5-49c0-4936-8145-cf185763e40f"],[6526,"8eaf0b68-e67d-450d-b95e-678818a9ff4d"],[3628,"c1d1e4d5-ec01-4e5e-a860-f048507ee34d"],[5833,"e49ce0f7-4d2c-4a86-be93-e9e91f703006"],[3650,"b6d34fff-46e9-4835-a5fc-67cc3df78733"],[8226,"38a33a23-1e8f-40eb-8563-e492ae3af255"],[641,"f0705bab-8c1d-4c21-8a79-972b5c98f8c6"],[2704,"1b3e8823-e9ae-429a-9fef-191f3c6445ee"],[6493,"5b295ba6-d517-4792-87c8-8380013b680d"],[1910,"3f67adf1-c025-4f7a-9a26-5de6782e466e"],[1907,"7bfb6c80-c443-4b62-a860-4ddd708d7bfe"],[9531,"68354184-2f58-483d-822b-6b5ff85a4169"],[3769,"359bb060-4ab6-4530-bc9b-eac6923402e1"],[3328,"aa1fdb58-19c4-4e42-bbf1-33262f32c567"],[2335,"33d1ba13-3730-4a80-8fee-0a1ba55e4a06"],[2393,"d3d78308-186e-4db7-a660-b9223aed2983"],[3352,"66e17050-e51a-4e86-8fa4-4bbf9b5c756a"],[5804,"00be944b-07e9-4814-9da3-2af3cbc6ceb7"],[7286,"23a4c649-3169-42a7-b5e7-1010e454f12d"],[8415,"46d13d4b-9208-40b0-965d-ba9c50977245"],[1942,"3c26e4b5-7843-4b46-a1fc-2206fd6924c2"],[5716,"93fe3497-df39-4a52-808e-26ffbbcd87e8"],[7089,"50c4e065-c791-4daf-843b-1efe34831cca"],[8078,"2f820a77-ce07-450c-a2cc-dccad942e7d7"],[3932,"0e1db249-46f7-4b9b-bef2-32ee952d8b1d"],[2826,"cc3e89d1-9935-4cc4-b8b4-7ed538c5785b"],[2525,"530fa48b-b9f6-4744-89cb-baec2ca56d70"],[9782,"7a1654fc-77f7-4bc7-ba83-15287d37be84"],[4213,"1d5fb863-d08b-4636-8bd7-844b5b7515f9"],[9713,"b24fae3e-081c-43f7-88b5-259f6fde21af"],[4350,"7c9cda7d-ec78-4067-ad2b-9c7c94bcb87b"],[5376,"e0f932b8-01a7-469b-a427-84bd04b2ea24"],[6091,"a6821ce5-755f-4340-b950-e1c6bd4e307f"],[5863,"b228ec43-dde8-480f-9909-5c37900b32db"],[5872,"c25e845e-bb6a-4e82-8771-4683068b95d0"],[4998,"a66edb06-801f-41be-a39e-4dc89b5fba39"],[7303,"1e5558b5-8a25-4666-adcf-118ec686a894"],[7003,"d07bb123-6c20-4b99-a246-5368e614244f"],[3536,"98f1c522-1593-41e9-a3c2-5aee3a64d082"],[7666,"a50442bd-6ade-4214-9e29-6e3fca0457b5"],[2433,"5e8b9060-c659-4326-a117-c9ec047aa6a6"],[1076,"c1b203ac-799e-4ad7-a769-350124b76f36"],[9158,"a10c0bc9-300a-420b-99be-21c44a2c897e"],[830,"22b71616-824c-4ccd-a019-747192368151"],[6565,"fe28b579-5cee-4db3-aecf-68e12f84fdfc"],[2423,"e7d930d8-256d-4658-9a40-fe2253347527"],[965,"eca3c34c-5de3-4728-8787-cd03310a5936"],[9769,"be723838-2010-41bd-885a-ff0309cb8913"],[9998,"e4af0229-7ef0-482b-adea-fb11e99e4ba3"],[2457,"2504ec81-65b6-4ce5-b583-082b5890dc49"],[7430,"92d96ad7-880d-4410-b6fc-33525bef6e41"],[9629,"a7589aeb-e3a2-4636-9499-cbeafbfa5698"],[1852,"694462be-5f56-4569-a4df-aa7466c59deb"],[8479,"fba00f0d-3e64-4d44-abf4-b10e5ef92159"],[7389,"d53fd8df-8207-4e47-8adf-8732341fb1b2"],[5060,"5e33c580-9137-492f-b9ac-be7daf61fc17"],[1989,"76906d58-12f0-4ad6-ba3f-d14a521b9d54"],[3160,"4b2ab2c9-62a8-4093-b4eb-632d39a704a5"],[4785,"35f20559-0d5e-4245-b6d4-a916f6da0299"],[6877,"bfad2f5d-0f0c-4592-84f9-c7d67785cc77"],[7813,"349024c7-57c9-4b45-b5bb-93704c0f467c"],[8866,"1c35fbd4-8811-475b-8af4-bf4c3b8d3fdc"],[7590,"4d8af0a4-360e-4b2e-9cc3-13b7e66702cd"],[4509,"011a2575-e992-4f8d-9d9e-870c15518099"],[1286,"93f5e378-b195-4767-b020-99e7481f10df"],[7252,"857b8c44-9218-4e92-8c7e-6743799b1cb4"],[7177,"dfb512cf-be64-4b0d-b516-48e854e8e889"],[5638,"14b6f289-32cb-4d89-b6a9-7dcc3db8b3c3"],[2510,"5612149f-16d5-43f3-91b8-3dbf63e81c54"],[1144,"7f8845fc-21f9-4a7a-85a5-ca7d33bbc181"],[7616,"229b6d4c-e106-41b3-9155-4a56b7e3c9e6"],[1690,"f0a48306-6ec3-419a-b735-6329ebe6f7d0"],[7885,"36433724-1492-4e78-917b-ef477a20e1f2"],[8819,"ce46bc55-5e61-470c-9e2f-1c842725b698"],[2802,"b359ccd8-2af6-4083-9e7e-0579e47e9d5d"],[9821,"f26937a5-9e52-458f-a098-53583573e0d8"],[9192,"9792733d-a646-4ced-bb95-4e0f4fd9749a"],[1015,"c0f2b39b-0a5a-4f87-87e6-019ff3909fa9"],[642,"f3b4c786-aeab-4a43-817d-62d0d9c1829e"],[6453,"dcee70d8-4402-43f9-ba14-3827e87bfe2e"],[467,"d1b97562-8c19-4e2f-9b93-9f7dbb417acb"],[9073,"10f04b8b-a149-4e31-a73d-188430fd41cf"],[1491,"6658e986-ad7e-4362-8be4-a8485b19e2be"],[54,"c7c5c104-28b2-4701-97e9-d77b48687ac9"],[7882,"915dd0d8-1263-45a9-90ed-4ef892ec8a40"],[6575,"7f8fea7f-5572-441d-96fd-50e946702646"],[10169,"fe690ada-ce41-49ff-b248-5c5897d06b7d"],[2347,"faad482f-c812-43f1-9c52-83260476bb77"],[3409,"b8ea67e3-6a01-4996-9e20-bf670d6f9890"],[3333,"0e295d82-1175-462b-a186-899a92572549"],[5798,"8d0e773f-654f-47f2-a255-dd757e2c569b"],[939,"6920c953-3523-4a93-9d65-d50ebebbde72"],[3253,"4c8d81c0-3067-4255-a68b-a0cc6313a98d"],[5033,"2bd9a1b2-7c85-4928-894f-e6de4b328b54"],[9082,"86bc4fda-a887-46d7-b685-21d0fa623c77"],[2083,"b872a025-1e1a-4877-af29-aecb5e650298"],[5686,"9bb903b7-a6Line truncated

test/bds/embeddings_test.exs

@@ -37,6 +37,40 @@ defmodule BDS.EmbeddingsTest do
     end
   end
 
+  defmodule CountingBackend do
+    @behaviour BDS.Embeddings.Backend
+
+    @counter :embeddings_force_counter
+
+    @impl true
+    def model_info, do: %{model_id: "counting/multilingual-e5-small", dimensions: 384}
+
+    @impl true
+    def embed(text, opts) do
+      Agent.update(@counter, &(&1 + 1))
+      BDS.Embeddings.Backends.InApp.embed(text, opts)
+    end
+
+    @impl true
+    def embed_many(texts, opts) do
+      Agent.update(@counter, &(&1 + length(texts)))
+      BDS.Embeddings.Backends.InApp.embed_many(texts, opts)
+    end
+  end
+
+  defmodule FailingBackend do
+    @behaviour BDS.Embeddings.Backend
+
+    @impl true
+    def model_info, do: %{model_id: "failing/multilingual-e5-small", dimensions: 384}
+
+    @impl true
+    def embed(_text, _opts), do: {:error, :model_unavailable}
+
+    @impl true
+    def embed_many(_texts, _opts), do: {:error, :model_unavailable}
+  end
+
   setup do
     :ok = Ecto.Adapters.SQL.Sandbox.checkout(BDS.Repo)
 
@@ -519,4 +553,75 @@ defmodule BDS.EmbeddingsTest do
     assert [%{post_id: post_id}] = similar
     assert post_id == beta.id
   end
+
+  test "explicit rebuild re-embeds every post even when content is unchanged", %{project: project} do
+    assert {:ok, _metadata} =
+             BDS.Metadata.update_project_metadata(project.id, %{semantic_similarity_enabled: true})
+
+    {:ok, _agent} = Agent.start_link(fn -> 0 end, name: :embeddings_force_counter)
+
+    Application.put_env(:bds, :embeddings,
+      backend: CountingBackend,
+      model_id: "counting/multilingual-e5-small",
+      dimensions: 384,
+      batch_size: 16
+    )
+
+    for index <- 1..3 do
+      assert {:ok, post} =
+               BDS.Posts.create_post(%{
+                 project_id: project.id,
+                 title: "Force #{index}",
+                 content: "space rocket orbit mission galaxy #{index}",
+                 language: "en"
+               })
+
+      assert {:ok, _post} = BDS.Posts.publish_post(post.id)
+    end
+
+    # Ignore embeds triggered while creating/publishing.
+    Agent.update(:embeddings_force_counter, fn _count -> 0 end)
+
+    # index_unindexed honours the content_hash skip: nothing to re-embed.
+    assert {:ok, _indexed} = BDS.Embeddings.index_unindexed(project.id)
+    assert Agent.get(:embeddings_force_counter, & &1) == 0
+
+    # An explicit rebuild re-embeds all three regardless (ReindexAll).
+    assert {:ok, rebuilt} = BDS.Embeddings.reindex_all(project.id)
+    assert length(rebuilt) == 3
+    assert Agent.get(:embeddings_force_counter, & &1) == 3
+  end
+
+  test "embedding operations degrade gracefully when the model is unavailable", %{
+    project: project
+  } do
+    assert {:ok, _metadata} =
+             BDS.Metadata.update_project_metadata(project.id, %{semantic_similarity_enabled: true})
+
+    Application.put_env(:bds, :embeddings,
+      backend: FailingBackend,
+      model_id: "failing/multilingual-e5-small",
+      dimensions: 384
+    )
+
+    # Saving a post must not crash even though embedding fails; it is just left
+    # unindexed.
+    assert {:ok, post} =
+             BDS.Posts.create_post(%{
+               project_id: project.id,
+               title: "Offline",
+               content: "space rocket orbit mission galaxy",
+               language: "en"
+             })
+
+    assert {:ok, post} = BDS.Posts.publish_post(post.id)
+    assert BDS.Repo.get_by(BDS.Embeddings.Key, project_id: project.id, post_id: post.id) == nil
+
+    # Explicit (re)index operations surface a clean error instead of crashing.
+    assert {:error, :model_unavailable} = BDS.Embeddings.reindex_all(project.id)
+    assert {:error, :model_unavailable} = BDS.Embeddings.index_unindexed(project.id)
+
+    # Queries stay safe.
+    assert {:ok, []} = BDS.Embeddings.find_similar(post.id, 5)
+  end
 end