fix: added things around project folder pollution from program runs

.gitignore

@@ -10,4 +10,7 @@
 /priv/data/*.db
 /priv/data/*.db-shm
 /priv/data/*.db-wal
+# Embeddings index artifacts are per-project runtime caches, never committed.
+*.usearch
+*.usearch.meta.json
 *.eztmp/

SPECGAPS.md

@@ -27,6 +27,7 @@ Gap categories: **SC** = spec correct, fix code | **CS** = code correct, update
 | A1-14b | ~~USearch HNSW ANN index + debounced persistence not implemented~~ | embedding.allium config/FindSimilar/DebouncedPersistence | `Embeddings.Index` is now an HNSW (hnswlib) ANN index with debounced persistence | **Resolved:** rewrote `Embeddings.Index` as a DB-free GenServer wrapping an hnswlib HNSW graph (cosine, M=16, efConstruction=128, efSearch=64) — O(n·log n) build, O(log n) queries, replacing the O(n²) JSON cosine snapshot; per-project in-memory index + `label→post_id` map; 5s debounced `save_index` + `.meta.json` sidecar, force-save on project switch (`set_active_project`) and shutdown (`terminate`), `forget/1` on project delete; lazy reload from disk with rebuild-from-DB self-heal on miss; `find_similar`/`find_duplicates`/`compute_similarities` rewired (no brute-force fallback); USearch has no Elixir binding so hnswlib provides the identical HNSW algorithm/params (spec reconciled); supervision + dialyzer PLT updated; tests updated for debounced/binary persistence + self-heal. Follow-up hardening: explicit rebuild now forces re-embedding regardless of content_hash (ReindexAll), and model-unavailable errors propagate cleanly (post saves degrade to unindexed + log; rebuild/index return `{:error, reason}` surfaced as a failed task with a user-facing message instead of crashing). |
 | A1-14c | ~~Embedding model runs on CPU only; no Apple GPU acceleration~~ | embedding.allium invariant NativeAcceleratedExecution | `Backends.Neural` now selects the defn compiler at serving-build time: Apple GPU via EMLX (MLX/Metal) on arm64 macOS, EXLA-CPU elsewhere | **Resolved:** added `{:emlx, "~> 0.2.0"}` dep (ships precompiled MLX binaries; EMLX 0.2.0 implements both `EMLX.Backend` and the `Nx.Defn.Compiler` behaviour, GPU-default); `Backends.Neural` gained a pure `select_accelerator/3` policy (`:auto` prefers EMLX only when available **and** on Apple Silicon; explicit `:emlx`/`:exla` honoured; forced `:emlx` degrades to EXLA when unavailable so misconfigured hosts still run), `current_accelerator/0`, and `defn_options/1`; `build_serving` places params on `{EMLX.Backend, device: :gpu}` and compiles with `EMLX` for the EMLX path, keeps `EXLA` otherwise; new `accelerator: :auto` config key; spec `NativeAcceleratedExecution` + `EmbeddingModel` updated; PLT app added; 7 tests added (offline — test config still uses the InApp stub). |
 | A1-15 | ~~Preview vs generation content source strategy undocumented~~ | preview.allium (no invariant), generation.allium (no invariant) | Generation uses only published .md file content (`Generation.Data` snapshots set `content: nil`); preview includes published+draft posts and prefers DB content over file (`Preview.Router` queries `:published`/`:draft`, uses `editor_body`) | **Resolved:** added `PreviewDraftOverlay` invariant to preview.allium and `GenerationPublishedOnly` invariant to generation.allium; both cross-reference each other; code already correct, 3 tests added for draft-in-preview behavior |
+| A1-16 | Public project content + data_path discovery not compliant with storage-location spec | project.allium `PublicContentLivesInProjectFolder` / `PrivateArtifactsLiveInOsAppDir` / `DataPathNotPersistedInProjectJson` / `DiscoverProjectDataPath` (newly added) | **Private side done:** `Projects.project_cache_root/0` now falls back to the OS private app dir (`:filename.basedir(:user_config, "bds")` → macOS `~/Library/Application Support/bds`) instead of `priv/data`, so the embeddings index no longer lands in the repo. **Still non-compliant (public side):** `project_data_dir/0` (projects.ex:97-99) falls back to `priv/data/projects/<id>` when `data_path` is nil, so the default project's *public* content (posts, media, templates, scripts, `meta/`, generated `html/`) is written into the application repo; there is no discovery of `data_path` from the `meta/project.json` location, and the `default` project is created with `data_path: nil` (projects.ex:80). | Implement project-folder discovery: `data_path` := the folder containing `meta/project.json` (never stored in project.json, keeping projects movable — `DiscoverProjectDataPath`); create the default project's folder at a per-user default content location on first launch (never in repo/private_dir); drop the `priv/data/projects/<id>` fallback in `project_data_dir/0`; persist the current project-folder location as a machine-local pointer (project registry) under `private_dir`. Migrate the committed `priv/data/projects/default/` content out of the repo. |
 
 ### A2. Spec Should Update (code is normative)
 
@@ -186,7 +187,8 @@ All reconciled to follow code. Specs must be self-consistent and match code.
 
 ## Priority Order for Resolution
 
-1. ~~**A1-1 through A1-14c**~~ — all resolved: auto-save, on-demand preview, template lookup, validation gates, real Pagefind, graceful shutdown, real embedding model, HNSW ANN index, and Apple GPU/EMLX acceleration (A1-14c)
+1. ~~**A1-1 through A1-15**~~ — all resolved: auto-save, on-demand preview, template lookup, validation gates, real Pagefind, graceful shutdown, real embedding model, HNSW ANN index, Apple GPU/EMLX acceleration (A1-14c), and preview/generation content strategy (A1-15)
+1b. **A1-16** — storage-location compliance: private side done (embeddings index → OS app dir); public side open (data_path discovery from meta/project.json, drop the `priv/data/projects/<id>` fallback, migrate committed default project out of repo)
 2. **D1-1 through D1-18** — untested invariants/guarantees
 3. **C-1 through C-3** — internal spec inconsistencies (reconcile to code)
 4. **B1-1 through B1-6** — major code behaviors missing from spec

lib/bds/projects.ex

@@ -268,20 +268,25 @@ defmodule BDS.Projects do
     not Repo.exists?(from project in Project, where: project.slug == ^slug)
   end
 
-  defp repo_data_dir do
-    Application.fetch_env!(:bds, BDS.Repo)
-    |> Keyword.fetch!(:database)
-    |> Path.expand()
-    |> Path.dirname()
-  end
-
   defp project_cache_root do
     case Application.get_env(:bds, :project_cache_root) do
       root when is_binary(root) -> Path.expand(root)
-      _other -> repo_data_dir()
+      # Private app-internal artifacts (e.g. the embeddings index) live under the
+      # OS private app directory — on macOS ~/Library/Application Support/bds —
+      # never inside the repo or a project's public folder. Colocating them with
+      # project_data_dir would pollute (and historically committed to) the repo.
+      _other -> private_app_dir()
     end
   end
 
+  defp private_app_dir do
+    case :filename.basedir(:user_config, "bds") do
+      path when is_list(path) -> List.to_string(path)
+      path -> path
+    end
+    |> Path.expand()
+  end
+
   defp attr(attrs, key) do
     cond do
       Map.has_key?(attrs, key) -> Map.get(attrs, key)

priv/data/projects/aceac6d3-8f3f-4a9c-a2b4-517b59b20f44/embeddings.usearch.meta.json

@@ -1 +0,0 @@
-{"dim":384,"labels":[[3399,"e03e9895-d886-4ced-9b01-25ec5a8e1236"],[2430,"8e43a02d-7f96-44a6-91e8-1a9df4c99fe5"],[8304,"0c16060a-a7f0-4c93-9b32-b4f2a64d00cc"],[7206,"64884b0c-31f0-4e5d-9df2-227b29241355"],[5732,"38201eed-8442-4203-8f7c-df64d4b9b1fa"],[3058,"95a4b14b-15b0-49ec-866e-fa87dbf7c77f"],[8341,"9aefc460-4daa-4de8-a126-2c442237076e"],[8646,"76cca26f-d0c0-4e43-962f-f4696076ece0"],[8670,"83bebb91-f500-4a70-b58c-188c6bbe7b2a"],[5951,"20d19d2c-b5db-4409-93b8-433366bf9211"],[631,"e0884bf0-31a8-477f-984f-b9405bd9e2e4"],[9480,"36e06d4d-d298-4587-aecd-cd5870e89be7"],[6543,"c1ca31f5-49c0-4936-8145-cf185763e40f"],[6526,"8eaf0b68-e67d-450d-b95e-678818a9ff4d"],[3628,"c1d1e4d5-ec01-4e5e-a860-f048507ee34d"],[5833,"e49ce0f7-4d2c-4a86-be93-e9e91f703006"],[3650,"b6d34fff-46e9-4835-a5fc-67cc3df78733"],[8226,"38a33a23-1e8f-40eb-8563-e492ae3af255"],[641,"f0705bab-8c1d-4c21-8a79-972b5c98f8c6"],[2704,"1b3e8823-e9ae-429a-9fef-191f3c6445ee"],[6493,"5b295ba6-d517-4792-87c8-8380013b680d"],[1910,"3f67adf1-c025-4f7a-9a26-5de6782e466e"],[1907,"7bfb6c80-c443-4b62-a860-4ddd708d7bfe"],[9531,"68354184-2f58-483d-822b-6b5ff85a4169"],[3769,"359bb060-4ab6-4530-bc9b-eac6923402e1"],[3328,"aa1fdb58-19c4-4e42-bbf1-33262f32c567"],[2335,"33d1ba13-3730-4a80-8fee-0a1ba55e4a06"],[2393,"d3d78308-186e-4db7-a660-b9223aed2983"],[3352,"66e17050-e51a-4e86-8fa4-4bbf9b5c756a"],[5804,"00be944b-07e9-4814-9da3-2af3cbc6ceb7"],[7286,"23a4c649-3169-42a7-b5e7-1010e454f12d"],[8415,"46d13d4b-9208-40b0-965d-ba9c50977245"],[1942,"3c26e4b5-7843-4b46-a1fc-2206fd6924c2"],[5716,"93fe3497-df39-4a52-808e-26ffbbcd87e8"],[7089,"50c4e065-c791-4daf-843b-1efe34831cca"],[8078,"2f820a77-ce07-450c-a2cc-dccad942e7d7"],[3932,"0e1db249-46f7-4b9b-bef2-32ee952d8b1d"],[2826,"cc3e89d1-9935-4cc4-b8b4-7ed538c5785b"],[2525,"530fa48b-b9f6-4744-89cb-baec2ca56d70"],[9782,"7a1654fc-77f7-4bc7-ba83-15287d37be84"],[4213,"1d5fb863-d08b-4636-8bd7-844b5b7515f9"],[9713,"b24fae3e-081c-43f7-88b5-259f6fde21af"],[4350,"7c9cda7d-ec78-4067-ad2b-9c7c94bcb87b"],[5376,"e0f932b8-01a7-469b-a427-84bd04b2ea24"],[6091,"a6821ce5-755f-4340-b950-e1c6bd4e307f"],[5863,"b228ec43-dde8-480f-9909-5c37900b32db"],[5872,"c25e845e-bb6a-4e82-8771-4683068b95d0"],[4998,"a66edb06-801f-41be-a39e-4dc89b5fba39"],[7303,"1e5558b5-8a25-4666-adcf-118ec686a894"],[7003,"d07bb123-6c20-4b99-a246-5368e614244f"],[3536,"98f1c522-1593-41e9-a3c2-5aee3a64d082"],[7666,"a50442bd-6ade-4214-9e29-6e3fca0457b5"],[2433,"5e8b9060-c659-4326-a117-c9ec047aa6a6"],[1076,"c1b203ac-799e-4ad7-a769-350124b76f36"],[9158,"a10c0bc9-300a-420b-99be-21c44a2c897e"],[830,"22b71616-824c-4ccd-a019-747192368151"],[6565,"fe28b579-5cee-4db3-aecf-68e12f84fdfc"],[2423,"e7d930d8-256d-4658-9a40-fe2253347527"],[965,"eca3c34c-5de3-4728-8787-cd03310a5936"],[9769,"be723838-2010-41bd-885a-ff0309cb8913"],[9998,"e4af0229-7ef0-482b-adea-fb11e99e4ba3"],[2457,"2504ec81-65b6-4ce5-b583-082b5890dc49"],[7430,"92d96ad7-880d-4410-b6fc-33525bef6e41"],[9629,"a7589aeb-e3a2-4636-9499-cbeafbfa5698"],[1852,"694462be-5f56-4569-a4df-aa7466c59deb"],[8479,"fba00f0d-3e64-4d44-abf4-b10e5ef92159"],[7389,"d53fd8df-8207-4e47-8adf-8732341fb1b2"],[5060,"5e33c580-9137-492f-b9ac-be7daf61fc17"],[1989,"76906d58-12f0-4ad6-ba3f-d14a521b9d54"],[3160,"4b2ab2c9-62a8-4093-b4eb-632d39a704a5"],[4785,"35f20559-0d5e-4245-b6d4-a916f6da0299"],[6877,"bfad2f5d-0f0c-4592-84f9-c7d67785cc77"],[7813,"349024c7-57c9-4b45-b5bb-93704c0f467c"],[8866,"1c35fbd4-8811-475b-8af4-bf4c3b8d3fdc"],[7590,"4d8af0a4-360e-4b2e-9cc3-13b7e66702cd"],[4509,"011a2575-e992-4f8d-9d9e-870c15518099"],[1286,"93f5e378-b195-4767-b020-99e7481f10df"],[7252,"857b8c44-9218-4e92-8c7e-6743799b1cb4"],[7177,"dfb512cf-be64-4b0d-b516-48e854e8e889"],[5638,"14b6f289-32cb-4d89-b6a9-7dcc3db8b3c3"],[2510,"5612149f-16d5-43f3-91b8-3dbf63e81c54"],[1144,"7f8845fc-21f9-4a7a-85a5-ca7d33bbc181"],[7616,"229b6d4c-e106-41b3-9155-4a56b7e3c9e6"],[1690,"f0a48306-6ec3-419a-b735-6329ebe6f7d0"],[7885,"36433724-1492-4e78-917b-ef477a20e1f2"],[8819,"ce46bc55-5e61-470c-9e2f-1c842725b698"],[2802,"b359ccd8-2af6-4083-9e7e-0579e47e9d5d"],[9821,"f26937a5-9e52-458f-a098-53583573e0d8"],[9192,"9792733d-a646-4ced-bb95-4e0f4fd9749a"],[1015,"c0f2b39b-0a5a-4f87-87e6-019ff3909fa9"],[642,"f3b4c786-aeab-4a43-817d-62d0d9c1829e"],[6453,"dcee70d8-4402-43f9-ba14-3827e87bfe2e"],[467,"d1b97562-8c19-4e2f-9b93-9f7dbb417acb"],[9073,"10f04b8b-a149-4e31-a73d-188430fd41cf"],[1491,"6658e986-ad7e-4362-8be4-a8485b19e2be"],[54,"c7c5c104-28b2-4701-97e9-d77b48687ac9"],[7882,"915dd0d8-1263-45a9-90ed-4ef892ec8a40"],[6575,"7f8fea7f-5572-441d-96fd-50e946702646"],[10169,"fe690ada-ce41-49ff-b248-5c5897d06b7d"],[2347,"faad482f-c812-43f1-9c52-83260476bb77"],[3409,"b8ea67e3-6a01-4996-9e20-bf670d6f9890"],[3333,"0e295d82-1175-462b-a186-899a92572549"],[5798,"8d0e773f-654f-47f2-a255-dd757e2c569b"],[939,"6920c953-3523-4a93-9d65-d50ebebbde72"],[3253,"4c8d81c0-3067-4255-a68b-a0cc6313a98d"],[5033,"2bd9a1b2-7c85-4928-894f-e6de4b328b54"],[9082,"86bc4fda-a887-46d7-b685-21d0fa623c77"],[2083,"b872a025-1e1a-4877-af29-aecb5e650298"],[5686,"9bb903b7-a6Line truncated

specs/media.allium

@@ -186,7 +186,7 @@ rule UpsertMediaTranslation {
 rule RebuildMediaFromFiles {
     when: RebuildMediaFromFilesRequested(project)
     -- Scans media directory for .meta sidecars, reimports to DB
-    for sidecar in scan_directory(project.effective_data_dir + "/media", "*.meta"):
+    for sidecar in scan_directory(project.public_dir + "/media", "*.meta"):
         let parsed = parse_sidecar(sidecar)
         ensures: Media.created(parsed)
             -- or updated if already exists

specs/metadata_diff.allium

@@ -61,7 +61,7 @@ rule RunMetadataDiff {
                 )
 
     -- Detect orphan files (on disk but not in DB)
-    for file in scan_directory(project.effective_data_dir + "/posts", "*.md"):
+    for file in scan_directory(project.public_dir + "/posts", "*.md"):
         let matching = Posts where file_path = file
         if matching.count = 0:
             ensures: OrphanReport.created(file_path: file)

specs/project.allium

@@ -8,6 +8,7 @@ surface ProjectControlSurface {
 
     provides:
         CreateProjectRequested(name, data_path)
+        OpenProjectRequested(folder_path)
         SetActiveProjectRequested(project)
         DeleteProjectRequested(project)
 }
@@ -27,11 +28,33 @@ entity Project {
     tags: Tag with project = this
 
     -- Derived
-    internal_base_dir: String
+    --
-        -- {user_data}/projects/{id}/
+    -- data_path is the project folder: the directory that CONTAINS
-        -- Contains: meta/, thumbnails/, tags.json
+    -- meta/project.json. It is DISCOVERED from the project.json location at
-    effective_data_dir: data_path ?? internal_base_dir
+    -- load time and is never written into project.json itself — so the folder
-        -- Custom data path overrides default
+    -- can be moved/renamed freely. The current location is remembered only as a
+    -- machine-local pointer (a project registry under private_dir), never
+    -- embedded in the portable project. See DataPathNotPersistedInProjectJson.
+    public_dir: data_path
+        -- All user-owned, portable, webserver-bound content lives here, under
+        -- the project folder:
+        --   posts (.md), media + thumbnails, templates/, scripts/,
+        --   meta/ (project.json, categories.json, category-meta.json,
+        --   publishing.json), tags.json, menu.opml, and generated html/ output.
+        -- See PublicContentLivesInProjectFolder.
+    private_dir: String
+        -- The OS per-user app-data directory. Holds app-internal,
+        -- machine-specific, regenerable artifacts ONLY — never inside the repo
+        -- or the project folder:
+        --   the SQLite database, the per-project embeddings index
+        --   (projects/{id}/embeddings.usearch + .meta.json sidecar), the
+        --   downloaded embedding-model cache, the project registry, and
+        --   window/UI state.
+        -- OS-specific base (resolved via :filename.basedir, app name "bds"):
+        --   macOS:   ~/Library/Application Support/bds   (:user_config)
+        --   Linux:   $XDG_CONFIG_HOME/bds  (default ~/.config/bds)
+        --   Windows: %APPDATA%\\bds
+        -- See PrivateArtifactsLiveInOsAppDir.
 }
 
 surface ProjectSurface {
@@ -48,8 +71,8 @@ surface ProjectSurface {
         project.posts.count
         project.media.count
         project.tags.count
-        project.internal_base_dir
+        project.public_dir
-        project.effective_data_dir
+        project.private_dir
 }
 
 invariant SingleActiveProject {
@@ -73,11 +96,52 @@ rule CreateProject {
         data_path: data_path,
         is_active: false
     )
+    @guidance
+        -- data_path is the chosen project folder. CreateProject writes
+        -- meta/project.json into {data_path}/meta/ but never records data_path
+        -- inside it (DataPathNotPersistedInProjectJson). The default project's
+        -- folder is created at a per-user default content location on first
+        -- launch — never inside the application repo or private_dir.
+}
+
+rule DiscoverProjectDataPath {
+    -- Opening or loading a project folder deduces its data_path from the
+    -- on-disk location of meta/project.json rather than from any stored value,
+    -- keeping projects movable (DataPathNotPersistedInProjectJson).
+    when: OpenProjectRequested(folder_path)
+        -- folder_path contains meta/project.json
+    let project = Projects where data_path = folder_path
+    ensures: project.data_path = folder_path
 }
 
 invariant ProjectTemplatesDirectoryReservedForUserTemplates {
     -- The project templates directory stores only user-managed templates.
-    -- Creating a project does not populate effective_data_dir/templates with bundled defaults.
+    -- Creating a project does not populate public_dir/templates with bundled defaults.
+}
+
+invariant PublicContentLivesInProjectFolder {
+    -- Every file the user owns or that must be served by the webserver lives
+    -- under public_dir (= the project folder containing meta/project.json):
+    -- posts, media + thumbnails, templates, scripts, the meta/ JSON files,
+    -- tags.json, menu.opml, and the generated html/ output. None of this
+    -- content is ever written to private_dir or into the application repo.
+}
+
+invariant PrivateArtifactsLiveInOsAppDir {
+    -- App-internal, machine-specific, regenerable artifacts — the SQLite
+    -- database, the per-project embeddings index and its sidecar, the
+    -- model cache, the project registry, and window/UI state — live ONLY
+    -- under private_dir (the OS per-user app-data directory). They are never
+    -- written into a project folder (public_dir) or into the application repo.
+    -- A regenerable artifact (e.g. the embeddings index) may be rebuilt from
+    -- the database when absent.
+}
+
+invariant DataPathNotPersistedInProjectJson {
+    -- meta/project.json never stores its own path or data_path. A project's
+    -- location is determined solely by where its meta/project.json file sits,
+    -- so the folder can be moved or renamed without editing any file. The app
+    -- remembers the current location as a machine-local pointer in private_dir.
 }
 
 rule SetActiveProject {

specs/script.allium

@@ -268,7 +268,7 @@ rule ExecuteTransform {
 
 rule RebuildScriptsFromFiles {
     when: RebuildScriptsFromFilesRequested(project)
-    for file in scan_directory(project.effective_data_dir + "/scripts", "*." + config.script_extension):
+    for file in scan_directory(project.public_dir + "/scripts", "*." + config.script_extension):
         let parsed = parse_script_file(file)
         ensures: Script.created(parsed)
 }

specs/template.allium

@@ -81,7 +81,7 @@ invariant UserTemplateDirectoryOverridesBundledDefaults {
 }
 
 invariant RebuildTemplatesIndexesOnlyProjectTemplates {
-    -- Rebuild-from-files scans only project.effective_data_dir/templates.
+    -- Rebuild-from-files scans only project.public_dir/templates.
     -- Bundled defaults are render-time fallbacks and are not indexed into Templates
     -- unless the user has created matching project files.
 }
@@ -171,7 +171,7 @@ rule CascadeSlugUpdate {
 
 rule RebuildTemplatesFromFiles {
     when: RebuildTemplatesFromFilesRequested(project)
-    for file in scan_directory(project.effective_data_dir + "/templates", "*.liquid"):
+    for file in scan_directory(project.public_dir + "/templates", "*.liquid"):
         let parsed = parse_template_file(file)
         ensures: Template.created(parsed)
             -- or updated if slug already exists

test/bds/projects_test.exs

@@ -117,6 +117,32 @@ defmodule BDS.ProjectsTest do
     assert Enum.count(BDS.Projects.list_projects(), & &1.is_active) == 1
   end
 
+  test "project_cache_dir never falls back into the project data directory" do
+    # Private app-internal artifacts (the embeddings index) must live under the
+    # OS private app directory (macOS: ~/Library/Application Support/bds), never
+    # inside priv/data/projects/<id> — leaving them in the project tree pollutes
+    # the repository.
+    saved = Application.get_env(:bds, :project_cache_root)
+    Application.delete_env(:bds, :project_cache_root)
+    on_exit(fn -> Application.put_env(:bds, :project_cache_root, saved) end)
+
+    project_id = "fallback-#{System.unique_integer([:positive])}"
+    cache_dir = BDS.Projects.project_cache_dir(project_id)
+
+    data_dir = Path.expand("../../priv/data/projects/#{project_id}", __DIR__)
+    refute cache_dir == data_dir
+    refute String.starts_with?(cache_dir, Path.expand("../../priv/data", __DIR__))
+
+    private_app_dir =
+      case :filename.basedir(:user_config, "bds") do
+        path when is_list(path) -> List.to_string(path)
+        path -> path
+      end
+      |> Path.expand()
+
+    assert String.starts_with?(cache_dir, private_app_dir)
+  end
+
   test "ensure_default_project creates the default project once and keeps it active" do
     Repo.delete_all(Project)