cache management for kvcache disc usage #3

New Issue

hugo · 2026-07-25T06:08:42Z

hugo commented

2026-07-25 06:08:42 +00:00

there needs to be an explorer for the kvcache disc usage in the stats panel and there needs to be ways to manage the cache size. the ds4 tooling has LRU mechanisms, we need to be able to set up the parameters for that in preferences and we need to be able to see the disc usage graphically in the stats panel. Numerical this is already covered in stats, but I want a graphic that shows the usage and the age distribution in the usage, maybe with a pie chart.

How ds4 actually manages it (`ds4_kvstore.c`)

It is not an LRU. It is a byte budget with score-based eviction:

One flat directory of sha1-named files. Each header carries model id, quant bits, store reason, tokens, hits, ctx size, created_at, last_used, payload bytes. There is no separate index; the in-memory list is rebuilt by scanning the directory (kv_cache_refresh).
Budget is a byte cap (--kv-disk-space-mb, default 4096 MiB). Eviction runs on open, and again before every store with the estimated new file size passed as extra_bytes, so the store never overshoots the budget. A single checkpoint that cannot fit the budget (plus ~1% slack) is refused outright instead of evicting everything else for it.
Victim is the lowest score, ties broken by oldest last_used:
```
score = (effective_hits + 1) * tokens / file_size
```
- effective_hits = hits * 2^(-age / 6h), clamped to 0 below 0.01. Hit evidence decays with a six hour half-life.
- x2.0 when the reason is an anchor (cold, evict, shutdown) — deliberate anchors outrank routine waypoints.
- x0.05..0.5 when a continued entry is a strict byte prefix of the incoming store and at least as reusable (same model, compatible quant, ctx <= incoming). A superseded waypoint becomes a cheap victim, scaled by its surviving hits.
So the currency is aged reuse density per byte, not recency. A fat checkpoint covering few tokens dies before a small hot one.
Store-side gates gate what ever reaches disc at all: min tokens 512, cold max tokens 30000 (0 disables), continued interval 10000 (0 disables), boundary trim 32, boundary align 2048, reject-different-quant off.
Every eviction and every skip is logged with reason, tokens, hits and size.

What DS4Server already has

src/engine/kvstore.rs is a partial port: same 6h half-life, same anchor x2, same density score, eviction on record with the fresh checkpoint protected. Missing is everything the user can see or set — the budget is a hardcoded 4 GiB DEFAULT_BUDGET_BYTES, there is no supersede-prefix factor, no store-side gates, and nothing in preferences or stats.

Shape of the automatic management

Preferences — new "KV cache" group.
- Disc budget in GiB. The only knob most users touch; wire it to KvStore::budget_bytes in place of the const. Changing it runs eviction immediately.
- Min tokens / cold max tokens / continued interval, ds4 defaults as above. 0 disables where ds4 allows it.
- Reject different quantization (toggle).
- Each with the usual hover description (per #21).
Stats — extend the existing numeric KV CACHE panel into the explorer.
- Usage arc/donut: used vs budget, segmented by age bucket. Buckets on the decay curve, not round human numbers: <1h, 1-6h, 6-24h, 1-7d, older. That way the chart explains why an entry is next to die instead of just how old it is.
- Second breakdown by store reason (cold / continued / agent) so anchor vs waypoint mix is visible.
- Row list under the chart: tokens, size, hits, age, reason; per-entry discard plus one "clear cache" action.
- Reuse the pie widget from #19 rather than growing a second chart implementation.
Keep eviction automatic and silent. On open and before each store, exactly as ds4 does. No background sweeper, no timer, no scheduling UI — the stats panel reports, it does not drive.

Non-goals for this issue: per-project quotas, pinning, manual eviction policy selection. Add only if the single budget knob measurably fails.

Plan placement: this belongs to PLAN.md §1, which already owns KV checkpoint lifecycle (discard payload without transcript, checkpoint invalidation). §3.1 SSD streaming has budget-and-stats work of the same shape but for the in-memory MoE expert cache, which is a different cache.

Two buckets, as in ds4

ds4 keeps the automatic cache and the session store apart on purpose, and we follow that:

ds4-server opens the budgeted ds4_kvstore on --kv-disk-dir: content-addressed, score-evicted, disposable.
ds4_agent.c borrows only the file-format helpers and manages ~/.ds4/kvcache itself — explicit saves, stable SHA1(title||created_at).kv names, a fixed sysprompt.kv, no ds4_kvstore_open, therefore no budget and no eviction. Policy comment at ds4_agent.c:4062.

We already mirror that split: kv-cache/<session_id>.bin is written directly by Engine::generate and never evicted, while kv-cache/http/ is the budgeted KvStore shared by the HTTP endpoint and one-shot app requests.

So the budget, the gates and the eviction score in this issue apply to the transient store only. The session store stays user-owned: it shrinks when a session is deleted, not when a budget is hit. The stats explorer still reports both, since disc usage is disc usage — one chart, segmented by origin (session vs transient), with the budget ring drawn against the transient half.

there needs to be an explorer for the kvcache disc usage in the stats panel and there needs to be ways to manage the cache size. the ds4 tooling has LRU mechanisms, we need to be able to set up the parameters for that in preferences and we need to be able to see the disc usage graphically in the stats panel. Numerical this is already covered in stats, but I want a graphic that shows the usage and the age distribution in the usage, maybe with a pie chart. ## How ds4 actually manages it (`ds4_kvstore.c`) It is not an LRU. It is a byte budget with score-based eviction: - One flat directory of sha1-named files. Each header carries model id, quant bits, store reason, tokens, hits, ctx size, created_at, last_used, payload bytes. There is no separate index; the in-memory list is rebuilt by scanning the directory (`kv_cache_refresh`). - Budget is a byte cap (`--kv-disk-space-mb`, default 4096 MiB). Eviction runs on open, and again before every store with the *estimated* new file size passed as `extra_bytes`, so the store never overshoots the budget. A single checkpoint that cannot fit the budget (plus ~1% slack) is refused outright instead of evicting everything else for it. - Victim is the lowest score, ties broken by oldest `last_used`: score = (effective_hits + 1) * tokens / file_size - `effective_hits = hits * 2^(-age / 6h)`, clamped to 0 below 0.01. Hit evidence decays with a six hour half-life. - x2.0 when the reason is an anchor (cold, evict, shutdown) — deliberate anchors outrank routine waypoints. - x0.05..0.5 when a *continued* entry is a strict byte prefix of the incoming store and at least as reusable (same model, compatible quant, ctx <= incoming). A superseded waypoint becomes a cheap victim, scaled by its surviving hits. So the currency is aged reuse density per byte, not recency. A fat checkpoint covering few tokens dies before a small hot one. - Store-side gates gate what ever reaches disc at all: min tokens 512, cold max tokens 30000 (0 disables), continued interval 10000 (0 disables), boundary trim 32, boundary align 2048, reject-different-quant off. - Every eviction and every skip is logged with reason, tokens, hits and size. ## What DS4Server already has `src/engine/kvstore.rs` is a partial port: same 6h half-life, same anchor x2, same density score, eviction on record with the fresh checkpoint protected. Missing is everything the user can see or set — the budget is a hardcoded 4 GiB `DEFAULT_BUDGET_BYTES`, there is no supersede-prefix factor, no store-side gates, and nothing in preferences or stats. ## Shape of the automatic management 1. **Preferences — new "KV cache" group.** - Disc budget in GiB. The only knob most users touch; wire it to `KvStore::budget_bytes` in place of the const. Changing it runs eviction immediately. - Min tokens / cold max tokens / continued interval, ds4 defaults as above. 0 disables where ds4 allows it. - Reject different quantization (toggle). - Each with the usual hover description (per #21). 2. **Stats — extend the existing numeric KV CACHE panel into the explorer.** - Usage arc/donut: used vs budget, segmented by age bucket. Buckets on the decay curve, not round human numbers: <1h, 1-6h, 6-24h, 1-7d, older. That way the chart explains *why* an entry is next to die instead of just how old it is. - Second breakdown by store reason (cold / continued / agent) so anchor vs waypoint mix is visible. - Row list under the chart: tokens, size, hits, age, reason; per-entry discard plus one "clear cache" action. - Reuse the pie widget from #19 rather than growing a second chart implementation. 3. **Keep eviction automatic and silent.** On open and before each store, exactly as ds4 does. No background sweeper, no timer, no scheduling UI — the stats panel reports, it does not drive. Non-goals for this issue: per-project quotas, pinning, manual eviction policy selection. Add only if the single budget knob measurably fails. Plan placement: this belongs to PLAN.md §1, which already owns KV checkpoint lifecycle (discard payload without transcript, checkpoint invalidation). §3.1 SSD streaming has budget-and-stats work of the same shape but for the in-memory MoE expert cache, which is a different cache. ## Two buckets, as in ds4 ds4 keeps the automatic cache and the session store apart on purpose, and we follow that: - ds4-server opens the budgeted `ds4_kvstore` on `--kv-disk-dir`: content-addressed, score-evicted, disposable. - `ds4_agent.c` borrows only the file-format helpers and manages `~/.ds4/kvcache` itself — explicit saves, stable `SHA1(title||created_at).kv` names, a fixed `sysprompt.kv`, no `ds4_kvstore_open`, therefore no budget and no eviction. Policy comment at `ds4_agent.c:4062`. We already mirror that split: `kv-cache/<session_id>.bin` is written directly by `Engine::generate` and never evicted, while `kv-cache/http/` is the budgeted `KvStore` shared by the HTTP endpoint and one-shot app requests. So the budget, the gates and the eviction score in this issue apply to the transient store only. The session store stays user-owned: it shrinks when a session is deleted, not when a budget is hit. The stats explorer still reports both, since disc usage is disc usage — one chart, segmented by origin (session vs transient), with the budget ring drawn against the transient half.

hugo added the enhancement label 2026-07-25 06:08:42 +00:00

hugo commented

2026-07-26 07:10:49 +00:00

Implemented in 63e2a74.

Preferences — new "KV CACHE" group. Disc budget in GiB, minimum tokens, cold maximum, continued interval, each with hover text and the DS4 default when left blank (4 GiB / 512 / 30000 / 10000). Persisted in four nullable columns; the round trip is covered by the database test. The settings live in TurnSettings, deliberately not in EngineSettings, so changing a cache knob never reloads the model.

Store. KvStore takes its budget from preferences instead of a hardcoded 4 GiB and evicts on open, so a lowered budget applies rather than waiting for the next store to push it over. The ds4 store gates now decide whether a checkpoint is written at all, with last_store_tokens spacing continued checkpoints like ds4's continued_last_store_tokens. A gated store still marks the live KV with the finished conversation's tag, otherwise the next turn would either re-prefill needlessly or continue from a stale prefix.

Stats — "KV CACHE DISC USAGE". Age-segmented usage bar with a legend, on the decay curve rather than round numbers: under 1h, 1–6h, 6–24h, 1–7d, older. Session and transient totals are reported apart, with the budget shown only against the transient bucket, since only that one is evicted. Below it the twelve oldest checkpoints, oldest first, each with a discard action, plus one "Clear transient cache". Rescans every two seconds while the panel is open, and the counters are re-derived after any deletion so they cannot drift from the disc.

Orphan sweep. Deleting a session or a project already removed the checkpoint, but that was the only cleanup and both paths swallow errors, so an interrupted or failed delete left a file that nothing would ever reclaim — session checkpoints are outside the budget by design. On startup and after every project reload, numbered checkpoints whose session no longer exists are now deleted. Conservatively scoped: only stems that parse as a session id are judged, and only on a successfully loaded project list, so a database read error cannot wipe every checkpoint.

Deliberately not done:

Pie chart. A stacked bar needs no iced canvas feature and matches the existing mini_chart idiom. #19 wants a real pie for the context fill; if that lands, this panel can share it.
Tokens, hits and reason columns in the rows. They exist only for transient entries and would mean exporting the index parser to the UI. The rows are filesystem-derived: name, size, age.
--kv-cache-boundary-trim-tokens / --kv-cache-boundary-align-tokens. They serve ds4's token-prefix lookup; our store keys on the rendered conversation.
--kv-cache-reject-different-quant. Checkpoints already bind the model file identity, so a checkpoint from a different quantization can never load. All four are recorded with their reason in the CLI inventory test.
Eviction at the moment Save is pressed. It happens when the store next opens, on the following request, rather than from the UI thread while the runtime may be mid-write.

Gates green: cargo fmt --all -- --check, cargo clippy --all-targets --all-features -- -D warnings, make bundle, 65 tests. Not verified visually yet — the panel has not been looked at in a running app.

Implemented in 63e2a74. **Preferences — new "KV CACHE" group.** Disc budget in GiB, minimum tokens, cold maximum, continued interval, each with hover text and the DS4 default when left blank (4 GiB / 512 / 30000 / 10000). Persisted in four nullable columns; the round trip is covered by the database test. The settings live in `TurnSettings`, deliberately not in `EngineSettings`, so changing a cache knob never reloads the model. **Store.** `KvStore` takes its budget from preferences instead of a hardcoded 4 GiB and evicts on open, so a lowered budget applies rather than waiting for the next store to push it over. The ds4 store gates now decide whether a checkpoint is written at all, with `last_store_tokens` spacing continued checkpoints like ds4's `continued_last_store_tokens`. A gated store still marks the live KV with the finished conversation's tag, otherwise the next turn would either re-prefill needlessly or continue from a stale prefix. **Stats — "KV CACHE DISC USAGE".** Age-segmented usage bar with a legend, on the decay curve rather than round numbers: under 1h, 1–6h, 6–24h, 1–7d, older. Session and transient totals are reported apart, with the budget shown only against the transient bucket, since only that one is evicted. Below it the twelve oldest checkpoints, oldest first, each with a discard action, plus one "Clear transient cache". Rescans every two seconds while the panel is open, and the counters are re-derived after any deletion so they cannot drift from the disc. **Orphan sweep.** Deleting a session or a project already removed the checkpoint, but that was the only cleanup and both paths swallow errors, so an interrupted or failed delete left a file that nothing would ever reclaim — session checkpoints are outside the budget by design. On startup and after every project reload, numbered checkpoints whose session no longer exists are now deleted. Conservatively scoped: only stems that parse as a session id are judged, and only on a successfully loaded project list, so a database read error cannot wipe every checkpoint. Deliberately not done: - **Pie chart.** A stacked bar needs no `iced` canvas feature and matches the existing `mini_chart` idiom. #19 wants a real pie for the context fill; if that lands, this panel can share it. - **Tokens, hits and reason columns** in the rows. They exist only for transient entries and would mean exporting the index parser to the UI. The rows are filesystem-derived: name, size, age. - **`--kv-cache-boundary-trim-tokens` / `--kv-cache-boundary-align-tokens`.** They serve ds4's token-prefix lookup; our store keys on the rendered conversation. - **`--kv-cache-reject-different-quant`.** Checkpoints already bind the model file identity, so a checkpoint from a different quantization can never load. All four are recorded with their reason in the CLI inventory test. - **Eviction at the moment Save is pressed.** It happens when the store next opens, on the following request, rather than from the UI thread while the runtime may be mid-write. Gates green: `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features -- -D warnings`, `make bundle`, 65 tests. Not verified visually yet — the panel has not been looked at in a running app.

hugo closed this issue

2026-07-26 07:10:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: hugo/DS4Server#3

cache management for kvcache disc usage #3

How ds4 actually manages it (ds4_kvstore.c)

What DS4Server already has

Shape of the automatic management

Two buckets, as in ds4

How ds4 actually manages it (`ds4_kvstore.c`)