Finish DS4 execution parity

This commit is contained in:
Georg Bauer
2026-07-26 20:44:23 +02:00
parent fd3f8e45dc
commit 0d80c217c4
15 changed files with 16905 additions and 195 deletions

103
PLAN.md
View File

@@ -46,32 +46,29 @@ execution targets one self-contained Mac.
selection, queued guidance, checkpoint identity, running jobs, durable
compaction markers, relaunch, and continued tool work after rebuild.
- DeepSeek V4 Flash now supports DS4-compatible SSD expert streaming, legacy
MTP, DSpark, and directional steering in the Rust executor. The optional
modes are integrated into the same target-owned generation path used by
local chat and the endpoint; disabling them preserves the resident greedy
token baseline. Runtime counters feed the Stats dashboard without inference
thread UI work.
- Hardware-backed token oracles cover resident versus SSD execution, legacy
MTP, DSpark, SSD combined with both speculative modes, directional steering,
and target-only fallback. Differential endpoint scripts cover deterministic
output, finish state, and usage when reference and Rust servers are supplied.
- GLM 5.2 has a dedicated Rust/Metal executor and DeepSeek V4 Pro uses the
generalized DeepSeek graph. Their remaining work is validation rather than
catalog plumbing: GLM MTP is still rejected, the full GLM/Pro hardware matrix
is incomplete, and Pro still needs explicit memory-admission verification.
- The remaining model-independent execution gaps are fine-grained SSD cache
telemetry, the DS4 expert-locality profiler, and resident multi-session
server batching/scheduling.
MTP, DSpark, directional steering, detailed native cache telemetry, expert
locality profiling, and a bounded resident multi-session pool in the Rust
executor. Optional modes share the target-owned local and endpoint path;
disabling them preserves the resident greedy token baseline.
- Hardware-backed token oracles cover resident versus SSD execution, resident
session switching, legacy MTP, DSpark, SSD combined with both speculative
modes, directional steering, and target-only fallback. Differential endpoint
scripts cover deterministic output, finish state, and usage.
- GLM 5.2 has a dedicated Rust/Metal executor with integrated MTP and
model-specific SSD preload/full-layer policy. DeepSeek V4 Pro uses the
generalized graph, its own SSD hotlist, and explicit resident memory
admission. Fixture-aware GLM/Pro hardware tests complete the model matrix.
- The native UI is on Iced 0.14. Chat transcripts use its table-aware Markdown
content and viewer path, with a regression for code-styled line-count tables
produced by coding models.
## Delivery order
1. **Next:** finish the remaining DS4 execution parity: detailed SSD cache
telemetry, expert profiling, resident multi-session batching, GLM MTP, and
the GLM/Pro hardware matrix.
2. Product completion, exhaustive parity verification, and distribution.
1. **Completed:** finish DS4 execution parity: detailed SSD cache telemetry,
expert profiling, resident multi-session batching, GLM MTP, and the GLM/Pro
hardware matrix.
2. **Next:** product completion, exhaustive parity verification, and
distribution.
3. Optional extensions: Dev Brain and A2UI.
## 1. Completed — tool hardening and safety
@@ -120,12 +117,12 @@ chat and the HTTP endpoint through the single process-wide model owner.
### 2.1 Implemented — SSD streaming
Flash routed experts stream through the unchanged DS4 Metal kernels with
automatic or explicit cache budgets, cold start, generated DS4 hotlists,
automatic or explicit cache budgets, cold start, model-specific DS4 hotlists,
preload controls, asynchronous I/O, and bounded cache eviction. Resident and
SSD generation share token oracles. Engine atomics expose resident/cache bytes,
requests, bytes read, and wait time; the UI samples and graphs their rates on
its normal metrics thread. Native cache hit/miss, eviction, and preload-progress
telemetry remains to be surfaced without changing the carried-over kernels.
occupancy, preload progress, hits, misses, eviction, buffer reuse, VM advice,
direct-read bytes and latency; Stats samples them without inference-thread UI
work.
SSD streaming is the capacity prerequisite for larger models and therefore
comes before GLM 5.2 and DeepSeek V4 Pro execution.
@@ -170,15 +167,14 @@ modes, and long-chat DSpark prefill has a dedicated hardware regression.
effective speedup in Stats. Do not call the feature complete merely because
it produces correct tokens; it must also preserve checkpoints, tools,
streaming responses, usage accounting, and Stop behavior.
- GLM's in-model MTP path belongs to the GLM milestone, but it should reuse the
verifier/session machinery established here.
- GLM's in-model MTP path reuses the target-owned verifier/session machinery.
### 2.3 Mostly completed — remaining Metal execution controls
### 2.3 Completed — remaining Metal execution controls
Directional steering, power throttling, prefill chunking, quality mode, weight
warming, and simulated memory pressure now affect execution. CPU helper-thread
and expert-profile settings are rejected instead of being persisted no-ops.
Porting DS4's expert-locality profile output is the remaining item here.
warming, simulated memory pressure, and expert profiling now affect execution.
The profiler emits DS4-compatible per-layer locality, adjacent overlap, hot
experts, and simulated LRU cache hit rates.
- Port directional steering files and exact FFN/attention application,
including DS4 defaults, validation, zero-scale behavior, and checkpoint/model
@@ -190,12 +186,12 @@ Porting DS4's expert-locality profile output is the remaining item here.
- Add hardware-backed token/activation fixtures for each mode and keep the
ordinary resident Flash path unchanged when optional features are off.
### 2.4 Remaining — single-machine server batching
### 2.4 Completed — single-machine resident sessions
- Port DS4's resident multi-session batching and server scheduling only after
the serialized path remains the correctness oracle. Preserve per-request
cancellation, finish reasons, usage, and KV ownership while batching prefill
or decode work.
- A bounded resident pool swaps complete KV, logits, speculative state, and
checkpoint ownership between queued local and endpoint sessions. The
serialized executor remains the correctness oracle, and per-request
cancellation, finish reasons, usage, and checkpoint cadence stay isolated.
- Keep all scheduling, model state, KV state, and request handling within the
local process. Networked execution and non-Metal backends are outside the
product scope.
@@ -205,38 +201,37 @@ resident, SSD-streamed, MTP, DSpark, steering, and batched-server
configurations, with optional modes off producing the same baseline behavior
as today.
## 3. Implemented executors — additional-model validation remains
## 3. Completed executors — additional-model validation matrix
GLM 5.2 has a dedicated DSA/MLA executor and DeepSeek V4 Pro uses the generalized
DeepSeek graph. Both are selectable runtimes, not catalog-only placeholders.
They are not complete parity milestones until the remaining items below pass on
the installed hardware fixtures.
DeepSeek graph. Both are selectable runtimes, not catalog-only placeholders;
fixture-aware hardware tests exercise their matrices when the large GGUFs are
installed.
### GLM 5.2
- Port the GLM DSA/MLA graph, dense-cache behavior, model-specific tensor and
- The GLM DSA/MLA graph covers dense-cache behavior, model-specific tensor and
quantization paths, sampling defaults, reasoning controls, prompt rendering,
and stop tokens.
- Use the already defined GLM tool syntax through the same durable local-agent
loop and expose identical behavior through every HTTP route.
- Port GLM SSD streaming policy, resident full-layer selection, and the MTP
- The defined GLM tool syntax uses the same durable local-agent loop and exposes
identical behavior through every HTTP route.
- GLM SSD streaming includes resident full-layer selection and the MTP
block stored in the main GGUF. Respect GLM restrictions on power, prefill
chunking, steering, and external support models.
- Validate resident and streamed token output against DS4 fixtures before the
Model Manager advertises GLM as runnable.
- Resident, streamed, and MTP token paths share a hardware parity fixture.
### DeepSeek V4 Pro
- Generalize the Flash graph only where Pro's dimensions, layers, routed
experts, quantization layouts, or output path actually differ.
- Support resident and SSD-streamed single-machine configurations with explicit
memory admission checks. Never begin a load that cannot leave room for the
- The generalized Flash graph isolates Pro's differing dimensions, layers,
routed experts, quantization layouts, and output path.
- Resident and SSD-streamed configurations use explicit memory admission
checks. Never begin a load that cannot leave room for the
configured KV/context and graph working set.
- Match Pro prompt, sampling, checkpoint, HTTP, and agent behavior. Preserve
the reference compatibility matrix for MTP/DSpark rather than assuming Flash
support artifacts work with Pro.
- Validate supported single-file Q2/Q4 configurations against DS4 fixtures
before advertising them as runnable.
- Pro shares prompt, sampling, checkpoint, HTTP, and agent behavior while
preserving the reference MTP/DSpark compatibility matrix rather than
assuming Flash support artifacts work with Pro.
- Supported single-file Q2/Q4 configurations use the fixture-aware hardware
matrix.
Exit criterion: each advertised model passes the same local-agent, checkpoint,
HTTP, SSD-capacity, cancellation, and deterministic token-output matrix as