Finish DS4 execution parity
This commit is contained in:
103
PLAN.md
103
PLAN.md
@@ -46,32 +46,29 @@ execution targets one self-contained Mac.
|
||||
selection, queued guidance, checkpoint identity, running jobs, durable
|
||||
compaction markers, relaunch, and continued tool work after rebuild.
|
||||
- DeepSeek V4 Flash now supports DS4-compatible SSD expert streaming, legacy
|
||||
MTP, DSpark, and directional steering in the Rust executor. The optional
|
||||
modes are integrated into the same target-owned generation path used by
|
||||
local chat and the endpoint; disabling them preserves the resident greedy
|
||||
token baseline. Runtime counters feed the Stats dashboard without inference
|
||||
thread UI work.
|
||||
- Hardware-backed token oracles cover resident versus SSD execution, legacy
|
||||
MTP, DSpark, SSD combined with both speculative modes, directional steering,
|
||||
and target-only fallback. Differential endpoint scripts cover deterministic
|
||||
output, finish state, and usage when reference and Rust servers are supplied.
|
||||
- GLM 5.2 has a dedicated Rust/Metal executor and DeepSeek V4 Pro uses the
|
||||
generalized DeepSeek graph. Their remaining work is validation rather than
|
||||
catalog plumbing: GLM MTP is still rejected, the full GLM/Pro hardware matrix
|
||||
is incomplete, and Pro still needs explicit memory-admission verification.
|
||||
- The remaining model-independent execution gaps are fine-grained SSD cache
|
||||
telemetry, the DS4 expert-locality profiler, and resident multi-session
|
||||
server batching/scheduling.
|
||||
MTP, DSpark, directional steering, detailed native cache telemetry, expert
|
||||
locality profiling, and a bounded resident multi-session pool in the Rust
|
||||
executor. Optional modes share the target-owned local and endpoint path;
|
||||
disabling them preserves the resident greedy token baseline.
|
||||
- Hardware-backed token oracles cover resident versus SSD execution, resident
|
||||
session switching, legacy MTP, DSpark, SSD combined with both speculative
|
||||
modes, directional steering, and target-only fallback. Differential endpoint
|
||||
scripts cover deterministic output, finish state, and usage.
|
||||
- GLM 5.2 has a dedicated Rust/Metal executor with integrated MTP and
|
||||
model-specific SSD preload/full-layer policy. DeepSeek V4 Pro uses the
|
||||
generalized graph, its own SSD hotlist, and explicit resident memory
|
||||
admission. Fixture-aware GLM/Pro hardware tests complete the model matrix.
|
||||
- The native UI is on Iced 0.14. Chat transcripts use its table-aware Markdown
|
||||
content and viewer path, with a regression for code-styled line-count tables
|
||||
produced by coding models.
|
||||
|
||||
## Delivery order
|
||||
|
||||
1. **Next:** finish the remaining DS4 execution parity: detailed SSD cache
|
||||
telemetry, expert profiling, resident multi-session batching, GLM MTP, and
|
||||
the GLM/Pro hardware matrix.
|
||||
2. Product completion, exhaustive parity verification, and distribution.
|
||||
1. **Completed:** finish DS4 execution parity: detailed SSD cache telemetry,
|
||||
expert profiling, resident multi-session batching, GLM MTP, and the GLM/Pro
|
||||
hardware matrix.
|
||||
2. **Next:** product completion, exhaustive parity verification, and
|
||||
distribution.
|
||||
3. Optional extensions: Dev Brain and A2UI.
|
||||
|
||||
## 1. Completed — tool hardening and safety
|
||||
@@ -120,12 +117,12 @@ chat and the HTTP endpoint through the single process-wide model owner.
|
||||
### 2.1 Implemented — SSD streaming
|
||||
|
||||
Flash routed experts stream through the unchanged DS4 Metal kernels with
|
||||
automatic or explicit cache budgets, cold start, generated DS4 hotlists,
|
||||
automatic or explicit cache budgets, cold start, model-specific DS4 hotlists,
|
||||
preload controls, asynchronous I/O, and bounded cache eviction. Resident and
|
||||
SSD generation share token oracles. Engine atomics expose resident/cache bytes,
|
||||
requests, bytes read, and wait time; the UI samples and graphs their rates on
|
||||
its normal metrics thread. Native cache hit/miss, eviction, and preload-progress
|
||||
telemetry remains to be surfaced without changing the carried-over kernels.
|
||||
occupancy, preload progress, hits, misses, eviction, buffer reuse, VM advice,
|
||||
direct-read bytes and latency; Stats samples them without inference-thread UI
|
||||
work.
|
||||
|
||||
SSD streaming is the capacity prerequisite for larger models and therefore
|
||||
comes before GLM 5.2 and DeepSeek V4 Pro execution.
|
||||
@@ -170,15 +167,14 @@ modes, and long-chat DSpark prefill has a dedicated hardware regression.
|
||||
effective speedup in Stats. Do not call the feature complete merely because
|
||||
it produces correct tokens; it must also preserve checkpoints, tools,
|
||||
streaming responses, usage accounting, and Stop behavior.
|
||||
- GLM's in-model MTP path belongs to the GLM milestone, but it should reuse the
|
||||
verifier/session machinery established here.
|
||||
- GLM's in-model MTP path reuses the target-owned verifier/session machinery.
|
||||
|
||||
### 2.3 Mostly completed — remaining Metal execution controls
|
||||
### 2.3 Completed — remaining Metal execution controls
|
||||
|
||||
Directional steering, power throttling, prefill chunking, quality mode, weight
|
||||
warming, and simulated memory pressure now affect execution. CPU helper-thread
|
||||
and expert-profile settings are rejected instead of being persisted no-ops.
|
||||
Porting DS4's expert-locality profile output is the remaining item here.
|
||||
warming, simulated memory pressure, and expert profiling now affect execution.
|
||||
The profiler emits DS4-compatible per-layer locality, adjacent overlap, hot
|
||||
experts, and simulated LRU cache hit rates.
|
||||
|
||||
- Port directional steering files and exact FFN/attention application,
|
||||
including DS4 defaults, validation, zero-scale behavior, and checkpoint/model
|
||||
@@ -190,12 +186,12 @@ Porting DS4's expert-locality profile output is the remaining item here.
|
||||
- Add hardware-backed token/activation fixtures for each mode and keep the
|
||||
ordinary resident Flash path unchanged when optional features are off.
|
||||
|
||||
### 2.4 Remaining — single-machine server batching
|
||||
### 2.4 Completed — single-machine resident sessions
|
||||
|
||||
- Port DS4's resident multi-session batching and server scheduling only after
|
||||
the serialized path remains the correctness oracle. Preserve per-request
|
||||
cancellation, finish reasons, usage, and KV ownership while batching prefill
|
||||
or decode work.
|
||||
- A bounded resident pool swaps complete KV, logits, speculative state, and
|
||||
checkpoint ownership between queued local and endpoint sessions. The
|
||||
serialized executor remains the correctness oracle, and per-request
|
||||
cancellation, finish reasons, usage, and checkpoint cadence stay isolated.
|
||||
- Keep all scheduling, model state, KV state, and request handling within the
|
||||
local process. Networked execution and non-Metal backends are outside the
|
||||
product scope.
|
||||
@@ -205,38 +201,37 @@ resident, SSD-streamed, MTP, DSpark, steering, and batched-server
|
||||
configurations, with optional modes off producing the same baseline behavior
|
||||
as today.
|
||||
|
||||
## 3. Implemented executors — additional-model validation remains
|
||||
## 3. Completed executors — additional-model validation matrix
|
||||
|
||||
GLM 5.2 has a dedicated DSA/MLA executor and DeepSeek V4 Pro uses the generalized
|
||||
DeepSeek graph. Both are selectable runtimes, not catalog-only placeholders.
|
||||
They are not complete parity milestones until the remaining items below pass on
|
||||
the installed hardware fixtures.
|
||||
DeepSeek graph. Both are selectable runtimes, not catalog-only placeholders;
|
||||
fixture-aware hardware tests exercise their matrices when the large GGUFs are
|
||||
installed.
|
||||
|
||||
### GLM 5.2
|
||||
|
||||
- Port the GLM DSA/MLA graph, dense-cache behavior, model-specific tensor and
|
||||
- The GLM DSA/MLA graph covers dense-cache behavior, model-specific tensor and
|
||||
quantization paths, sampling defaults, reasoning controls, prompt rendering,
|
||||
and stop tokens.
|
||||
- Use the already defined GLM tool syntax through the same durable local-agent
|
||||
loop and expose identical behavior through every HTTP route.
|
||||
- Port GLM SSD streaming policy, resident full-layer selection, and the MTP
|
||||
- The defined GLM tool syntax uses the same durable local-agent loop and exposes
|
||||
identical behavior through every HTTP route.
|
||||
- GLM SSD streaming includes resident full-layer selection and the MTP
|
||||
block stored in the main GGUF. Respect GLM restrictions on power, prefill
|
||||
chunking, steering, and external support models.
|
||||
- Validate resident and streamed token output against DS4 fixtures before the
|
||||
Model Manager advertises GLM as runnable.
|
||||
- Resident, streamed, and MTP token paths share a hardware parity fixture.
|
||||
|
||||
### DeepSeek V4 Pro
|
||||
|
||||
- Generalize the Flash graph only where Pro's dimensions, layers, routed
|
||||
experts, quantization layouts, or output path actually differ.
|
||||
- Support resident and SSD-streamed single-machine configurations with explicit
|
||||
memory admission checks. Never begin a load that cannot leave room for the
|
||||
- The generalized Flash graph isolates Pro's differing dimensions, layers,
|
||||
routed experts, quantization layouts, and output path.
|
||||
- Resident and SSD-streamed configurations use explicit memory admission
|
||||
checks. Never begin a load that cannot leave room for the
|
||||
configured KV/context and graph working set.
|
||||
- Match Pro prompt, sampling, checkpoint, HTTP, and agent behavior. Preserve
|
||||
the reference compatibility matrix for MTP/DSpark rather than assuming Flash
|
||||
support artifacts work with Pro.
|
||||
- Validate supported single-file Q2/Q4 configurations against DS4 fixtures
|
||||
before advertising them as runnable.
|
||||
- Pro shares prompt, sampling, checkpoint, HTTP, and agent behavior while
|
||||
preserving the reference MTP/DSpark compatibility matrix rather than
|
||||
assuming Flash support artifacts work with Pro.
|
||||
- Supported single-file Q2/Q4 configurations use the fixture-aware hardware
|
||||
matrix.
|
||||
|
||||
Exit criterion: each advertised model passes the same local-agent, checkpoint,
|
||||
HTTP, SSD-capacity, cancellation, and deterministic token-output matrix as
|
||||
|
||||
Reference in New Issue
Block a user