diff --git a/PLAN.md b/PLAN.md index a68d856..033b255 100644 --- a/PLAN.md +++ b/PLAN.md @@ -45,18 +45,31 @@ execution targets one self-contained Mac. - Focused coverage exercises triggers, summary bounds and sanitizing, tail selection, queued guidance, checkpoint identity, running jobs, durable compaction markers, relaunch, and continued tool work after rebuild. -- The next baseline gap is SSD streaming. Speculative decoding, steering, GLM - 5.2 execution, and DeepSeek V4 Pro execution are not implemented in the Rust - executor. Related catalog, validation, and preference plumbing must not be - treated as runtime support. +- DeepSeek V4 Flash now supports DS4-compatible SSD expert streaming, legacy + MTP, DSpark, and directional steering in the Rust executor. The optional + modes are integrated into the same target-owned generation path used by + local chat and the endpoint; disabling them preserves the resident greedy + token baseline. Runtime counters feed the Stats dashboard without inference + thread UI work. +- Hardware-backed token oracles cover resident versus SSD execution, legacy + MTP, DSpark, SSD combined with both speculative modes, directional steering, + and target-only fallback. Differential endpoint scripts cover deterministic + output, finish state, and usage when reference and Rust servers are supplied. +- GLM 5.2 has a dedicated Rust/Metal executor and DeepSeek V4 Pro uses the + generalized DeepSeek graph. Their remaining work is validation rather than + catalog plumbing: GLM MTP is still rejected, the full GLM/Pro hardware matrix + is incomplete, and Pro still needs explicit memory-admission verification. +- The remaining model-independent execution gaps are fine-grained SSD cache + telemetry, the DS4 expert-locality profiler, and resident multi-session + server batching/scheduling. ## Delivery order -1. **Next:** remaining DS4 execution technology, starting with SSD streaming, then - speculative decoding and the other Metal/runtime parity work. -2. Additional model execution: GLM 5.2 and DeepSeek V4 Pro. -3. Product completion, exhaustive parity verification, and distribution. -4. Optional extensions: Dev Brain and A2UI. +1. **Next:** finish the remaining DS4 execution parity: detailed SSD cache + telemetry, expert profiling, resident multi-session batching, GLM MTP, and + the GLM/Pro hardware matrix. +2. Product completion, exhaustive parity verification, and distribution. +3. Optional extensions: Dev Brain and A2UI. ## 1. Completed — tool hardening and safety @@ -95,13 +108,21 @@ Exit criterion: use the agent for a real inspect/edit/test cycle while every side effect is visible, risky actions require consent, Stop works at every stage, and no file tool can escape the selected project. -## 2. DS4 execution technology parity +## 2. In progress — DS4 execution technology parity Goal: finish the model-independent Metal/runtime capabilities in `ds4.c` before adding larger model families. Every capability must be shared by local chat and the HTTP endpoint through the single process-wide model owner. -### 2.1 SSD streaming — first runtime priority +### 2.1 Implemented — SSD streaming + +Flash routed experts stream through the unchanged DS4 Metal kernels with +automatic or explicit cache budgets, cold start, generated DS4 hotlists, +preload controls, asynchronous I/O, and bounded cache eviction. Resident and +SSD generation share token oracles. Engine atomics expose resident/cache bytes, +requests, bytes read, and wait time; the UI samples and graphs their rates on +its normal metrics thread. Native cache hit/miss, eviction, and preload-progress +telemetry remains to be surfaced without changing the carried-over kernels. SSD streaming is the capacity prerequisite for larger models and therefore comes before GLM 5.2 and DeepSeek V4 Pro execution. @@ -122,7 +143,14 @@ comes before GLM 5.2 and DeepSeek V4 Pro execution. the cache and I/O layer model-aware so later GLM/Pro milestones add policy and graph support rather than a second streaming subsystem. -### 2.2 Speculative decoding: legacy MTP and DSpark +### 2.2 Completed for Flash — legacy MTP and DSpark + +Both support GGUF formats are validated and executed by Rust. Draft generation, +confidence/margin gates, target verification, accepted-prefix commit, rollback, +strict/quality fallback, cancellation, checkpoint identity, streaming output, +usage accounting, and Stats counters use the existing generation lifecycle. +Sampled decoding remains target-only as in DS4. SSD streaming works with both +modes, and long-chat DSpark prefill has a dedicated hardware regression. - Load and validate the optional Flash legacy-MTP or DSpark support GGUF without treating either as a standalone model. Preserve exact support-kind and target @@ -142,7 +170,12 @@ comes before GLM 5.2 and DeepSeek V4 Pro execution. - GLM's in-model MTP path belongs to the GLM milestone, but it should reuse the verifier/session machinery established here. -### 2.3 Remaining Metal execution controls +### 2.3 Mostly completed — remaining Metal execution controls + +Directional steering, power throttling, prefill chunking, quality mode, weight +warming, and simulated memory pressure now affect execution. CPU helper-thread +and expert-profile settings are rejected instead of being persisted no-ops. +Porting DS4's expert-locality profile output is the remaining item here. - Port directional steering files and exact FFN/attention application, including DS4 defaults, validation, zero-scale behavior, and checkpoint/model @@ -154,7 +187,7 @@ comes before GLM 5.2 and DeepSeek V4 Pro execution. - Add hardware-backed token/activation fixtures for each mode and keep the ordinary resident Flash path unchanged when optional features are off. -### 2.4 Single-machine server batching +### 2.4 Remaining — single-machine server batching - Port DS4's resident multi-session batching and server scheduling only after the serialized path remains the correctness oracle. Preserve per-request @@ -169,11 +202,12 @@ resident, SSD-streamed, MTP, DSpark, steering, and batched-server configurations, with optional modes off producing the same baseline behavior as today. -## 3. Additional model execution +## 3. Implemented executors — additional-model validation remains -Start these only after the shared capacity and execution technology above is -stable. Catalog entries, settings, tokenizer work, or GGUF validation alone do -not make a model selectable for inference. +GLM 5.2 has a dedicated DSA/MLA executor and DeepSeek V4 Pro uses the generalized +DeepSeek graph. Both are selectable runtimes, not catalog-only placeholders. +They are not complete parity milestones until the remaining items below pass on +the installed hardware fixtures. ### GLM 5.2 diff --git a/scripts/execution_parity.py b/scripts/execution_parity.py new file mode 100755 index 0000000..737e487 --- /dev/null +++ b/scripts/execution_parity.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 +"""Run the DS4 token oracles, then the existing endpoint smoke corpus.""" + +import argparse +import pathlib +import subprocess +import sys + + +ROOT = pathlib.Path(__file__).resolve().parent.parent +HARDWARE_TESTS = ( + "flash_resident_and_ssd_streaming_choose_the_same_tokens", + "legacy_mtp_runs_a_target_owned_greedy_cycle", + "dspark_runs_a_target_owned_greedy_cycle", + "ssd_streaming_supports_legacy_mtp_and_dspark", + "directional_steering_matches_the_ds4_token_oracle", +) +ENDPOINT_SCRIPTS = ( + "endpoint_parity.py", + "endpoint_reasoning.py", + "endpoint_continuation.py", +) + + +def run(command): + print("+", " ".join(map(str, command)), flush=True) + subprocess.run(command, cwd=ROOT, check=True) + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument( + "--endpoint", + help="also run the existing endpoint parity scripts against this URL", + ) + parser.add_argument( + "--skip-hardware", + action="store_true", + help="skip tests requiring the local Flash/MTP/DSpark GGUF fixtures", + ) + args = parser.parse_args() + + if not args.skip_hardware: + for test in HARDWARE_TESTS: + run( + [ + "cargo", + "test", + "--all-features", + test, + "--", + "--ignored", + "--nocapture", + ] + ) + if args.endpoint: + for script in ENDPOINT_SCRIPTS: + run([sys.executable, ROOT / "scripts" / script, args.endpoint]) + + +if __name__ == "__main__": + main() diff --git a/scripts/fixtures/mtp_copy_prompt.txt b/scripts/fixtures/mtp_copy_prompt.txt new file mode 100644 index 0000000..29e33fc --- /dev/null +++ b/scripts/fixtures/mtp_copy_prompt.txt @@ -0,0 +1,23 @@ +Reproduce the following C code EXACTLY, character for character, inside a single code block and output nothing else: + +```c +static uint32_t clamp_u32(uint32_t v, uint32_t lo, uint32_t hi) { + if (v < lo) return lo; + if (v > hi) return hi; + return v; +} + +static uint32_t ring_advance(uint32_t pos, uint32_t cap) { + uint32_t next = pos + 1u; + return next >= cap ? 0u : next; +} + +static int scratch_init(scratch *s, uint32_t ctx_size) { + if (ctx_size == 0u) ctx_size = 1u; + s->ctx_size = ctx_size; + s->comp_cap = ctx_size / 4u + 2u; + s->rows = clamp_u32(s->comp_cap, 1u, 4096u); + s->head = 0u; + return s->rows > 0u ? 0 : -1; +} +``` diff --git a/scripts/speculative_parity.py b/scripts/speculative_parity.py new file mode 100755 index 0000000..fa99611 --- /dev/null +++ b/scripts/speculative_parity.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Compare DS4 and Rust endpoints configured for the same decoding mode.""" + +import argparse +import json +import urllib.request + + +CASES = [ + "hi", + "Reply with exactly three words describing a calm sea.", + "Write the first eight positive odd numbers separated by commas.", +] + + +def post(base_url, path, payload): + request = urllib.request.Request( + base_url.rstrip("/") + path, + data=json.dumps(payload).encode(), + headers={"Content-Type": "application/json"}, + ) + with urllib.request.urlopen(request, timeout=600) as response: + return json.load(response) + + +def chat(base_url, prompt): + result = post( + base_url, + "/v1/chat/completions", + { + "model": "deepseek-v4-flash", + "messages": [{"role": "user", "content": prompt}], + "reasoning_effort": "none", + "temperature": 0, + "max_tokens": 64, + }, + ) + choice = result["choices"][0] + message = choice["message"] + return { + "content": message.get("content"), + "reasoning": message.get("reasoning_content"), + "finish_reason": choice.get("finish_reason"), + "usage": result.get("usage"), + } + + +def anthropic(base_url, prompt): + result = post( + base_url, + "/v1/messages", + { + "model": "deepseek-v4-flash", + "messages": [{"role": "user", "content": prompt}], + "thinking": {"type": "disabled"}, + "temperature": 0, + "max_tokens": 64, + }, + ) + return { + "content": result.get("content"), + "stop_reason": result.get("stop_reason"), + "usage": result.get("usage"), + } + + +def responses(base_url, prompt): + result = post( + base_url, + "/v1/responses", + { + "model": "deepseek-v4-flash", + "input": prompt, + "reasoning": {"effort": "none"}, + "temperature": 0, + "max_output_tokens": 64, + }, + ) + return { + "output": normalize(result.get("output")), + "status": result.get("status"), + "usage": result.get("usage"), + } + + +def normalize(value): + if isinstance(value, list): + return [normalize(item) for item in value] + if isinstance(value, dict): + return { + key: normalize(item) + for key, item in value.items() + if key not in {"id", "created_at"} + } + return value + + +def main(): + parser = argparse.ArgumentParser() + parser.add_argument("reference_url", help="DS4 reference server URL") + parser.add_argument("rust_url", help="Rust DS4Server URL in the same mode") + args = parser.parse_args() + + failures = [] + for prompt in CASES: + for name, request in ( + ("chat", chat), + ("anthropic", anthropic), + ("responses", responses), + ): + reference = request(args.reference_url, prompt) + rust = request(args.rust_url, prompt) + if reference != rust: + failures.append( + { + "case": name, + "prompt": prompt, + "reference": reference, + "rust": rust, + } + ) + else: + print(f"ok {name}: {prompt}") + if failures: + print(json.dumps(failures, indent=2, ensure_ascii=False)) + raise SystemExit(1) + print("all deterministic DS4/Rust endpoint outputs and usage records match") + + +if __name__ == "__main__": + main() diff --git a/src/app.rs b/src/app.rs index 5bbedf9..11d3bed 100644 --- a/src/app.rs +++ b/src/app.rs @@ -90,6 +90,9 @@ pub(crate) struct App { pub(super) kv_cache_report: KvCacheReport, last_cache_scan: Instant, last_http_requests: u64, + last_ssd_selected_requests: u64, + last_ssd_requested_bytes: u64, + last_ssd_wait_ms: u64, #[cfg(target_os = "macos")] generation_service: Option, #[cfg(target_os = "macos")] @@ -136,6 +139,9 @@ pub(super) struct MetricsPoint { pub(super) http_requests_per_second: f32, pub(super) kv_read_bytes_per_second: f32, pub(super) kv_write_bytes_per_second: f32, + pub(super) ssd_requests_per_second: f32, + pub(super) ssd_bytes_per_second: f32, + pub(super) ssd_wait_ms_per_second: f32, } #[derive(Debug, Clone)] @@ -154,6 +160,7 @@ pub(crate) enum Message { FocusNext, FocusPrevious, PreferenceModelChanged(ModelChoice), + PreferenceLegacyMtpChanged(bool), PreferenceDsparkChanged(bool), PreferenceTimeoutChanged(String), PreferenceEndpointPortChanged(String), @@ -313,6 +320,9 @@ impl App { kv_cache_report: KvCacheReport::default(), last_cache_scan: Instant::now() - CACHE_SCAN_INTERVAL, last_http_requests: 0, + last_ssd_selected_requests: 0, + last_ssd_requested_bytes: 0, + last_ssd_wait_ms: 0, #[cfg(target_os = "macos")] generation_service, #[cfg(target_os = "macos")] @@ -416,6 +426,9 @@ impl App { kv_cache_report: KvCacheReport::default(), last_cache_scan: Instant::now() - CACHE_SCAN_INTERVAL, last_http_requests: 0, + last_ssd_selected_requests: 0, + last_ssd_requested_bytes: 0, + last_ssd_wait_ms: 0, #[cfg(target_os = "macos")] generation_service, #[cfg(target_os = "macos")] @@ -535,6 +548,7 @@ impl App { Message::PreferenceModelChanged(model) => { self.preference_draft.model = model; if !model.supports_dspark() { + self.preference_draft.legacy_mtp_enabled = false; self.preference_draft.dspark_enabled = false; self.preference_draft.dspark_confidence_threshold.clear(); self.preference_draft.dspark_strict = false; @@ -552,6 +566,16 @@ impl App { } self.preference_error = None; } + Message::PreferenceLegacyMtpChanged(enabled) => { + self.preference_draft.legacy_mtp_enabled = + self.preference_draft.model.supports_dspark() && enabled; + if self.preference_draft.legacy_mtp_enabled { + self.preference_draft.dspark_enabled = false; + self.preference_draft.dspark_confidence_threshold.clear(); + self.preference_draft.dspark_strict = false; + } + self.preference_error = None; + } Message::PreferenceDsparkChanged(enabled) => { self.preference_draft.dspark_enabled = self.preference_draft.model.supports_dspark() && enabled; @@ -559,7 +583,7 @@ impl App { self.preference_draft.dspark_confidence_threshold.clear(); self.preference_draft.dspark_strict = false; } else { - self.preference_draft.ssd_streaming = false; + self.preference_draft.legacy_mtp_enabled = false; } self.preference_error = None; } @@ -665,7 +689,7 @@ impl App { .is_empty() { self.preference_draft.dspark_enabled = true; - self.preference_draft.ssd_streaming = false; + self.preference_draft.legacy_mtp_enabled = false; } self.preference_error = None; } @@ -674,17 +698,12 @@ impl App { self.preference_draft.model.supports_dspark() && value; if self.preference_draft.dspark_strict { self.preference_draft.dspark_enabled = true; - self.preference_draft.ssd_streaming = false; + self.preference_draft.legacy_mtp_enabled = false; } self.preference_error = None; } Message::PreferenceSsdChanged(value) => { self.preference_draft.ssd_streaming = value; - if value { - self.preference_draft.dspark_enabled = false; - self.preference_draft.dspark_confidence_threshold.clear(); - self.preference_draft.dspark_strict = false; - } self.preference_error = None; } Message::PreferenceSsdColdChanged(value) => { @@ -1226,7 +1245,23 @@ impl App { let kv_read_bytes_per_second = kv_read_bytes as f32 / METRICS_SAMPLE_INTERVAL.as_secs_f32(); let kv_write_bytes_per_second = kv_write_bytes as f32 / METRICS_SAMPLE_INTERVAL.as_secs_f32(); + let sample_seconds = METRICS_SAMPLE_INTERVAL.as_secs_f32(); + let ssd_requests_per_second = snapshot + .ssd_selected_requests + .saturating_sub(self.last_ssd_selected_requests) + as f32 + / sample_seconds; + let ssd_bytes_per_second = snapshot + .ssd_requested_bytes + .saturating_sub(self.last_ssd_requested_bytes) + as f32 + / sample_seconds; + let ssd_wait_ms_per_second = + snapshot.ssd_wait_ms.saturating_sub(self.last_ssd_wait_ms) as f32 / sample_seconds; self.last_http_requests = snapshot.http_requests; + self.last_ssd_selected_requests = snapshot.ssd_selected_requests; + self.last_ssd_requested_bytes = snapshot.ssd_requested_bytes; + self.last_ssd_wait_ms = snapshot.ssd_wait_ms; self.metrics_history.push_back(MetricsPoint { decode_tokens_per_second: if snapshot.phase == crate::metrics::RuntimePhase::Generating { @@ -1238,6 +1273,9 @@ impl App { http_requests_per_second, kv_read_bytes_per_second, kv_write_bytes_per_second, + ssd_requests_per_second, + ssd_bytes_per_second, + ssd_wait_ms_per_second, }); if self.metrics_history.len() > 120 { self.metrics_history.pop_front(); diff --git a/src/app/preferences.rs b/src/app/preferences.rs index 73a7b31..052e182 100644 --- a/src/app/preferences.rs +++ b/src/app/preferences.rs @@ -3,6 +3,7 @@ use super::*; #[derive(Clone)] pub(super) struct PreferenceDraft { pub(super) model: ModelChoice, + pub(super) legacy_mtp_enabled: bool, pub(super) dspark_enabled: bool, pub(super) idle_timeout_minutes: String, pub(super) endpoint_port: String, @@ -51,6 +52,7 @@ impl PreferenceDraft { let speculative = &runtime.speculative; Self { model: config.model, + legacy_mtp_enabled: speculative.legacy_mtp_enabled, dspark_enabled: speculative.dspark_enabled, idle_timeout_minutes: config.idle_timeout_minutes.to_string(), endpoint_port: config.endpoint.port.to_string(), @@ -136,6 +138,7 @@ impl PreferenceDraft { Ok(SpeculativePreferences { mtp_draft_tokens: parse_positive_i32("MTP draft tokens", &self.mtp_draft_tokens)?, mtp_margin: parse_f32("MTP margin", &self.mtp_margin)?, + legacy_mtp_enabled: self.legacy_mtp_enabled, glm_mtp: self.glm_mtp, glm_mtp_timing: self.glm_mtp_timing, dspark_enabled: self.dspark_enabled, diff --git a/src/app/view/preferences.rs b/src/app/view/preferences.rs index 581e1e5..fbf9650 100644 --- a/src/app/view/preferences.rs +++ b/src/app/view/preferences.rs @@ -3,6 +3,19 @@ use iced::widget::column; impl App { pub(super) fn preferences_panel(&self) -> Element<'_, Message> { + let legacy_mtp_toggle: Option Message> = self + .preference_draft + .model + .supports_dspark() + .then_some(Message::PreferenceLegacyMtpChanged); + let legacy_mtp = hint( + checkbox( + "Enable legacy MTP for this model", + self.preference_draft.legacy_mtp_enabled, + ) + .on_toggle_maybe(legacy_mtp_toggle), + "Uses the managed one-stage MTP support GGUF. The target model verifies every drafted token; it is mutually exclusive with DSpark.", + ); let dspark_toggle: Option Message> = self .preference_draft .model @@ -14,7 +27,7 @@ impl App { self.preference_draft.dspark_enabled, ) .on_toggle_maybe(dspark_toggle), - "Speculative decoding with the managed DSpark draft artifact: a small model proposes tokens that the main model verifies in one pass. Usually a large speedup, and it cannot be combined with SSD streaming.", + "Speculative decoding with the managed DSpark draft artifact: a small model proposes tokens that the main model verifies in one pass. Usually a large speedup; the target model may also stream routed experts from SSD.", ); let glm_mtp_toggle: Option Message> = (self.preference_draft.model == ModelChoice::Glm52) @@ -311,6 +324,7 @@ impl App { .on_toggle_maybe(glm_mtp_timing_toggle), "Records per-stage timings of the speculative path to the log, to show where the acceleration actually goes. A diagnostic aid that costs a little throughput.", ), + legacy_mtp, dspark, preference_input_row( "DSpark confidence threshold", @@ -326,7 +340,7 @@ impl App { "Lets the draft model only propose, never decide: every token is sampled by the full model. Gives up some of the speedup in exchange for output identical to non-speculative decoding.", ), text(if self.preference_draft.model.supports_dspark() { - "DSpark uses the managed support artifact; entering a threshold or enabling strict mode also enables DSpark." + "Legacy MTP and DSpark use separate managed support artifacts; entering a DSpark threshold or enabling strict mode selects DSpark." } else if self.preference_draft.model == ModelChoice::Glm52 { "GLM MTP is integrated; DSpark is unavailable for this model." } else { @@ -339,9 +353,10 @@ impl App { |engine| { let settings = engine.speculative; format!( - "Engine: MTP draft {} • margin {} • GLM MTP {} • timing {} • DSpark {} • confidence {}{} • target-only {}", + "Engine: MTP draft {} • margin {} • legacy MTP {} • GLM MTP {} • timing {} • DSpark {} • confidence {}{} • target-only {}", settings.mtp_draft_tokens, settings.mtp_margin, + if self.preference_draft.legacy_mtp_enabled { "on" } else { "off" }, if settings.glm_mtp { "on" } else { "off" }, if settings.glm_mtp_timing { "on" } else { "off" }, if settings.dspark { "on" } else { "off" }, @@ -357,7 +372,7 @@ impl App { hint( checkbox("Enable SSD-backed model streaming", self.preference_draft.ssd_streaming) .on_toggle(Message::PreferenceSsdChanged), - "Leaves the routed expert weights on disk and pages them in as they are needed, so a model larger than this machine's memory still runs. Every cache miss waits for the SSD, and DSpark cannot run alongside it.", + "Leaves the routed expert weights on disk and pages them in as they are needed, so a model larger than this machine's memory still runs. Every cache miss waits for the SSD; speculative support weights remain resident while target experts stream.", ), hint( checkbox("Skip automatic expert preload", self.preference_draft.ssd_streaming_cold) @@ -381,7 +396,7 @@ impl App { text_input("Automatic", &self.preference_draft.ssd_preload_experts) .on_input(Message::PreferenceSsdPreloadChanged), ), - text("A blank full-layer value is automatic; an explicit 0 disables fully resident GLM layers. SSD streaming and DSpark are mutually exclusive.") + text("A blank full-layer value is automatic; an explicit 0 disables fully resident GLM layers. Flash legacy MTP and DSpark support weights remain resident when target experts stream.") .size(12), text(engine.as_ref().map_or_else( || "Effective SSD settings will appear after valid values are entered." diff --git a/src/app/view/stats.rs b/src/app/view/stats.rs index 3d4fd3b..3e977e4 100644 --- a/src/app/view/stats.rs +++ b/src/app/view/stats.rs @@ -198,6 +198,55 @@ impl App { .spacing(7) .into(), ); + let ssd_activity = stats_panel( + "SSD STREAMING ACTIVITY · LAST 24 SECONDS", + column![ + mini_chart( + &self.metrics_history, + |point| point.ssd_requests_per_second, + Color::from_rgb8(240, 180, 70), + ), + row![ + text("Selected loads") + .size(12) + .color(Color::from_rgb8(240, 180, 70)), + Space::with_width(Length::Fill), + text(format!("{:.1}/s", latest.ssd_requests_per_second)) + .size(12) + .color(muted_text()), + ], + mini_chart( + &self.metrics_history, + |point| point.ssd_bytes_per_second, + Color::from_rgb8(67, 194, 203), + ), + row![ + text("Requested expert data") + .size(12) + .color(Color::from_rgb8(67, 194, 203)), + Space::with_width(Length::Fill), + text(format_rate(latest.ssd_bytes_per_second)) + .size(12) + .color(muted_text()), + ], + mini_chart( + &self.metrics_history, + |point| point.ssd_wait_ms_per_second, + Color::from_rgb8(220, 80, 86), + ), + row![ + text("Inference wait") + .size(12) + .color(Color::from_rgb8(220, 80, 86)), + Space::with_width(Length::Fill), + text(format!("{:.0} ms/s", latest.ssd_wait_ms_per_second)) + .size(12) + .color(muted_text()), + ], + ] + .spacing(7) + .into(), + ); let model = stats_panel( "MODEL CORE", @@ -249,6 +298,86 @@ impl App { .spacing(9) .into(), ); + let acceptance = if stats.drafted_tokens == 0 { + 0.0 + } else { + stats.accepted_draft_tokens as f64 / stats.drafted_tokens as f64 + }; + let target_passes = stats + .speculative_cycles + .saturating_add(stats.verifier_passes); + let effective_speedup = if target_passes == 0 { + 1.0 + } else { + stats + .speculative_cycles + .saturating_add(stats.accepted_draft_tokens) as f64 + / target_passes as f64 + }; + let speculative = stats_panel( + "SPECULATIVE DECODING", + column![ + metric_row( + "Mode", + match stats.speculative_mode { + 1 => "Legacy MTP", + 2 => "DSpark", + _ => "Off", + }, + ), + metric_row("Cycles", format_count(stats.speculative_cycles)), + metric_row("Drafted", format_count(stats.drafted_tokens)), + metric_row("Accepted", format_count(stats.accepted_draft_tokens)), + metric_row("Acceptance", format!("{:.1}%", acceptance * 100.0)), + metric_row( + "Target verifier passes", + format_count(stats.verifier_passes) + ), + metric_row("Verifier wall time", format_milliseconds(stats.verifier_ms)), + metric_row( + "Effective target-pass speedup", + format!("{effective_speedup:.2}×") + ), + ] + .spacing(9) + .into(), + ); + let ssd = stats_panel( + "SSD EXPERT STREAMING", + column![ + metric_row("State", if stats.ssd_enabled { "Enabled" } else { "Off" }), + metric_row("Resident weights", format_bytes(stats.ssd_resident_bytes)), + metric_row("Expert cache", format_bytes(stats.ssd_cache_bytes)), + metric_row( + "Cache capacity", + format!("{} experts", stats.ssd_cache_experts) + ), + metric_row( + "Preloaded", + format!("{} experts", stats.ssd_preloaded_experts) + ), + metric_row( + "Selected-load requests", + format_count(stats.ssd_selected_requests) + ), + metric_row( + "Requested expert bytes", + format_bytes(stats.ssd_requested_bytes) + ), + metric_row("Selected-load wait", format_milliseconds(stats.ssd_wait_ms)), + metric_row( + "Average load wait", + format_milliseconds( + stats + .ssd_wait_ms + .checked_div(stats.ssd_selected_requests) + .unwrap_or(0) + ) + ), + ] + .spacing(9) + .into(), + ); let cache = stats_panel( "KV CACHE", column![ @@ -352,10 +481,12 @@ impl App { heading, headline, throughput, + ssd_activity, kv_io, requests, disc, row![model, runtime].spacing(10), + row![speculative, ssd].spacing(10), row![cache, server].spacing(10), text("Counters are published by the runtime with relaxed atomics and sampled by the UI every 200 ms.") .size(11) diff --git a/src/engine.rs b/src/engine.rs index e20a80f..ff5ee52 100644 --- a/src/engine.rs +++ b/src/engine.rs @@ -22,7 +22,22 @@ use std::sync::atomic::{AtomicBool, Ordering}; #[cfg(target_os = "macos")] use std::time::Instant; use tokenizer::Tokenizer; -use validation::{validate_dspark, validate_main}; +use validation::{SupportKind, validate_main, validate_support}; + +#[cfg(target_os = "macos")] +unsafe extern "C" { + fn mmap( + address: *mut std::ffi::c_void, + length: usize, + protection: i32, + flags: i32, + fd: i32, + offset: i64, + ) -> *mut std::ffi::c_void; + fn mlock(address: *const std::ffi::c_void, length: usize) -> i32; + fn munlock(address: *const std::ffi::c_void, length: usize) -> i32; + fn munmap(address: *mut std::ffi::c_void, length: usize) -> i32; +} pub(crate) use validation::validate_model_artifact; @@ -189,6 +204,7 @@ const GLM: Shape = Shape { pub(crate) struct Model { main: Gguf, support: Option, + support_kind: Option, shape: Shape, tokenizer: Tokenizer, } @@ -206,10 +222,17 @@ impl Model { #[allow(dead_code)] pub(crate) fn open(settings: &EngineSettings) -> Result { let mut model = Self::open_main(&settings.artifacts.model, settings.model)?; + if settings.execution.warm_weights { + model.main.warm()?; + } if let Some(path) = &settings.artifacts.mtp { let support = Gguf::open(path)?; - validate_dspark(&support, &model.shape)?; + let kind = validate_support(&support, &model.shape)?; + if settings.execution.warm_weights { + support.warm()?; + } model.support = Some(support); + model.support_kind = Some(kind); } Ok(model) } @@ -228,6 +251,7 @@ impl Model { Ok(Self { main, support: None, + support_kind: None, shape, tokenizer, }) @@ -333,6 +357,7 @@ impl Model { #[cfg(target_os = "macos")] pub(crate) struct Generator { executor: metal::Executor, + _simulated_memory: Option, checkpoint: Option, /// Token frontier of the last transient store, so continued checkpoints are /// spaced like ds4's `continued_last_store_tokens`. @@ -340,6 +365,68 @@ pub(crate) struct Generator { metrics: Arc, } +#[cfg(target_os = "macos")] +struct SimulatedMemory { + address: std::ptr::NonNull, + bytes: usize, +} + +#[cfg(target_os = "macos")] +impl SimulatedMemory { + fn acquire(bytes: u64) -> Result, String> { + if bytes == 0 { + return Ok(None); + } + let bytes = usize::try_from(bytes).map_err(|_| "simulated memory size is too large")?; + let address = unsafe { mmap(std::ptr::null_mut(), bytes, 1 | 2, 2 | 0x1000, -1, 0) }; + let Some(address) = std::ptr::NonNull::new(address) else { + return Err(format!( + "cannot reserve simulated used memory: {}", + std::io::Error::last_os_error() + )); + }; + if address.as_ptr() as isize == -1 { + return Err(format!( + "cannot reserve simulated used memory: {}", + std::io::Error::last_os_error() + )); + } + let chunk = 256 * 1024 * 1024; + let mut locked = 0; + while locked < bytes { + let length = (bytes - locked).min(chunk); + let start = unsafe { address.as_ptr().cast::().add(locked) }; + for page in (0..length).step_by(16 * 1024) { + unsafe { start.add(page).write((page / (16 * 1024)) as u8) }; + } + unsafe { start.add(length - 1).write(1) }; + if unsafe { mlock(start.cast(), length) } != 0 { + if locked != 0 { + unsafe { munlock(address.as_ptr(), locked) }; + } + unsafe { munmap(address.as_ptr(), bytes) }; + return Err(format!( + "cannot lock simulated used memory after {:.2} GiB: {}", + locked as f64 / 1_073_741_824.0, + std::io::Error::last_os_error() + )); + } + locked += length; + } + Ok(Some(Self { address, bytes })) + } +} + +#[cfg(target_os = "macos")] +impl Drop for SimulatedMemory { + fn drop(&mut self) { + unsafe { + munlock(self.address.as_ptr(), self.bytes); + munmap(self.address.as_ptr(), self.bytes); + } + } +} + #[derive(Clone)] pub(crate) struct ChatTurn { pub(crate) user: bool, @@ -377,25 +464,41 @@ impl Generator { "GLM MTP requires the shared speculative verifier, which is not enabled".into(), ); } - if settings.speculative.dspark - || (settings.ssd.enabled && settings.model != ModelChoice::Glm52) - || settings.steering.file.is_some() - { - return Err( - "DSpark, SSD streaming, and steering are not yet available in the Rust executor" - .into(), - ); - } + let simulated_memory = + SimulatedMemory::acquire(settings.diagnostics.simulated_used_memory_bytes)?; let model = Model::open(settings)?; let executor = metal::Executor::open_configured( model, settings.context_tokens.max(1) as u32, settings.execution.quality, settings.execution.prefill_chunk, + settings.execution.power_percent, + settings.speculative, settings.ssd, + settings.steering.clone(), )?; + let stats = executor.execution_stats(); + metrics.speculative_stats( + stats.speculative_mode, + stats.speculative_cycles, + stats.drafted_tokens, + stats.accepted_draft_tokens, + stats.verifier_passes, + stats.verifier_ms, + ); + metrics.ssd_stats( + stats.ssd_enabled, + stats.ssd_resident_bytes, + stats.ssd_cache_bytes, + stats.ssd_cache_experts, + stats.ssd_preloaded_experts, + stats.ssd_selected_requests, + stats.ssd_requested_bytes, + stats.ssd_wait_ms, + ); Ok(Self { executor, + _simulated_memory: simulated_memory, checkpoint: None, last_store_tokens: 0, metrics, @@ -422,8 +525,9 @@ impl Generator { checkpoint, conversation_tag(&settings.system_prompt, settings.reasoning_mode, history), )?; - let (mut output, prompt_complete) = - self.generate_inner(messages, settings, cancelled, &mut emit, &mut progress)?; + let result = self.generate_inner(messages, settings, cancelled, &mut emit, &mut progress); + self.publish_execution_stats(); + let (mut output, prompt_complete) = result?; let mut completed = messages.to_vec(); completed.push(output.message.clone()); output.previous_checkpoint_bytes = @@ -442,6 +546,28 @@ impl Generator { Ok(output) } + fn publish_execution_stats(&self) { + let stats = self.executor.execution_stats(); + self.metrics.speculative_stats( + stats.speculative_mode, + stats.speculative_cycles, + stats.drafted_tokens, + stats.accepted_draft_tokens, + stats.verifier_passes, + stats.verifier_ms, + ); + self.metrics.ssd_stats( + stats.ssd_enabled, + stats.ssd_resident_bytes, + stats.ssd_cache_bytes, + stats.ssd_cache_experts, + stats.ssd_preloaded_experts, + stats.ssd_selected_requests, + stats.ssd_requested_bytes, + stats.ssd_wait_ms, + ); + } + pub(crate) fn generate_transient( &mut self, directory: &Path, @@ -481,8 +607,9 @@ impl Generator { self.metrics.kv_lookup(KvLookup::MemoryHit); } - let (mut output, prompt_complete) = - self.generate_inner(messages, settings, cancelled, &mut emit, &mut progress)?; + let result = self.generate_inner(messages, settings, cancelled, &mut emit, &mut progress); + self.publish_execution_stats(); + let (mut output, prompt_complete) = result?; let mut completed = messages.to_vec(); completed.push(output.message.clone()); output.previous_checkpoint_bytes = previous_checkpoint @@ -809,6 +936,7 @@ impl Generator { } completed }; + self.publish_execution_stats(); if completed != suffix.len() { return Ok(( GenerationOutput { @@ -826,11 +954,12 @@ impl Generator { progress(self.executor.position(), self.executor.context(), Some(0.0)); let generation_started = Instant::now(); let mut generated_tokens = 0_u32; - for _ in 0..settings + let generation_limit = settings .max_generated_tokens .max(0) .min((max_context - self.executor.position() as usize) as i32) - { + as u32; + while generated_tokens < generation_limit { if cancelled.load(Ordering::Relaxed) { append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); flush_generated( @@ -887,86 +1016,132 @@ impl Generator { true, )); } - if self.executor.model().is_think_start_token(token) { - append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); - reasoning = true; - generated.reasoning.get_or_insert_default(); - } else if self.executor.model().is_think_end_token(token) { - append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); - if let Some(reasoning_text) = &mut generated.reasoning - && emit_safe_text( - reasoning_text, - &mut emitted_reasoning, - &settings.stops, - true, - true, - emit, - ) + let cycle = if settings.temperature <= 0.0 { + self.executor.eval_speculative_greedy( + token, + generation_limit - generated_tokens, + settings.reasoning_mode, + cancelled, + )? + } else { + self.executor.eval(token)?; + vec![token] + }; + self.publish_execution_stats(); + for token in cycle { + if generated_tokens >= generation_limit + || self + .executor + .model() + .is_stop_token_for_reasoning(token, settings.reasoning_mode) { - return Ok(( - GenerationOutput { - message: generated, - prompt_tokens: prompt_tokens as u32, - cached_tokens: reused as u32, - completion_tokens: generated_tokens + 1, - finish_reason: "stop", - previous_checkpoint_bytes: None, - checkpoint_bytes: 0, - }, - false, - )); - } - reasoning = false; - generated.reasoning_complete = true; - emit(false, String::new()); - } else if let Some(bytes) = self.executor.model().token_bytes(token) { - append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &bytes, false); - let stopped = if reasoning { - let text = generated.reasoning.get_or_insert_default(); - emit_safe_text( - text, + append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); + flush_generated( + &mut generated, &mut emitted_reasoning, - &settings.stops, - false, - true, - emit, - ) - } else { - generated.reasoning_complete = true; - emit_safe_text( - &mut generated.content, &mut emitted_content, &settings.stops, - false, - false, emit, - ) - }; - if stopped { + ); return Ok(( GenerationOutput { message: generated, prompt_tokens: prompt_tokens as u32, cached_tokens: reused as u32, - completion_tokens: generated_tokens + 1, + completion_tokens: generated_tokens, finish_reason: "stop", previous_checkpoint_bytes: None, checkpoint_bytes: 0, }, - false, + true, )); } + if self.executor.model().is_think_start_token(token) { + append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); + reasoning = true; + generated.reasoning.get_or_insert_default(); + } else if self.executor.model().is_think_end_token(token) { + append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); + if let Some(reasoning_text) = &mut generated.reasoning + && emit_safe_text( + reasoning_text, + &mut emitted_reasoning, + &settings.stops, + true, + true, + emit, + ) + { + return Ok(( + GenerationOutput { + message: generated, + prompt_tokens: prompt_tokens as u32, + cached_tokens: reused as u32, + completion_tokens: generated_tokens + 1, + finish_reason: "stop", + previous_checkpoint_bytes: None, + checkpoint_bytes: 0, + }, + false, + )); + } + reasoning = false; + generated.reasoning_complete = true; + emit(false, String::new()); + } else if let Some(bytes) = self.executor.model().token_bytes(token) { + append_generated_bytes( + &mut generated, + reasoning, + &mut pending_utf8, + &bytes, + false, + ); + let stopped = if reasoning { + let text = generated.reasoning.get_or_insert_default(); + emit_safe_text( + text, + &mut emitted_reasoning, + &settings.stops, + false, + true, + emit, + ) + } else { + generated.reasoning_complete = true; + emit_safe_text( + &mut generated.content, + &mut emitted_content, + &settings.stops, + false, + false, + emit, + ) + }; + if stopped { + return Ok(( + GenerationOutput { + message: generated, + prompt_tokens: prompt_tokens as u32, + cached_tokens: reused as u32, + completion_tokens: generated_tokens + 1, + finish_reason: "stop", + previous_checkpoint_bytes: None, + checkpoint_bytes: 0, + }, + false, + )); + } + } + generated_tokens += 1; + progress( + self.executor.position(), + self.executor.context(), + Some( + generated_tokens as f32 + / generation_started.elapsed().as_secs_f32().max(1.0e-6), + ), + ); } - self.executor.eval(token)?; - generated_tokens += 1; - progress( - self.executor.position(), - self.executor.context(), - Some( - generated_tokens as f32 - / generation_started.elapsed().as_secs_f32().max(1.0e-6), - ), - ); } append_generated_bytes(&mut generated, reasoning, &mut pending_utf8, &[], true); flush_generated( diff --git a/src/engine/gguf.rs b/src/engine/gguf.rs index e39cc19..d52d314 100644 --- a/src/engine/gguf.rs +++ b/src/engine/gguf.rs @@ -1,4 +1,4 @@ -use memmap2::{Mmap, MmapOptions}; +use memmap2::{Advice, Mmap, MmapOptions}; use sha2::{Digest, Sha256}; use std::collections::HashMap; use std::fs::File; @@ -215,6 +215,23 @@ impl Gguf { self.max_tensor_bytes } + pub(super) fn warm(&self) -> Result<(), String> { + let start = self.data_offset as usize; + if start >= self.map.len() { + return Ok(()); + } + self.map + .advise_range(Advice::WillNeed, start, self.map.len() - start) + .map_err(|error| format!("Cannot warm {}: {error}", self.path.display()))?; + let mut checksum = 0_u64; + for offset in (start..self.map.len()).step_by(16 * 1024) { + checksum = checksum.wrapping_add(u64::from(self.map[offset])); + } + checksum = checksum.wrapping_add(u64::from(self.map[self.map.len() - 1])); + std::hint::black_box(checksum); + Ok(()) + } + pub(super) fn tensor(&self, name: &str) -> Result<&Tensor, String> { self.tensors .get(name) @@ -544,6 +561,7 @@ mod tests { assert_eq!(model.bytes("general.architecture").unwrap(), b"deepseek4"); assert_eq!(model.tensor("weight").unwrap().dims, [1]); assert_eq!(model.tensor_data("weight").unwrap(), 1_f32.to_le_bytes()); + model.warm().unwrap(); fs::remove_file(path).unwrap(); } diff --git a/src/engine/metal.rs b/src/engine/metal.rs index 64b1fe2..6b2a9ad 100644 --- a/src/engine/metal.rs +++ b/src/engine/metal.rs @@ -1,19 +1,26 @@ mod checkpoint; mod glm; mod gpu; +mod hotlist; use glm::GlmExecutor; use gpu::*; -use super::gguf::{F16, Gguf, Q8_0, Tensor as GgufTensor}; +use super::gguf::{F16, F32, Gguf, Q4_K, Q8_0, Tensor as GgufTensor}; +use super::validation::{DsparkConfig, SupportKind, dspark_config}; use super::{Model, ModelFamily}; +use crate::model::ModelChoice; +use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, ReasoningMode, +}; +use sha2::{Digest, Sha256}; use std::env; use std::ffi::c_void; use std::fs::{self, File}; use std::io::{Read, Write}; use std::path::Path; use std::ptr::NonNull; -use std::time::UNIX_EPOCH; +use std::time::{Duration, Instant, UNIX_EPOCH}; const CHECKPOINT_MAGIC: &[u8; 8] = b"DS4RKV01"; const CHECKPOINT_VERSION: u32 = 2; @@ -153,8 +160,35 @@ struct IndexerWeights { impl Layer { fn bind(model: &Gguf, shape: super::Shape, index: u32) -> Result { - let required = |suffix: &str| Weight::bind(model, &format!("blk.{index}.{suffix}")); - let optional = |suffix: &str| Weight::optional(model, &format!("blk.{index}.{suffix}")); + let layer = Self::bind_prefix(model, shape, index, &format!("blk.{index}"))?; + for (name, weight) in [ + ("attention q", layer.attn_q_b), + ("attention output A", layer.attn_output_a), + ("attention output B", layer.attn_output_b), + ("shared gate", layer.shared_gate), + ("shared up", layer.shared_up), + ("shared down", layer.shared_down), + ] { + if weight.kind != Q8_0 { + return Err(format!("DeepSeek Metal path requires Q8_0 {name} weights")); + } + } + if let Some(indexer) = layer.indexer + && (!matches!(indexer.q.kind, F16 | Q8_0) || indexer.proj.kind != F16) + { + return Err("DeepSeek Metal path requires F16/Q8 indexer weights".into()); + } + Ok(layer) + } + + fn bind_prefix( + model: &Gguf, + shape: super::Shape, + index: u32, + prefix: &str, + ) -> Result { + let required = |suffix: &str| Weight::bind(model, &format!("{prefix}.{suffix}")); + let optional = |suffix: &str| Weight::optional(model, &format!("{prefix}.{suffix}")); let attn_compressor = (compression_ratio(shape, index) != 0) .then(|| { Ok::<_, String>(CompressorWeights { @@ -179,7 +213,7 @@ impl Layer { }) }) .transpose()?; - let layer = Self { + Ok(Self { hc_attn_fn: required("hc_attn_fn.weight")?, hc_attn_scale: required("hc_attn_scale.weight")?, hc_attn_base: required("hc_attn_base.weight")?, @@ -207,25 +241,7 @@ impl Layer { shared_gate: required("ffn_gate_shexp.weight")?, shared_up: required("ffn_up_shexp.weight")?, shared_down: required("ffn_down_shexp.weight")?, - }; - for (name, weight) in [ - ("attention q", layer.attn_q_b), - ("attention output A", layer.attn_output_a), - ("attention output B", layer.attn_output_b), - ("shared gate", layer.shared_gate), - ("shared up", layer.shared_up), - ("shared down", layer.shared_down), - ] { - if weight.kind != Q8_0 { - return Err(format!("DeepSeek Metal path requires Q8_0 {name} weights")); - } - } - if let Some(indexer) = layer.indexer - && (!matches!(indexer.q.kind, F16 | Q8_0) || indexer.proj.kind != F16) - { - return Err("DeepSeek Metal path requires F16/Q8 indexer weights".into()); - } - Ok(layer) + }) } } @@ -239,6 +255,1576 @@ struct Weights { layers: Vec, } +struct LegacyMtpWeights { + e_proj: Weight, + h_proj: Weight, + enorm: Weight, + hnorm: Weight, + norm: Weight, + hc_head_base: Weight, + hc_head_fn: Weight, + hc_head_scale: Weight, + block: Layer, +} + +impl LegacyMtpWeights { + fn bind(model: &Gguf, shape: super::Shape) -> Result { + Ok(Self { + e_proj: Weight::bind(model, "mtp.0.e_proj.weight")?, + h_proj: Weight::bind(model, "mtp.0.h_proj.weight")?, + enorm: Weight::bind(model, "mtp.0.enorm.weight")?, + hnorm: Weight::bind(model, "mtp.0.hnorm.weight")?, + norm: Weight::bind(model, "mtp.0.norm.weight")?, + hc_head_base: Weight::bind(model, "mtp.0.hc_head_base.weight")?, + hc_head_fn: Weight::bind(model, "mtp.0.hc_head_fn.weight")?, + hc_head_scale: Weight::bind(model, "mtp.0.hc_head_scale.weight")?, + block: Layer::bind_prefix(model, shape, 1, "mtp.0")?, + }) + } +} + +struct LegacyMtp { + weights: LegacyMtpWeights, + layer: LayerState, + state_hc: Buffer, + next_hc: Buffer, + draft_token: Option, + raw_rows: u32, + draft_limit: u32, + margin: f32, + drafted: u64, + accepted: u64, +} + +struct DsparkStageWeights { + block: Layer, + main_proj: Option, + main_norm: Option, + norm: Option, + hc_head_base: Option, + hc_head_fn: Option, + hc_head_scale: Option, + markov_w1: Option, + markov_w2: Option, + confidence: Option, +} + +impl DsparkStageWeights { + fn bind(model: &Gguf, shape: super::Shape, stage: u32) -> Result { + let prefix = format!("mtp.{stage}"); + Ok(Self { + block: Layer::bind_prefix(model, shape, 1, &prefix)?, + main_proj: Weight::optional(model, &format!("{prefix}.main_proj.weight")), + main_norm: Weight::optional(model, &format!("{prefix}.main_norm.weight")), + norm: Weight::optional(model, &format!("{prefix}.norm.weight")), + hc_head_base: Weight::optional(model, &format!("{prefix}.hc_head_base.weight")), + hc_head_fn: Weight::optional(model, &format!("{prefix}.hc_head_fn.weight")), + hc_head_scale: Weight::optional(model, &format!("{prefix}.hc_head_scale.weight")), + markov_w1: Weight::optional(model, &format!("{prefix}.markov_head.markov_w1.weight")), + markov_w2: Weight::optional(model, &format!("{prefix}.markov_head.markov_w2.weight")), + confidence: Weight::optional(model, &format!("{prefix}.confidence_head.proj.weight")), + }) + } +} + +struct Dspark { + config: DsparkConfig, + weights: Vec, + mean_weights: Buffer, + mean_rows: Buffer, + target_hidden: Buffer, + target_hidden_batch: Buffer, + packed_target_hidden: Buffer, + stage0_proj: Buffer, + main_x: Buffer, + stage_input_hc: Buffer, + stage_output_hc: Buffer, + draft_tokens: Buffer, + raw_caches: Vec, + scratch: BatchScratch, + logits: Buffer, + capture_mask: u32, + cache_start: u32, + cache_len: u32, + confidence_threshold: f32, + strict: bool, + drafted: u64, + accepted: u64, +} + +impl Dspark { + fn new( + model: &Model, + support: &Gguf, + session: &Session, + settings: EngineSpeculativeSettings, + quality: bool, + ) -> Result { + let config = dspark_config(support)?; + let shape = model.shape; + let rows = config.block_size + 1; + if rows > session.prefill_cap { + return Err(format!( + "DSpark block needs {rows} rows, but prefill workspace has {}", + session.prefill_cap + )); + } + let weights = (0..config.stages) + .map(|stage| DsparkStageWeights::bind(support, shape, stage)) + .collect::, _>>()?; + let hc_dim = shape.hc * shape.embd; + let mean_weights = Buffer::floats(shape.hc)?; + let mean_rows = Buffer::floats(u64::from(session.prefill_cap) * shape.hc)?; + let mean = vec![1.0 / shape.hc as f32; shape.hc as usize]; + let repeated = + vec![1.0 / shape.hc as f32; (session.prefill_cap as u64 * shape.hc) as usize]; + mean_weights.write_f32(&mean)?; + mean_rows.write_f32(&repeated)?; + Ok(Self { + target_hidden: Buffer::floats(config.target_layers.len() as u64 * shape.embd)?, + target_hidden_batch: Buffer::floats( + config.target_layers.len() as u64 * u64::from(session.prefill_cap) * shape.embd, + )?, + packed_target_hidden: Buffer::floats( + u64::from(session.prefill_cap) * config.target_layers.len() as u64 * shape.embd, + )?, + stage0_proj: Buffer::floats(shape.embd)?, + main_x: Buffer::floats(shape.embd)?, + stage_input_hc: Buffer::floats(u64::from(rows) * hc_dim)?, + stage_output_hc: Buffer::floats(u64::from(config.block_size) * hc_dim)?, + draft_tokens: Buffer::bytes(u64::from(config.block_size) * 4)?, + raw_caches: (0..config.stages) + .map(|_| Buffer::floats(u64::from(session.raw_cap) * shape.head_dim)) + .collect::>()?, + scratch: BatchScratch::allocate(model, session.context, rows)?, + logits: Buffer::floats(u64::from(config.block_size) * shape.vocab)?, + config, + weights, + mean_weights, + mean_rows, + capture_mask: 0, + cache_start: 0, + cache_len: 0, + confidence_threshold: settings.dspark_confidence_threshold, + strict: settings.dspark_strict || quality, + drafted: 0, + accepted: 0, + }) + } + + fn target_slot(&self, layer: u32) -> Option { + self.config + .target_layers + .iter() + .position(|target| *target == layer) + .map(|slot| slot as u32) + } + + fn begin_capture(&mut self) { + self.capture_mask = 0; + } + + fn commit_proposed_prefix(&mut self, rows: u32, raw_cap: u32) { + let added = rows.min(raw_cap); + let total = self.cache_len.saturating_add(added); + if total > raw_cap { + self.cache_start = (self.cache_start + total - raw_cap) % raw_cap; + } + self.cache_len = total.min(raw_cap); + } + + fn capture_decode( + &mut self, + layer: u32, + hc: &Buffer, + shape: super::Shape, + ) -> Result<(), String> { + let Some(slot) = self.target_slot(layer) else { + return Ok(()); + }; + let target = self + .target_hidden + .view(u64::from(slot) * shape.embd * 4, shape.embd * 4)?; + call( + unsafe { + ds4_gpu_hc_weighted_sum_tensor( + target.raw(), + hc.raw(), + self.mean_weights.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "capturing DSpark target hidden state", + )?; + self.capture_mask |= 1 << slot; + Ok(()) + } + + fn capture_batch( + &mut self, + layer: u32, + hc: &Buffer, + rows: u32, + prefill_cap: u32, + shape: super::Shape, + ) -> Result<(), String> { + let Some(slot) = self.target_slot(layer) else { + return Ok(()); + }; + let batch = self.target_hidden_batch.view( + u64::from(slot) * u64::from(prefill_cap) * shape.embd * 4, + u64::from(rows) * shape.embd * 4, + )?; + call( + unsafe { + ds4_gpu_hc_weighted_sum_tensor( + batch.raw(), + hc.raw(), + self.mean_rows.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "capturing batched DSpark target hidden states", + )?; + call( + unsafe { + ds4_gpu_tensor_copy( + self.target_hidden.raw(), + u64::from(slot) * shape.embd * 4, + batch.raw(), + u64::from(rows - 1) * shape.embd * 4, + shape.embd * 4, + ) + }, + "capturing final DSpark target hidden state", + )?; + self.capture_mask |= 1 << slot; + Ok(()) + } + + fn capture_complete(&self) -> bool { + self.capture_mask == (1_u32 << self.config.target_layers.len()) - 1 + } + + fn seed_batch_cache( + &mut self, + support: &Gguf, + pos: u32, + rows: u32, + prefill_cap: u32, + raw_cap: u32, + shape: super::Shape, + ) -> Result<(), String> { + if !self.capture_complete() { + return Err("DSpark target-layer capture is incomplete".into()); + } + let stage0 = self.weights.first().ok_or("DSpark has no stages")?; + let main_proj = stage0 + .main_proj + .ok_or("DSpark stage 0 projection is missing")?; + let main_norm = stage0.main_norm.ok_or("DSpark stage 0 norm is missing")?; + let map = support.map_ptr().cast(); + let size = support.len(); + let input = self.config.target_layers.len() as u64 * shape.embd; + let projected = self + .target_hidden_batch + .view(0, u64::from(rows) * shape.embd * 4)?; + let norm = self + .packed_target_hidden + .view(0, u64::from(rows) * shape.embd * 4)?; + let kv_bytes = u64::from(rows) * shape.head_dim * 4; + let kv_raw = self.target_hidden_batch.view(0, kv_bytes)?; + let kv = self.target_hidden_batch.view(kv_bytes, kv_bytes)?; + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_pack_slot_rows_f32_tensor( + self.packed_target_hidden.raw(), + self.target_hidden_batch.raw(), + rows, + shape.embd as u32, + self.config.target_layers.len() as u32, + prefill_cap, + ) + }, + "packing DSpark target hidden states", + )?; + matmul_rows( + &projected, + main_proj, + input, + shape.embd, + &self.packed_target_hidden, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + norm.raw(), + projected.raw(), + map, + size, + main_norm.offset, + shape.embd as u32, + rows, + shape.rms_epsilon, + ) + }, + "normalizing DSpark target hidden states", + )?; + for (stage, cache) in self.weights.iter().zip(&self.raw_caches) { + matmul_rows( + &kv_raw, + stage.block.attn_kv, + shape.embd, + shape.head_dim, + &norm, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + kv.raw(), + kv_raw.raw(), + map, + size, + stage.block.attn_kv_norm.offset, + shape.head_dim as u32, + rows, + shape.rms_epsilon, + ) + }, + "normalizing DSpark target KV", + )?; + call( + unsafe { + ds4_gpu_rope_tail_tensor( + kv.raw(), + rows, + 1, + shape.head_dim as u32, + shape.rot as u32, + pos, + 0, + false, + shape.rope_base, + 1.0, + 0.0, + 1.0, + shape.rope_beta_fast, + shape.rope_beta_slow, + ) + }, + "applying DSpark target KV RoPE", + )?; + call( + unsafe { + ds4_gpu_dsv4_fp8_kv_quantize_tensor( + kv.raw(), + rows, + shape.head_dim as u32, + shape.rot as u32, + ) + }, + "quantizing DSpark target KV", + )?; + call( + unsafe { + ds4_gpu_store_raw_kv_batch_tensor( + cache.raw(), + kv.raw(), + raw_cap, + pos, + rows, + shape.head_dim as u32, + ) + }, + "seeding DSpark target KV cache", + )?; + } + commands.finish()?; + self.cache_start = pos % raw_cap; + self.cache_len = rows; + Ok(()) + } + + fn seed_current_cache( + &mut self, + support: &Gguf, + pos: u32, + raw_cap: u32, + shape: super::Shape, + ) -> Result<(), String> { + if !self.capture_complete() { + return Err("DSpark target-layer capture is incomplete".into()); + } + let stage0 = self.weights.first().ok_or("DSpark has no stages")?; + let main_proj = stage0 + .main_proj + .ok_or("DSpark stage 0 projection is missing")?; + let main_norm = stage0.main_norm.ok_or("DSpark stage 0 norm is missing")?; + let map = support.map_ptr().cast(); + let size = support.len(); + let input = self.config.target_layers.len() as u64 * shape.embd; + let commands = Commands::begin()?; + matmul( + &self.stage0_proj, + main_proj, + input, + shape.embd, + &self.target_hidden, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_tensor( + self.main_x.raw(), + self.stage0_proj.raw(), + map, + size, + main_norm.offset, + shape.embd as u32, + shape.rms_epsilon, + ) + }, + "normalizing current DSpark target state", + )?; + for (stage, cache) in self.weights.iter().zip(&self.raw_caches) { + matmul( + &self.scratch.kv_raw, + stage.block.attn_kv, + shape.embd, + shape.head_dim, + &self.main_x, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + self.scratch.kv.raw(), + self.scratch.kv_raw.raw(), + map, + size, + stage.block.attn_kv_norm.offset, + shape.head_dim as u32, + 1, + shape.rms_epsilon, + ) + }, + "normalizing current DSpark target KV", + )?; + call( + unsafe { + ds4_gpu_rope_tail_tensor( + self.scratch.kv.raw(), + 1, + 1, + shape.head_dim as u32, + shape.rot as u32, + pos, + 0, + false, + shape.rope_base, + 1.0, + 0.0, + 1.0, + shape.rope_beta_fast, + shape.rope_beta_slow, + ) + }, + "applying current DSpark target KV RoPE", + )?; + call( + unsafe { + ds4_gpu_dsv4_fp8_kv_quantize_tensor( + self.scratch.kv.raw(), + 1, + shape.head_dim as u32, + shape.rot as u32, + ) + }, + "quantizing current DSpark target KV", + )?; + call( + unsafe { + ds4_gpu_store_raw_kv_batch_tensor( + cache.raw(), + self.scratch.kv.raw(), + raw_cap, + pos, + 1, + shape.head_dim as u32, + ) + }, + "storing current DSpark target KV", + )?; + } + commands.finish()?; + let append = (self.cache_start + self.cache_len) % raw_cap; + if self.cache_len == 0 || append != pos % raw_cap { + self.cache_start = pos % raw_cap; + self.cache_len = 1; + } else if self.cache_len < raw_cap { + self.cache_len += 1; + } else { + self.cache_start = (self.cache_start + 1) % raw_cap; + } + Ok(()) + } + + fn propose( + &mut self, + base: &Model, + base_weights: &Weights, + token: i32, + pos: u32, + raw_cap: u32, + ) -> Result, String> { + if self.strict || !self.capture_complete() { + return Ok(Vec::new()); + } + let support = base + .support + .as_ref() + .ok_or("DSpark support model is missing")?; + let shape = base.shape; + let rows = self.config.block_size + 1; + if rows > raw_cap { + return Err("DSpark block exceeds its raw-cache capacity".into()); + } + let max_support = raw_cap - rows; + if self.cache_len > max_support { + let discard = self.cache_len - max_support; + self.cache_start = (self.cache_start + discard) % raw_cap; + self.cache_len = max_support; + } + let map = support.map_ptr().cast(); + let size = support.len(); + let stage0 = self.weights.first().ok_or("DSpark has no stages")?; + let main_proj = stage0 + .main_proj + .ok_or("DSpark stage 0 projection is missing")?; + let main_norm = stage0.main_norm.ok_or("DSpark stage 0 norm is missing")?; + let input = self.config.target_layers.len() as u64 * shape.embd; + let mut ids = vec![self.config.noise_token as i32; self.config.block_size as usize]; + ids[0] = token; + self.draft_tokens.write_i32(&ids)?; + self.scratch.tokens.write_i32(&ids)?; + let commands = Commands::begin()?; + matmul( + &self.stage0_proj, + main_proj, + input, + shape.embd, + &self.target_hidden, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_tensor( + self.main_x.raw(), + self.stage0_proj.raw(), + map, + size, + main_norm.offset, + shape.embd as u32, + shape.rms_epsilon, + ) + }, + "normalizing DSpark stage-0 projection", + )?; + call( + unsafe { + ds4_gpu_repeat_hc_tensor( + self.stage_input_hc.raw(), + self.main_x.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "expanding DSpark target hidden state", + )?; + let draft_hc = self.stage_input_hc.view( + shape.hc * shape.embd * 4, + u64::from(self.config.block_size) * shape.hc * shape.embd * 4, + )?; + call( + unsafe { + ds4_gpu_embed_tokens_hc_tensor( + draft_hc.raw(), + self.draft_tokens.raw(), + base.main.map_ptr().cast(), + base.main.len(), + base_weights.token_embedding.offset, + shape.vocab as u32, + self.config.block_size, + shape.embd as u32, + shape.hc as u32, + ) + }, + "embedding DSpark draft block", + )?; + commands.finish()?; + + for stage in 0..self.weights.len() { + self.eval_stage(support, stage, pos, raw_cap, shape)?; + if stage + 1 < self.weights.len() { + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_tensor_copy( + self.stage_input_hc.raw(), + shape.hc * shape.embd * 4, + self.scratch.next_hc.raw(), + 0, + u64::from(self.config.block_size) * shape.hc * shape.embd * 4, + ) + }, + "feeding the next DSpark stage", + )?; + commands.finish()?; + } + } + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_tensor_copy( + self.stage_output_hc.raw(), + 0, + self.scratch.next_hc.raw(), + 0, + u64::from(self.config.block_size) * shape.hc * shape.embd * 4, + ) + }, + "capturing DSpark stage output", + )?; + commands.finish()?; + self.eval_output_heads(base, support, base_weights, token) + } + + fn eval_stage( + &mut self, + support: &Gguf, + stage_index: usize, + pos: u32, + raw_cap: u32, + shape: super::Shape, + ) -> Result<(), String> { + let stage = &self.weights[stage_index]; + let w = &stage.block; + let map = support.map_ptr().cast(); + let size = support.len(); + let draft = self.config.block_size; + let rows = draft + 1; + let hc_dim = shape.hc * shape.embd; + let mix_hc = 2 * shape.hc + shape.hc * shape.hc; + let q_dim = shape.heads * shape.head_dim; + let group_dim = shape.head_dim * (shape.heads / shape.out_groups); + let draft_norm = self + .scratch + .norm + .view(shape.embd * 4, u64::from(draft) * shape.embd * 4)?; + let draft_input = self + .stage_input_hc + .view(hc_dim * 4, u64::from(draft) * hc_dim * 4)?; + let draft_split = self + .scratch + .hc_split + .view(mix_hc * 4, u64::from(draft) * mix_hc * 4)?; + let after_attention_hc = self + .scratch + .after_attention_hc + .view(0, u64::from(draft) * hc_dim * 4)?; + let target_kv = self.scratch.kv.view(0, shape.head_dim * 4)?; + let draft_kv = self + .scratch + .kv + .view(shape.head_dim * 4, u64::from(draft) * shape.head_dim * 4)?; + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_rms_norm_plain_rows_tensor( + self.scratch.flat_hc.raw(), + self.stage_input_hc.raw(), + hc_dim as u32, + rows, + shape.rms_epsilon, + ) + }, + "normalizing DSpark attention HC rows", + )?; + matmul_rows( + &self.scratch.hc_mix, + w.hc_attn_fn, + hc_dim, + mix_hc, + &self.scratch.flat_hc, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_hc_split_weighted_sum_norm_tensor( + self.scratch.current.raw(), + self.scratch.norm.raw(), + self.scratch.hc_split.raw(), + self.scratch.hc_mix.raw(), + self.stage_input_hc.raw(), + map, + size, + w.hc_attn_scale.offset, + w.hc_attn_base.offset, + w.attn_norm.offset, + shape.embd as u32, + shape.hc as u32, + shape.hc_sinkhorn as u32, + shape.hc_epsilon, + shape.rms_epsilon, + ) + }, + "mixing DSpark attention HC rows", + )?; + matmul_rows( + &self.scratch.q_rank, + w.attn_q_a, + shape.embd, + shape.lora_q, + &draft_norm, + draft, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + self.scratch.q_rank_norm.raw(), + self.scratch.q_rank.raw(), + map, + size, + w.attn_q_a_norm.offset, + shape.lora_q as u32, + draft, + shape.rms_epsilon, + ) + }, + "normalizing DSpark Q rank", + )?; + matmul_rows( + &self.scratch.q, + w.attn_q_b, + shape.lora_q, + q_dim, + &self.scratch.q_rank_norm, + draft, + map, + size, + )?; + call( + unsafe { + ds4_gpu_head_rms_norm_tensor( + self.scratch.q.raw(), + draft, + shape.heads as u32, + shape.head_dim as u32, + shape.rms_epsilon, + ) + }, + "normalizing DSpark Q heads", + )?; + call( + unsafe { + ds4_gpu_rope_tail_tensor( + self.scratch.q.raw(), + draft, + shape.heads as u32, + shape.head_dim as u32, + shape.rot as u32, + pos, + 0, + false, + shape.rope_base, + 1.0, + 0.0, + 1.0, + shape.rope_beta_fast, + shape.rope_beta_slow, + ) + }, + "applying DSpark Q RoPE", + )?; + matmul_rows( + &self.scratch.kv_raw, + w.attn_kv, + shape.embd, + shape.head_dim, + &self.scratch.norm, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + self.scratch.kv.raw(), + self.scratch.kv_raw.raw(), + map, + size, + w.attn_kv_norm.offset, + shape.head_dim as u32, + rows, + shape.rms_epsilon, + ) + }, + "normalizing DSpark KV rows", + )?; + for (view, count) in [(&target_kv, 1), (&draft_kv, draft)] { + call( + unsafe { + ds4_gpu_rope_tail_tensor( + view.raw(), + count, + 1, + shape.head_dim as u32, + shape.rot as u32, + pos, + 0, + false, + shape.rope_base, + 1.0, + 0.0, + 1.0, + shape.rope_beta_fast, + shape.rope_beta_slow, + ) + }, + "applying DSpark KV RoPE", + )?; + } + call( + unsafe { + ds4_gpu_dsv4_fp8_kv_quantize_tensor( + self.scratch.kv.raw(), + rows, + shape.head_dim as u32, + shape.rot as u32, + ) + }, + "quantizing DSpark KV rows", + )?; + let append = (self.cache_start + self.cache_len) % raw_cap; + call( + unsafe { + ds4_gpu_store_raw_kv_batch_tensor( + self.raw_caches[stage_index].raw(), + self.scratch.kv.raw(), + raw_cap, + append, + rows, + shape.head_dim as u32, + ) + }, + "storing DSpark stage KV rows", + )?; + call( + unsafe { + ds4_gpu_attention_noncausal_raw_batch_heads_tensor( + self.scratch.heads.raw(), + map, + size, + w.attn_sinks.offset, + self.scratch.q.raw(), + self.raw_caches[stage_index].raw(), + draft, + self.cache_len + rows, + raw_cap, + self.cache_start, + shape.heads as u32, + shape.head_dim as u32, + ) + }, + "running DSpark noncausal attention", + )?; + call( + unsafe { + ds4_gpu_rope_tail_tensor( + self.scratch.heads.raw(), + draft, + shape.heads as u32, + shape.head_dim as u32, + shape.rot as u32, + pos, + 0, + true, + shape.rope_base, + 1.0, + 0.0, + 1.0, + shape.rope_beta_fast, + shape.rope_beta_slow, + ) + }, + "applying inverse DSpark attention RoPE", + )?; + call( + unsafe { + ds4_gpu_attention_output_q8_batch_tensor( + self.scratch.attention_out.raw(), + self.scratch.attention_low.raw(), + self.scratch.attention_group_tmp.raw(), + self.scratch.attention_low_tmp.raw(), + map, + size, + w.attn_output_a.offset, + w.attn_output_b.offset, + group_dim, + shape.lora_o, + shape.out_groups as u32, + shape.embd, + self.scratch.heads.raw(), + draft, + ) + }, + "projecting DSpark attention output", + )?; + call( + unsafe { + ds4_gpu_hc_expand_split_tensor( + after_attention_hc.raw(), + self.scratch.attention_out.raw(), + draft_input.raw(), + draft_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "expanding DSpark attention HC", + )?; + self.eval_stage_ffn(stage_index, shape, support)?; + commands.finish() + } + + fn eval_stage_ffn( + &mut self, + stage_index: usize, + shape: super::Shape, + support: &Gguf, + ) -> Result<(), String> { + let w = &self.weights[stage_index].block; + let map = support.map_ptr().cast(); + let size = support.len(); + let rows = self.config.block_size; + let hc_dim = shape.hc * shape.embd; + let mix_hc = 2 * shape.hc + shape.hc * shape.hc; + let hc_bytes = u64::from(rows) * hc_dim * 4; + let mix_bytes = u64::from(rows) * mix_hc * 4; + let embd_bytes = u64::from(rows) * shape.embd * 4; + let after_attention_hc = self.scratch.after_attention_hc.view(0, hc_bytes)?; + let flat_hc = self.scratch.flat_hc.view(0, hc_bytes)?; + let hc_mix = self.scratch.hc_mix.view(0, mix_bytes)?; + let hc_split = self.scratch.hc_split.view(0, mix_bytes)?; + let current = self.scratch.current.view(0, embd_bytes)?; + let norm = self.scratch.norm.view(0, embd_bytes)?; + let next_hc = self.scratch.next_hc.view(0, hc_bytes)?; + call( + unsafe { + ds4_gpu_rms_norm_plain_rows_tensor( + flat_hc.raw(), + after_attention_hc.raw(), + hc_dim as u32, + rows, + shape.rms_epsilon, + ) + }, + "normalizing DSpark FFN HC rows", + )?; + matmul_rows( + &hc_mix, + w.hc_ffn_fn, + hc_dim, + mix_hc, + &flat_hc, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_hc_split_weighted_sum_norm_tensor( + current.raw(), + norm.raw(), + hc_split.raw(), + hc_mix.raw(), + after_attention_hc.raw(), + map, + size, + w.hc_ffn_scale.offset, + w.hc_ffn_base.offset, + w.ffn_norm.offset, + shape.embd as u32, + shape.hc as u32, + shape.hc_sinkhorn as u32, + shape.hc_epsilon, + shape.rms_epsilon, + ) + }, + "mixing DSpark FFN HC rows", + )?; + matmul_rows( + &self.scratch.router_logits, + w.router, + shape.embd, + shape.experts, + &norm, + rows, + map, + size, + )?; + call( + unsafe { + ds4_gpu_router_select_batch_tensor( + self.scratch.router_selected.raw(), + self.scratch.router_weights.raw(), + self.scratch.router_probs.raw(), + map, + size, + w.router_bias.map_or(0, |weight| weight.offset), + 0, + 0, + 0, + 0, + w.router_bias.is_some(), + false, + self.scratch.router_logits.raw(), + self.scratch.tokens.raw(), + shape.experts as u32, + shape.experts_used as u32, + shape.expert_weight_scale, + rows, + ) + }, + "routing DSpark experts", + )?; + for (out, weight) in [ + (&self.scratch.shared_gate, w.shared_gate), + (&self.scratch.shared_up, w.shared_up), + ] { + matmul_rows( + out, + weight, + shape.embd, + shape.ff_expert, + &norm, + rows, + map, + size, + )?; + } + call( + unsafe { + ds4_gpu_swiglu_tensor( + self.scratch.shared_mid.raw(), + self.scratch.shared_gate.raw(), + self.scratch.shared_up.raw(), + rows * shape.ff_expert as u32, + shape.swiglu_clamp, + 1.0, + ) + }, + "activating DSpark shared expert", + )?; + matmul_rows( + &self.scratch.shared_out, + w.shared_down, + shape.ff_expert, + shape.embd, + &self.scratch.shared_mid, + rows, + map, + size, + )?; + let gate_row = w.expert_gate.bytes / (w.expert_gate.dims[1] * w.expert_gate.dims[2]); + let down_row = w.expert_down.bytes / (w.expert_down.dims[1] * w.expert_down.dims[2]); + let mut mid_f16 = false; + call( + unsafe { + ds4_gpu_routed_moe_batch_tensor( + self.scratch.routed_out.raw(), + self.scratch.routed_gate.raw(), + self.scratch.routed_up.raw(), + self.scratch.routed_mid.raw(), + self.scratch.routed_experts.raw(), + map, + size, + w.expert_gate.offset, + w.expert_up.offset, + w.expert_down.offset, + w.expert_gate.kind, + w.expert_down.kind, + w.expert_gate.dims[1] * gate_row, + gate_row, + w.expert_down.dims[1] * down_row, + down_row, + w.expert_gate.dims[0] as u32, + w.expert_down.dims[0] as u32, + w.expert_down.dims[1] as u32, + self.scratch.router_selected.raw(), + self.scratch.router_weights.raw(), + shape.experts as u32, + shape.experts_used as u32, + shape.swiglu_clamp, + norm.raw(), + stage_index as u32, + rows, + &mut mid_f16, + false, + ) + }, + "running DSpark routed experts", + )?; + call( + unsafe { + ds4_gpu_hc_expand_add_split_tensor( + next_hc.raw(), + self.scratch.routed_out.raw(), + self.scratch.shared_out.raw(), + after_attention_hc.raw(), + hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "expanding DSpark FFN HC", + ) + } + + fn eval_output_heads( + &mut self, + base: &Model, + support: &Gguf, + base_weights: &Weights, + first_token: i32, + ) -> Result, String> { + let shape = base.shape; + let draft = self.config.block_size; + let hc_dim = shape.hc * shape.embd; + let final_stage = self.weights.last().ok_or("DSpark has no final stage")?; + let norm = final_stage.norm.ok_or("DSpark final norm is missing")?; + let head_base = final_stage + .hc_head_base + .ok_or("DSpark final HC base is missing")?; + let head_fn = final_stage + .hc_head_fn + .ok_or("DSpark final HC projection is missing")?; + let head_scale = final_stage + .hc_head_scale + .ok_or("DSpark final HC scale is missing")?; + let markov_w1 = final_stage.markov_w1.ok_or("DSpark Markov W1 is missing")?; + let markov_w2 = final_stage.markov_w2.ok_or("DSpark Markov W2 is missing")?; + let map = support.map_ptr().cast(); + let size = support.len(); + let output_pre = self + .scratch + .hc_mix + .view(0, u64::from(draft) * shape.hc * 4)?; + let output_weights = self + .scratch + .hc_split + .view(0, u64::from(draft) * shape.hc * 4)?; + let output_embedding = self + .scratch + .current + .view(0, u64::from(draft) * shape.embd * 4)?; + let output_norm = self + .scratch + .norm + .view(0, u64::from(draft) * shape.embd * 4)?; + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_rms_norm_plain_rows_tensor( + self.scratch.flat_hc.raw(), + self.stage_output_hc.raw(), + hc_dim as u32, + draft, + shape.rms_epsilon, + ) + }, + "normalizing DSpark output HC rows", + )?; + matmul_rows( + &output_pre, + head_fn, + hc_dim, + shape.hc, + &self.scratch.flat_hc, + draft, + map, + size, + )?; + call( + unsafe { + ds4_gpu_output_hc_weights_tensor( + output_weights.raw(), + output_pre.raw(), + map, + size, + head_scale.offset, + head_base.offset, + shape.hc as u32, + shape.hc_epsilon, + ) + }, + "computing DSpark output HC weights", + )?; + call( + unsafe { + ds4_gpu_hc_weighted_sum_tensor( + output_embedding.raw(), + self.stage_output_hc.raw(), + output_weights.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "collapsing DSpark output HC rows", + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + output_norm.raw(), + output_embedding.raw(), + map, + size, + norm.offset, + shape.embd as u32, + draft, + shape.rms_epsilon, + ) + }, + "normalizing DSpark output rows", + )?; + matmul_rows( + &self.logits, + base_weights.output, + shape.embd, + shape.vocab, + &output_norm, + draft, + base.main.map_ptr().cast(), + base.main.len(), + )?; + commands.finish()?; + + let confidence = final_stage + .confidence + .ok_or("DSpark confidence head is missing")?; + let mut logits = vec![0.0; (u64::from(draft) * shape.vocab) as usize]; + let mut hidden = vec![0.0; (u64::from(draft) * shape.embd) as usize]; + self.logits.read_f32(&mut logits)?; + output_norm.read_f32(&mut hidden)?; + let mut proposals = Vec::with_capacity(draft as usize); + let mut previous = first_token as u32; + for row in 0..draft as usize { + let state = dense_row(support, markov_w1, previous)?; + let mut features = Vec::with_capacity(shape.embd as usize + state.len()); + features.extend_from_slice( + &hidden[row * shape.embd as usize..(row + 1) * shape.embd as usize], + ); + features.extend_from_slice(&state); + let confidence_logit = dense_dot(support, confidence, 0, &features)?; + let confidence_value = if confidence_logit >= 0.0 { + 1.0 / (1.0 + (-confidence_logit).exp()) + } else { + let value = confidence_logit.exp(); + value / (1.0 + value) + }; + if self.confidence_threshold > 0.0 && confidence_value < self.confidence_threshold { + break; + } + let row_logits = &logits[row * shape.vocab as usize..(row + 1) * shape.vocab as usize]; + let mut best = (0_i32, f32::NEG_INFINITY); + for (token, &logit) in row_logits.iter().enumerate() { + let value = logit + dense_dot(support, markov_w2, token as u32, &state)?; + if value > best.1 { + best = (token as i32, value); + } + } + proposals.push(best.0); + previous = best.0 as u32; + } + self.drafted += proposals.len() as u64; + Ok(proposals) + } +} + +struct Steering { + directions: Buffer, + attention_scale: f32, + ffn_scale: f32, + width: u32, + identity: [u8; 32], +} + +impl Steering { + fn load(model: &Model, settings: EngineSteeringSettings) -> Result, String> { + if settings.attention_scale == 0.0 && settings.ffn_scale == 0.0 { + return Ok(None); + } + let path = settings + .file + .as_deref() + .ok_or("directional steering needs a direction-vector file")?; + let bytes = fs::read(path) + .map_err(|error| format!("Cannot read directional steering file {path}: {error}"))?; + let expected = model + .shape + .layers + .checked_mul(model.shape.embd as u32) + .and_then(|values| values.checked_mul(4)) + .ok_or("directional steering size overflow")? as usize; + if bytes.len() != expected { + return Err(format!( + "directional steering file has {} bytes, expected {expected}", + bytes.len() + )); + } + let directions = Buffer::bytes(expected as u64)?; + directions.write(0, &bytes)?; + let mut hash = Sha256::new(); + hash.update(b"DS4Server directional steering v1"); + hash.update(settings.attention_scale.to_bits().to_le_bytes()); + hash.update(settings.ffn_scale.to_bits().to_le_bytes()); + hash.update(&bytes); + Ok(Some(Self { + directions, + attention_scale: settings.attention_scale, + ffn_scale: settings.ffn_scale, + width: model.shape.embd as u32, + identity: hash.finalize().into(), + })) + } + + fn apply(&self, output: &Buffer, layer: u32, rows: u32, attention: bool) -> Result<(), String> { + let scale = if attention { + self.attention_scale + } else { + self.ffn_scale + }; + if scale == 0.0 { + return Ok(()); + } + call( + unsafe { + ds4_gpu_directional_steering_project_tensor( + output.raw(), + self.directions.raw(), + layer, + self.width, + rows, + scale, + ) + }, + "applying directional steering", + ) + } +} + +struct SsdPlan { + model_spans: Vec<(u64, u64)>, + resident_bytes: u64, + cache_experts: u32, + per_expert_bytes: u64, + admission_bytes: u64, + preload_experts: u32, + cold: bool, + loader: Option, + selected_requests: std::sync::atomic::AtomicU64, + selected_experts: std::sync::atomic::AtomicU64, + selected_wait_ns: std::sync::atomic::AtomicU64, +} + +#[derive(Clone, Copy)] +struct SelectedLoadJob { + selected: usize, + count: u32, + event: u64, + model_map: usize, + model_size: u64, + layer: u32, + total_experts: u32, + gate_offset: u64, + up_offset: u64, + down_offset: u64, + gate_expert_bytes: u64, + down_expert_bytes: u64, +} + +struct SelectedLoadResult { + ids: Vec, + loaded: bool, + job: SelectedLoadJob, +} + +struct SelectedLoadWorker { + jobs: std::sync::mpsc::Sender>, + results: std::sync::mpsc::Receiver>, + thread: Option>, +} + +impl SelectedLoadWorker { + fn new() -> Self { + let (jobs, incoming) = std::sync::mpsc::channel::>(); + let (outgoing, results) = std::sync::mpsc::channel(); + let thread = std::thread::spawn(move || { + unsafe { ds4_gpu_stream_expert_cache_note_service_thread() }; + while let Ok(Some(job)) = incoming.recv() { + let result = load_selected_experts(job); + if outgoing.send(result).is_err() { + break; + } + } + }); + Self { + jobs, + results, + thread: Some(thread), + } + } + + fn begin(&self, selected: &Buffer, table: StreamExpertTable, count: u32) -> Result<(), String> { + let mut event = 0; + call( + unsafe { ds4_gpu_signal_selected_readback_ready(&mut event) }, + "signalling SSD expert selection", + )?; + self.jobs + .send(Some(SelectedLoadJob { + selected: selected.raw() as usize, + count, + event, + model_map: table.model_map as usize, + model_size: table.model_size, + layer: table.layer, + total_experts: table.total_experts, + gate_offset: table.gate_offset, + up_offset: table.up_offset, + down_offset: table.down_offset, + gate_expert_bytes: table.gate_expert_bytes, + down_expert_bytes: table.down_expert_bytes, + })) + .map_err(|_| "SSD expert loader stopped unexpectedly".to_string()) + } + + fn finish(&self, set_override: bool) -> Result<(), String> { + call( + unsafe { ds4_gpu_flush_commands() }, + "overlapping SSD expert loading", + )?; + let result = self + .results + .recv() + .map_err(|_| "SSD expert loader stopped unexpectedly".to_string())??; + if !result.loaded { + let table = result.job.table(); + call( + unsafe { + ds4_gpu_stream_expert_cache_begin_selected_load( + &table, + result.ids.as_ptr(), + result.ids.len() as u32, + ) + }, + "retrying selected SSD expert loading", + )?; + } + if set_override { + call( + unsafe { + ds4_gpu_routed_moe_set_selected_override( + result.ids.as_ptr(), + result.ids.len() as u32, + ) + }, + "selecting streamed experts", + )?; + } + Ok(()) + } +} + +impl Drop for SelectedLoadWorker { + fn drop(&mut self) { + let _ = self.jobs.send(None); + if let Some(thread) = self.thread.take() { + let _ = thread.join(); + } + } +} + +fn load_selected_experts(job: SelectedLoadJob) -> Result { + call( + unsafe { + ds4_gpu_wait_selected_readback_ready( + job.event, + c"selected-id async expert load".as_ptr(), + ) + }, + "waiting for SSD expert selection", + )?; + let mut ids = vec![0_i32; job.count as usize]; + call( + unsafe { + ds4_gpu_tensor_read( + job.selected as *const GpuTensor, + 0, + ids.as_mut_ptr().cast(), + u64::from(job.count) * 4, + ) + }, + "reading selected SSD experts", + )?; + if ids + .iter() + .any(|expert| *expert < 0 || *expert as u32 >= job.total_experts) + { + return Err(format!( + "SSD router selected an expert outside 0..{} at layer {}", + job.total_experts, job.layer + )); + } + let table = job.table(); + let loaded = unsafe { + ds4_gpu_stream_expert_cache_begin_selected_load(&table, ids.as_ptr(), ids.len() as u32) + } != 0; + Ok(SelectedLoadResult { ids, loaded, job }) +} + +impl SelectedLoadJob { + fn table(self) -> StreamExpertTable { + StreamExpertTable { + model_map: self.model_map as *const c_void, + model_size: self.model_size, + layer: self.layer, + total_experts: self.total_experts, + gate_offset: self.gate_offset, + up_offset: self.up_offset, + down_offset: self.down_offset, + gate_expert_bytes: self.gate_expert_bytes, + down_expert_bytes: self.down_expert_bytes, + } + } +} + impl Weights { fn bind(model: &Model) -> Result { if model.shape.family != ModelFamily::DeepSeek { @@ -264,6 +1850,339 @@ impl Weights { } } +impl SsdPlan { + fn new( + model: &Model, + weights: &Weights, + settings: EngineSsdSettings, + context: u32, + prefill_chunk: u32, + ) -> Result { + let first = weights + .layers + .first() + .ok_or("model has no routed expert layers")?; + let gate = first.expert_gate.bytes / model.shape.experts; + let down = first.expert_down.bytes / model.shape.experts; + let per_expert_bytes = gate + .checked_mul(2) + .and_then(|bytes| bytes.checked_add(down)) + .ok_or("routed expert size overflow")?; + if gate == 0 || down == 0 { + return Err("routed expert tensors have an invalid layout".into()); + } + let model_spans = streaming_model_spans(model, weights, per_expert_bytes)?; + let resident_bytes = model_spans.iter().try_fold(0_u64, |total, (_, bytes)| { + total + .checked_add(*bytes) + .ok_or("resident weight size overflow") + })?; + let runtime_bytes = estimated_deepseek_runtime_bytes(model.shape, context, prefill_chunk); + let recommended = unsafe { ds4_gpu_recommended_working_set_size() }; + if recommended == 0 && settings.cache_experts == 0 && settings.cache_bytes == 0 { + return Err( + "Metal did not report a working-set size; set an explicit SSD cache budget".into(), + ); + } + let prefill_headroom = weights + .layers + .iter() + .map(|layer| { + layer + .expert_gate + .bytes + .saturating_add(layer.expert_up.bytes) + .saturating_add(layer.expert_down.bytes) + }) + .max() + .unwrap_or(0) + .saturating_mul(2); + let max_experts = model + .shape + .layers + .saturating_mul(model.shape.experts as u32); + let (cache_experts, reserved_headroom) = if settings.cache_experts != 0 { + (settings.cache_experts.min(max_experts), 0) + } else { + let total = if settings.cache_bytes != 0 { + const GIB: u64 = 1024 * 1024 * 1024; + if recommended == 0 { + settings.cache_bytes + } else { + let safe = recommended + .saturating_mul(7) + .checked_div(8) + .unwrap_or(recommended) + .saturating_sub(runtime_bytes) + / GIB + * GIB; + settings.cache_bytes.min(safe.max(GIB)) + } + } else { + let percent = env::var("DS4_SSD_AUTO_CACHE_PCT") + .ok() + .and_then(|value| value.parse::().ok()) + .filter(|value| (50..=95).contains(value)) + .unwrap_or(80); + recommended + .saturating_mul(percent) + .checked_div(100) + .unwrap_or(recommended) + .saturating_sub(resident_bytes) + }; + if total <= prefill_headroom { + return Err(format!( + "SSD cache budget is too small: two routed prefill layers need {:.2} GiB", + prefill_headroom as f64 / 1_073_741_824.0 + )); + } + ( + u32::try_from((total - prefill_headroom) / per_expert_bytes) + .unwrap_or(u32::MAX) + .min(max_experts), + prefill_headroom, + ) + }; + if cache_experts == 0 { + return Err("SSD streaming has no memory for an expert cache".into()); + } + let cache_bytes = per_expert_bytes.saturating_mul(u64::from(cache_experts)); + let admission_bytes = resident_bytes + .saturating_add(runtime_bytes) + .saturating_add(cache_bytes) + .saturating_add(reserved_headroom); + Ok(Self { + model_spans, + resident_bytes, + cache_experts, + per_expert_bytes, + admission_bytes, + preload_experts: if settings.cold { + 0 + } else if settings.preload_experts != 0 { + settings.preload_experts.min(cache_experts) + } else { + let cap = env::var("DS4_METAL_STREAMING_EXPERT_AUTO_PRELOAD_CAP") + .ok() + .and_then(|value| value.parse::().ok()) + .unwrap_or(4096); + if cap == 0 { + cache_experts + } else { + cache_experts.min(cap) + } + }, + cold: settings.cold, + loader: None, + selected_requests: std::sync::atomic::AtomicU64::new(0), + selected_experts: std::sync::atomic::AtomicU64::new(0), + selected_wait_ns: std::sync::atomic::AtomicU64::new(0), + }) + } + + fn configure(&mut self, model: &Model, weights: &Weights) -> Result<(), String> { + unsafe { + ds4_gpu_set_streaming_expert_cache_expert_bytes(self.per_expert_bytes); + ds4_gpu_set_streaming_expert_cache_budget(self.cache_experts); + } + if self.cold || self.preload_experts == 0 { + self.loader = Some(SelectedLoadWorker::new()); + return Ok(()); + } + let mut by_layer = vec![Vec::<(i32, u32)>::new(); model.shape.layers as usize]; + let mut loaded = 0_u32; + for &(layer, expert) in hotlist::FLASH { + if loaded == self.preload_experts { + break; + } + if u32::from(layer) >= model.shape.layers || u64::from(expert) >= model.shape.experts { + continue; + } + by_layer[layer as usize].push((i32::from(expert), self.preload_experts - loaded)); + loaded += 1; + } + for (layer_index, entries) in by_layer.iter().enumerate() { + if entries.is_empty() { + continue; + } + let ids = entries.iter().map(|entry| entry.0).collect::>(); + let priorities = entries.iter().map(|entry| entry.1).collect::>(); + let count = ids.len() as u32; + let layer = &weights.layers[layer_index]; + let table = stream_expert_table(model, layer, layer_index as u32, model.shape.experts); + call( + unsafe { + ds4_gpu_stream_expert_cache_seed_experts( + &table, + ids.as_ptr(), + priorities.as_ptr(), + count, + ) + }, + "preloading SSD experts", + )?; + } + self.loader = Some(SelectedLoadWorker::new()); + Ok(()) + } + + fn begin_selected( + &self, + selected: &Buffer, + table: StreamExpertTable, + count: u32, + ) -> Result<(), String> { + self.selected_requests + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.selected_experts + .fetch_add(u64::from(count), std::sync::atomic::Ordering::Relaxed); + self.loader + .as_ref() + .ok_or("SSD expert loader was not initialized")? + .begin(selected, table, count) + } + + fn finish_selected(&self, set_override: bool) -> Result<(), String> { + let started = Instant::now(); + let result = self + .loader + .as_ref() + .ok_or("SSD expert loader was not initialized")? + .finish(set_override); + self.selected_wait_ns.fetch_add( + u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX), + std::sync::atomic::Ordering::Relaxed, + ); + result + } +} + +fn stream_expert_table( + model: &Model, + layer: &Layer, + index: u32, + experts: u64, +) -> StreamExpertTable { + StreamExpertTable { + model_map: model.main.map_ptr().cast(), + model_size: model.main.len(), + layer: index, + total_experts: experts as u32, + gate_offset: layer.expert_gate.offset, + up_offset: layer.expert_up.offset, + down_offset: layer.expert_down.offset, + gate_expert_bytes: layer.expert_gate.bytes / experts, + down_expert_bytes: layer.expert_down.bytes / experts, + } +} + +fn streaming_model_spans( + model: &Model, + weights: &Weights, + slab_bytes: u64, +) -> Result, String> { + let mut streamed = Vec::new(); + for layer in &weights.layers { + let bytes = (layer.expert_gate.bytes + layer.expert_up.bytes + layer.expert_down.bytes) + / model.shape.experts; + if bytes == slab_bytes { + streamed.extend([ + (layer.expert_gate.offset, layer.expert_gate.bytes), + (layer.expert_up.offset, layer.expert_up.bytes), + (layer.expert_down.offset, layer.expert_down.bytes), + ]); + } + } + let mut spans = model + .main + .tensors + .values() + .filter(|tensor| { + !streamed + .iter() + .any(|&(offset, bytes)| tensor.offset == offset && tensor.bytes == bytes) + }) + .map(|tensor| (tensor.offset, tensor.bytes)) + .collect::>(); + spans.sort_unstable_by_key(|span| span.0); + let mut merged: Vec<(u64, u64)> = Vec::new(); + for (offset, bytes) in spans { + let end = offset.checked_add(bytes).ok_or("model span overflow")?; + if let Some((previous_offset, previous_bytes)) = merged.last_mut() { + let previous_end = previous_offset.saturating_add(*previous_bytes); + if offset <= previous_end { + *previous_bytes = previous_end.max(end) - *previous_offset; + continue; + } + } + merged.push((offset, bytes)); + } + if merged.is_empty() { + return Err("SSD streaming found no resident model tensors".into()); + } + Ok(merged) +} + +fn estimated_deepseek_runtime_bytes(shape: super::Shape, context: u32, prefill: u32) -> u64 { + let prefill_cap = effective_prefill_cap(context, prefill); + let raw_cap = effective_raw_cap(shape, context, prefill_cap); + let raw = u64::from(shape.layers) + .saturating_mul(raw_cap.into()) + .saturating_mul(shape.head_dim) + .saturating_mul(4); + let compressed = (0..shape.layers).fold(0_u64, |total, layer| { + let ratio = compression_ratio(shape, layer); + if ratio == 0 { + return total; + } + let rows = u64::from(context / ratio + 2); + total + .saturating_add(rows.saturating_mul(shape.head_dim).saturating_mul(2)) + .saturating_add(if ratio == 4 { + rows.saturating_mul(shape.indexer_head_dim) + .saturating_mul(4) + } else { + 0 + }) + }); + let min_ratio = (0..shape.layers) + .map(|layer| compression_ratio(shape, layer)) + .filter(|ratio| *ratio != 0) + .min() + .unwrap_or(context.max(1)); + let comp_cap = context / min_ratio + 2; + let attention_stage = u64::from(prefill_cap / min_ratio + 2) + .max(2) + .saturating_mul(shape.head_dim) + .saturating_mul(4); + let scratch = 2_u64 + .saturating_mul(comp_cap.into()) + .saturating_mul(prefill_cap.into()) + .saturating_mul(4) + .saturating_add(attention_stage); + raw.saturating_add(compressed).saturating_add(scratch) +} + +fn effective_prefill_cap(context: u32, requested: u32) -> u32 { + if requested == 0 { + context.clamp(1, DEFAULT_PREFILL_CHUNK) + } else { + context.min(requested).max(1) + } +} + +fn effective_raw_cap(shape: super::Shape, context: u32, prefill_cap: u32) -> u32 { + let window = (shape.sliding_window as u32).min(context).max(1); + let wanted = u64::from(window) + .saturating_add(prefill_cap.into()) + .min(context.into()) + .max(1) + .div_ceil(256) + .saturating_mul(256) + .min(8192); + u32::try_from(wanted).unwrap_or(8192).max(window) +} + struct Scratch { current_hc: Buffer, next_hc: Buffer, @@ -485,6 +2404,29 @@ struct CompressionState { rows: u32, } +struct CompressionFrontier { + state_kv: Buffer, + state_score: Buffer, + bytes: u64, + rows: u32, +} + +struct LayerFrontier { + compression: Option, + indexer: Option, +} + +struct SpecFrontier { + layers: Vec, + position: u32, + token_len: usize, + logits: Vec, + dspark_target_hidden: Option, + dspark_capture_mask: u32, + dspark_cache_start: u32, + dspark_cache_len: u32, +} + impl LayerState { fn allocate(model: &Model, index: u32, context: u32, raw_cap: u32) -> Result { let shape = model.shape; @@ -537,15 +2479,8 @@ impl Session { if context == 0 { return Err("context must contain at least one token".into()); } - let prefill_cap = prefill_chunk.max(1).min(context); - let raw_cap = prefill_cap - .saturating_add(model.shape.sliding_window as u32) - .min(context) - .div_ceil(256) - .saturating_mul(256) - .min(8192) - .min(context) - .max(1); + let prefill_cap = effective_prefill_cap(context, prefill_chunk); + let raw_cap = effective_raw_cap(model.shape, context, prefill_cap); let layers = (0..model.shape.layers) .map(|index| LayerState::allocate(model, index, context, raw_cap)) .collect::>()?; @@ -565,12 +2500,40 @@ impl Session { // and `_context` must drop before `model` unmaps memory wrapped without copying // by native/metal/ds4_metal.m:10329. This intentionally differs from // ../ds4/ds4.c:56287-56288; do not reorder these fields to match it. +#[derive(Clone, Copy, Default)] +pub(super) struct ExecutionStats { + pub(super) speculative_mode: u8, + pub(super) speculative_cycles: u64, + pub(super) drafted_tokens: u64, + pub(super) accepted_draft_tokens: u64, + pub(super) verifier_passes: u64, + pub(super) verifier_ms: u64, + pub(super) ssd_enabled: bool, + pub(super) ssd_resident_bytes: u64, + pub(super) ssd_cache_bytes: u64, + pub(super) ssd_cache_experts: u64, + pub(super) ssd_preloaded_experts: u64, + pub(super) ssd_selected_requests: u64, + pub(super) ssd_requested_bytes: u64, + pub(super) ssd_wait_ms: u64, +} + pub(super) struct DeepSeekExecutor { weights: Weights, session: Session, + legacy_mtp: Option, + dspark: Option, + steering: Option, + ssd: Option, logits: Vec, tokens: Vec, quality: bool, + power_percent: u8, + prefill_layer_average: Vec, + decode_average: f64, + speculative_cycles: u64, + verifier_passes: u64, + verifier_ns: u64, checkpoint_tag: [u8; 32], model_modified: (u64, u32), model_identity: [u8; 32], @@ -579,14 +2542,26 @@ pub(super) struct DeepSeekExecutor { } impl DeepSeekExecutor { + #[allow(clippy::too_many_arguments)] pub(super) fn open( model: Model, context: u32, quality: bool, prefill_chunk: u32, + power_percent: u8, + speculative: EngineSpeculativeSettings, + ssd: EngineSsdSettings, + steering: EngineSteeringSettings, ) -> Result { let weights = Weights::bind(&model)?; - let context_handle = Context::open(&model, quality, false, 0)?; + let mut ssd_plan = ssd + .enabled + .then(|| SsdPlan::new(&model, &weights, ssd, context, prefill_chunk)) + .transpose()?; + let spans = ssd_plan.as_ref().map(|plan| plan.model_spans.as_slice()); + let admission = ssd_plan.as_ref().map_or(0, |plan| plan.admission_bytes); + let context_handle = Context::open(&model, quality, ssd.enabled, admission, spans)?; + let steering = Steering::load(&model, steering)?; let session = Session::new( &model, context, @@ -596,19 +2571,75 @@ impl DeepSeekExecutor { prefill_chunk }, )?; + let legacy_mtp = match (model.support_kind, model.support.as_ref()) { + (Some(SupportKind::LegacyMtp), Some(support)) => { + let hc_dim = model.shape.hc * model.shape.embd; + Some(LegacyMtp { + weights: LegacyMtpWeights::bind(support, model.shape)?, + layer: LayerState::allocate(&model, 1, context, session.raw_cap)?, + state_hc: Buffer::floats(hc_dim)?, + next_hc: Buffer::floats(hc_dim)?, + draft_token: None, + raw_rows: 0, + draft_limit: speculative.mtp_draft_tokens.max(1) as u32, + margin: speculative.mtp_margin, + drafted: 0, + accepted: 0, + }) + } + _ => None, + }; + let dspark = match ( + model.support_kind, + model.support.as_ref(), + speculative.dspark, + ) { + (Some(SupportKind::DSpark), Some(support), true) => Some(Dspark::new( + &model, + support, + &session, + speculative, + quality, + )?), + _ => None, + }; + if let Some(plan) = &mut ssd_plan { + plan.configure(&model, &weights)?; + } let model_modified = fs::metadata(model.main.path()) .and_then(|metadata| metadata.modified()) .ok() .and_then(|modified| modified.duration_since(UNIX_EPOCH).ok()) .map(|duration| (duration.as_secs(), duration.subsec_nanos())) .unwrap_or_default(); - let model_identity = model.checkpoint_identity(); + let model_identity = if let Some(steering) = &steering { + let mut hash = Sha256::new(); + hash.update(model.checkpoint_identity()); + hash.update(steering.identity); + hash.finalize().into() + } else { + model.checkpoint_identity() + }; Ok(Self { weights, session, + legacy_mtp, + dspark, + steering, + ssd: ssd_plan, logits: vec![0.0; model.shape.vocab as usize], tokens: Vec::new(), quality, + power_percent: if power_percent == 0 { + 100 + } else { + power_percent + }, + prefill_layer_average: vec![0.0; model.shape.layers as usize], + decode_average: 0.0, + speculative_cycles: 0, + verifier_passes: 0, + verifier_ns: 0, checkpoint_tag: [0; 32], model_modified, model_identity, @@ -618,6 +2649,19 @@ impl DeepSeekExecutor { } pub(super) fn eval(&mut self, token: i32) -> Result<(), String> { + self.eval_target(token)?; + if let (Some(dspark), Some(support)) = (&mut self.dspark, self.model.support.as_ref()) { + dspark.seed_current_cache( + support, + self.session.position - 1, + self.session.raw_cap, + self.model.shape, + )?; + } + Ok(()) + } + + fn eval_target(&mut self, token: i32) -> Result<(), String> { if token < 0 || token as u64 >= self.model.shape.vocab { return Err(format!("token {token} is outside the vocabulary")); } @@ -627,20 +2671,554 @@ impl DeepSeekExecutor { self.session.context )); } + let started = Instant::now(); + if let Some(dspark) = &mut self.dspark { + dspark.begin_capture(); + } let commands = Commands::begin()?; self.encode_token(token as u32)?; commands.finish()?; self.session.scratch.logits.read_f32(&mut self.logits)?; self.session.position += 1; self.tokens.push(token); + throttle( + &mut self.decode_average, + started.elapsed(), + self.power_percent, + ); Ok(()) } + fn snapshot_spec_frontier(&self) -> Result { + fn snapshot(state: &CompressionState, bytes: u64) -> Result { + let state_kv = Buffer::bytes(bytes)?; + let state_score = Buffer::bytes(bytes)?; + call( + unsafe { ds4_gpu_tensor_copy(state_kv.raw(), 0, state.state_kv.raw(), 0, bytes) }, + "saving speculative compressor KV state", + )?; + call( + unsafe { + ds4_gpu_tensor_copy(state_score.raw(), 0, state.state_score.raw(), 0, bytes) + }, + "saving speculative compressor score state", + )?; + Ok(CompressionFrontier { + state_kv, + state_score, + bytes, + rows: state.rows, + }) + } + + let shape = self.model.shape; + let commands = Commands::begin()?; + let layers = self + .session + .layers + .iter() + .map(|layer| { + let compression = layer + .compression + .as_ref() + .map(|state| { + let coefficient = if state.ratio == 4 { 2 } else { 1 }; + snapshot( + state, + coefficient * coefficient * state.ratio as u64 * shape.head_dim * 4, + ) + }) + .transpose()?; + let indexer = layer + .indexer + .as_ref() + .map(|state| { + snapshot(state, 4 * state.ratio as u64 * shape.indexer_head_dim * 4) + }) + .transpose()?; + Ok::<_, String>(LayerFrontier { + compression, + indexer, + }) + }) + .collect::, _>>()?; + let dspark_target_hidden = self + .dspark + .as_ref() + .map(|dspark| { + let bytes = dspark.config.target_layers.len() as u64 * shape.embd * 4; + let saved = Buffer::bytes(bytes)?; + call( + unsafe { + ds4_gpu_tensor_copy(saved.raw(), 0, dspark.target_hidden.raw(), 0, bytes) + }, + "saving speculative DSpark target state", + )?; + Ok::<_, String>(saved) + }) + .transpose()?; + commands.finish()?; + Ok(SpecFrontier { + layers, + position: self.session.position, + token_len: self.tokens.len(), + logits: self.logits.clone(), + dspark_target_hidden, + dspark_capture_mask: self.dspark.as_ref().map_or(0, |value| value.capture_mask), + dspark_cache_start: self.dspark.as_ref().map_or(0, |value| value.cache_start), + dspark_cache_len: self.dspark.as_ref().map_or(0, |value| value.cache_len), + }) + } + + fn restore_spec_frontier(&mut self, frontier: &SpecFrontier) -> Result<(), String> { + fn restore( + state: &mut CompressionState, + saved: &CompressionFrontier, + ) -> Result<(), String> { + call( + unsafe { + ds4_gpu_tensor_copy( + state.state_kv.raw(), + 0, + saved.state_kv.raw(), + 0, + saved.bytes, + ) + }, + "restoring speculative compressor KV state", + )?; + call( + unsafe { + ds4_gpu_tensor_copy( + state.state_score.raw(), + 0, + saved.state_score.raw(), + 0, + saved.bytes, + ) + }, + "restoring speculative compressor score state", + )?; + state.rows = saved.rows; + Ok(()) + } + + if frontier.layers.len() != self.session.layers.len() { + return Err("speculative frontier layer count changed".into()); + } + let commands = Commands::begin()?; + for (layer, saved) in self.session.layers.iter_mut().zip(&frontier.layers) { + match (&mut layer.compression, &saved.compression) { + (Some(state), Some(saved)) => restore(state, saved)?, + (None, None) => {} + _ => return Err("speculative compressor layout changed".into()), + } + match (&mut layer.indexer, &saved.indexer) { + (Some(state), Some(saved)) => restore(state, saved)?, + (None, None) => {} + _ => return Err("speculative indexer layout changed".into()), + } + } + if let (Some(dspark), Some(saved)) = (&mut self.dspark, &frontier.dspark_target_hidden) { + let bytes = dspark.config.target_layers.len() as u64 * self.model.shape.embd * 4; + call( + unsafe { + ds4_gpu_tensor_copy(dspark.target_hidden.raw(), 0, saved.raw(), 0, bytes) + }, + "restoring speculative DSpark target state", + )?; + dspark.capture_mask = frontier.dspark_capture_mask; + dspark.cache_start = frontier.dspark_cache_start; + dspark.cache_len = frontier.dspark_cache_len; + } + commands.finish()?; + self.session.position = frontier.position; + self.tokens.truncate(frontier.token_len); + self.logits.clone_from(&frontier.logits); + Ok(()) + } + + fn verify_target_suffix( + &mut self, + proposals: &[i32], + cancelled: &std::sync::atomic::AtomicBool, + ) -> Result, String> { + if proposals.is_empty() + || argmax(&self.logits) != proposals[0] + || cancelled.load(std::sync::atomic::Ordering::Relaxed) + { + return Ok(Vec::new()); + } + let started = Instant::now(); + if self.quality + || proposals.len() == 1 + || self + .ssd + .as_ref() + .is_some_and(|ssd| u64::from(ssd.cache_experts) < self.model.shape.experts) + { + let mut accepted = Vec::new(); + for &proposal in proposals { + if argmax(&self.logits) != proposal + || cancelled.load(std::sync::atomic::Ordering::Relaxed) + { + break; + } + self.eval_target(proposal)?; + self.verifier_passes += 1; + accepted.push(proposal); + } + self.verifier_ns = self + .verifier_ns + .saturating_add(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + return Ok(accepted); + } + + let frontier = self.snapshot_spec_frontier()?; + let row_tops = match self.eval_batch_tops(proposals) { + Ok(tops) => { + self.verifier_passes += 1; + tops + } + Err(error) => { + self.restore_spec_frontier(&frontier)?; + self.verifier_ns = self.verifier_ns.saturating_add( + u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX), + ); + return Err(error); + } + }; + let mut commit = 1_usize; + while commit < proposals.len() && row_tops[commit - 1] == proposals[commit] { + commit += 1; + } + if commit == proposals.len() { + self.verifier_ns = self + .verifier_ns + .saturating_add(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + return Ok(proposals.to_vec()); + } + + self.restore_spec_frontier(&frontier)?; + if let Some(dspark) = &mut self.dspark { + dspark.commit_proposed_prefix(1, self.session.raw_cap); + } + for (index, &proposal) in proposals[..commit].iter().enumerate() { + if index == 0 && self.dspark.is_some() { + self.eval_target(proposal)?; + } else { + self.eval(proposal)?; + } + self.verifier_passes += 1; + } + self.verifier_ns = self + .verifier_ns + .saturating_add(u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX)); + Ok(proposals[..commit].to_vec()) + } + + pub(super) fn eval_speculative_greedy( + &mut self, + first_token: i32, + max_tokens: u32, + reasoning: ReasoningMode, + cancelled: &std::sync::atomic::AtomicBool, + ) -> Result, String> { + if self.dspark.is_some() || self.legacy_mtp.is_some() { + self.speculative_cycles += 1; + } + self.eval_target(first_token)?; + let mut accepted = vec![first_token]; + if self.dspark.as_ref().is_some_and(|dspark| dspark.strict) { + return Ok(accepted); + } + if self.dspark.is_some() { + let mut dspark = self.dspark.take().expect("DSpark disappeared"); + let proposals = dspark.propose( + &self.model, + &self.weights, + first_token, + self.session.position.saturating_sub(1), + self.session.raw_cap, + ); + self.dspark = Some(dspark); + let mut proposals = proposals?; + proposals.truncate( + max_tokens + .saturating_sub(1) + .min(self.session.context.saturating_sub(self.session.position)) + as usize, + ); + if let Some(stop) = proposals + .iter() + .position(|token| self.model.is_stop_token_for_reasoning(*token, reasoning)) + { + proposals.truncate(stop + 1); + } + let verified = self.verify_target_suffix(&proposals, cancelled)?; + if verified.is_empty() { + self.dspark + .as_mut() + .expect("DSpark disappeared") + .commit_proposed_prefix(1, self.session.raw_cap); + } + accepted.extend_from_slice(&verified); + self.dspark.as_mut().expect("DSpark disappeared").accepted += verified.len() as u64; + return Ok(accepted); + } + let Some(draft_cap) = self.legacy_mtp.as_ref().map(|mtp| { + mtp.draft_limit + .min(max_tokens.saturating_sub(1)) + .min(self.session.context.saturating_sub(self.session.position)) + }) else { + return Ok(accepted); + }; + if self + .legacy_mtp + .as_ref() + .is_some_and(|mtp| mtp.draft_limit <= 1) + || draft_cap == 0 + || cancelled.load(std::sync::atomic::Ordering::Relaxed) + { + return Ok(accepted); + } + + let (first_proposal, _) = self.legacy_mtp_draft(first_token, true)?; + let mtp_base_raw = self + .legacy_mtp + .as_ref() + .expect("legacy MTP disappeared") + .raw_rows; + if argmax(&self.logits) != first_proposal { + return Ok(accepted); + } + let mut proposals = vec![first_proposal]; + let mut last_margin = f32::INFINITY; + while proposals.len() < draft_cap as usize + && !self.model.is_stop_token_for_reasoning( + *proposals.last().expect("MTP proposal disappeared"), + reasoning, + ) + && !cancelled.load(std::sync::atomic::Ordering::Relaxed) + { + let (proposal, margin) = + self.legacy_mtp_draft(*proposals.last().expect("MTP proposal disappeared"), false)?; + proposals.push(proposal); + last_margin = margin; + } + if proposals.len() == 2 + && self + .legacy_mtp + .as_ref() + .is_some_and(|mtp| mtp.margin > 0.0 && last_margin < mtp.margin) + { + proposals.truncate(1); + } + let verified = self.verify_target_suffix(&proposals, cancelled)?; + accepted.extend_from_slice(&verified); + let mtp = self.legacy_mtp.as_mut().expect("legacy MTP disappeared"); + mtp.raw_rows = mtp_base_raw + .saturating_add(verified.len() as u32) + .min(self.model.shape.sliding_window as u32) + .min(self.session.raw_cap); + mtp.accepted += verified.len() as u64; + mtp.draft_token = None; + Ok(accepted) + } + + fn legacy_mtp_draft(&mut self, token: i32, target_hc: bool) -> Result<(i32, f32), String> { + let support = self + .model + .support + .as_ref() + .ok_or("legacy MTP support model is missing")?; + let mtp = self + .legacy_mtp + .as_mut() + .ok_or("legacy MTP is not configured")?; + let shape = self.model.shape; + let base_map = self.model.main.map_ptr().cast(); + let base_size = self.model.main.len(); + let support_map = support.map_ptr().cast(); + let support_size = support.len(); + let scratch = &self.session.scratch; + let draft_pos = if target_hc { + self.session.position.saturating_sub(1) + } else { + self.session.position + }; + let prev = if target_hc { + scratch.current_hc.raw().cast_const() + } else { + mtp.state_hc.raw().cast_const() + }; + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_embed_token_hc_tensor( + scratch.current.raw(), + base_map, + base_size, + self.weights.token_embedding.offset, + shape.vocab as u32, + token as u32, + shape.embd as u32, + 1, + ) + }, + "MTP token embedding", + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_tensor( + scratch.norm.raw(), + scratch.current.raw(), + support_map, + support_size, + mtp.weights.enorm.offset, + shape.embd as u32, + shape.rms_epsilon, + ) + }, + "MTP embedding norm", + )?; + q8( + &scratch.attention_out, + mtp.weights.e_proj, + shape.embd, + shape.embd, + &scratch.norm, + support_map, + support_size, + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_rows_tensor( + scratch.flat_hc.raw(), + prev, + support_map, + support_size, + mtp.weights.hnorm.offset, + shape.embd as u32, + shape.hc as u32, + shape.rms_epsilon, + ) + }, + "MTP hidden norm", + )?; + q8_rows( + &scratch.next_hc, + mtp.weights.h_proj, + shape.embd, + shape.embd, + &scratch.flat_hc, + shape.hc as u32, + support_map, + support_size, + )?; + call( + unsafe { + ds4_gpu_repeat_hc_tensor( + scratch.current_hc.raw(), + scratch.attention_out.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "MTP embedding repeat", + )?; + call( + unsafe { + ds4_gpu_add_tensor( + scratch.current_hc.raw(), + scratch.current_hc.raw(), + scratch.next_hc.raw(), + (shape.hc * shape.embd) as u32, + ) + }, + "MTP input projection", + )?; + encode_layer_with_cache_rows( + scratch, + &mut mtp.layer, + &mtp.weights.block, + shape, + support_map, + support_size, + 1, + draft_pos, + token as u32, + self.session.raw_cap, + None, + None, + Some(mtp.raw_rows), + )?; + call( + unsafe { + ds4_gpu_tensor_copy( + mtp.next_hc.raw(), + 0, + scratch.current_hc.raw(), + 0, + shape.hc * shape.embd * 4, + ) + }, + "MTP hidden-state capture", + )?; + encode_mtp_output( + scratch, + &mtp.weights, + self.weights.output, + shape, + support_map, + support_size, + base_map, + base_size, + )?; + commands.finish()?; + let mut logits = vec![0.0_f32; shape.vocab as usize]; + scratch.logits.read_f32(&mut logits)?; + let (top, margin) = top2(&logits); + std::mem::swap(&mut mtp.state_hc, &mut mtp.next_hc); + mtp.raw_rows = (mtp.raw_rows + 1) + .min(shape.sliding_window as u32) + .min(self.session.raw_cap); + mtp.draft_token = Some(top); + mtp.drafted += 1; + Ok((top, margin)) + } + pub(super) fn prefill( &mut self, tokens: &[i32], mut progress: impl FnMut(u32) -> bool, ) -> Result { + if self.session.position == 0 && self.ssd.is_some() { + unsafe { ds4_gpu_stream_expert_cache_reset_route_hotness() }; + } + let streaming_decode_cap = if self.model.shape.model == ModelChoice::DeepSeekV4Flash + && self.weights.layers.first().is_some_and(|layer| { + layer.expert_gate.kind == Q4_K + && layer.expert_up.kind == Q4_K + && layer.expert_down.kind == Q4_K + }) { + 64 + } else { + 18 + }; + if self.ssd.is_some() + && !self.quality + && !tokens.is_empty() + && tokens.len() <= streaming_decode_cap + { + for (index, &token) in tokens.iter().enumerate() { + if !progress(self.session.position) { + return Ok(index); + } + self.eval(token)?; + } + return Ok(tokens.len()); + } if tokens.len() < 4 && self.session.position != 0 { for (index, &token) in tokens.iter().enumerate() { self.eval(token)?; @@ -687,6 +3265,14 @@ impl DeepSeekExecutor { } fn eval_batch(&mut self, tokens: &[i32]) -> Result<(), String> { + self.eval_batch_inner(tokens, false).map(|_| ()) + } + + fn eval_batch_tops(&mut self, tokens: &[i32]) -> Result, String> { + self.eval_batch_inner(tokens, true) + } + + fn eval_batch_inner(&mut self, tokens: &[i32], collect_tops: bool) -> Result, String> { let rows = u32::try_from(tokens.len()).map_err(|_| "prefill batch is too large")?; if rows == 0 || rows > self.session.prefill_cap { return Err("prefill batch exceeds the configured prefill workspace".into()); @@ -698,6 +3284,9 @@ impl DeepSeekExecutor { return Err("prefill contains a token outside the vocabulary".into()); } let mut batch = BatchScratch::allocate(&self.model, self.session.position, rows)?; + if let Some(dspark) = &mut self.dspark { + dspark.begin_capture(); + } batch.tokens.write_i32(tokens)?; let map = self.model.main.map_ptr().cast(); let size = self.model.main.len(); @@ -730,6 +3319,7 @@ impl DeepSeekExecutor { .zip(&mut self.session.layers) .enumerate() { + let started = Instant::now(); let commands = Commands::begin()?; encode_batch_layer( &batch, @@ -742,36 +3332,110 @@ impl DeepSeekExecutor { pos, rows, self.session.raw_cap, + self.steering.as_ref(), + self.ssd.as_ref(), )?; + if let Some(dspark) = &mut self.dspark { + dspark.capture_batch( + index as u32, + &batch.next_hc, + rows, + self.session.prefill_cap, + shape, + )?; + } commands.finish()?; + throttle( + &mut self.prefill_layer_average[index], + started.elapsed(), + self.power_percent, + ); std::mem::swap(&mut batch.current_hc, &mut batch.next_hc); } - let commands = Commands::begin()?; - call( - unsafe { - ds4_gpu_tensor_copy( - self.session.scratch.current_hc.raw(), - 0, - batch.current_hc.raw(), - u64::from(rows - 1) * shape.hc * shape.embd * 4, - shape.hc * shape.embd * 4, - ) - }, - "selecting the final prefill row", - )?; - encode_output(&self.session.scratch, &self.weights, shape, map, size)?; - commands.finish()?; - self.session.scratch.logits.read_f32(&mut self.logits)?; + if let (Some(dspark), Some(support)) = (&mut self.dspark, self.model.support.as_ref()) { + dspark.seed_batch_cache( + support, + pos, + rows, + self.session.prefill_cap, + self.session.raw_cap, + shape, + )?; + } + + let output_rows = if collect_tops { rows } else { 1 }; + let first_output = rows - output_rows; + let mut tops = Vec::with_capacity(output_rows as usize); + for row in first_output..rows { + let commands = Commands::begin()?; + call( + unsafe { + ds4_gpu_tensor_copy( + self.session.scratch.current_hc.raw(), + 0, + batch.current_hc.raw(), + u64::from(row) * shape.hc * shape.embd * 4, + shape.hc * shape.embd * 4, + ) + }, + "selecting a prefill output row", + )?; + encode_output(&self.session.scratch, &self.weights, shape, map, size)?; + commands.finish()?; + self.session.scratch.logits.read_f32(&mut self.logits)?; + tops.push(argmax(&self.logits)); + } self.session.position += rows; self.tokens.extend_from_slice(tokens); - Ok(()) + Ok(tops) } pub(super) fn logits(&self) -> &[f32] { &self.logits } + fn execution_stats(&self) -> ExecutionStats { + let (speculative_mode, drafted_tokens, accepted_draft_tokens) = + if let Some(dspark) = &self.dspark { + (2, dspark.drafted, dspark.accepted) + } else if let Some(mtp) = &self.legacy_mtp { + (1, mtp.drafted, mtp.accepted) + } else { + (0, 0, 0) + }; + let mut stats = ExecutionStats { + speculative_mode, + speculative_cycles: self.speculative_cycles, + drafted_tokens, + accepted_draft_tokens, + verifier_passes: self.verifier_passes, + verifier_ms: self.verifier_ns / 1_000_000, + ..ExecutionStats::default() + }; + if let Some(ssd) = &self.ssd { + let selected = ssd + .selected_experts + .load(std::sync::atomic::Ordering::Relaxed); + stats.ssd_enabled = true; + stats.ssd_resident_bytes = ssd.resident_bytes; + stats.ssd_cache_experts = u64::from(ssd.cache_experts); + stats.ssd_cache_bytes = ssd + .per_expert_bytes + .saturating_mul(u64::from(ssd.cache_experts)); + stats.ssd_preloaded_experts = u64::from(ssd.preload_experts); + stats.ssd_selected_requests = ssd + .selected_requests + .load(std::sync::atomic::Ordering::Relaxed); + stats.ssd_requested_bytes = ssd.per_expert_bytes.saturating_mul(selected); + stats.ssd_wait_ms = ssd + .selected_wait_ns + .load(std::sync::atomic::Ordering::Relaxed) + / 1_000_000; + } + stats + } + pub(super) fn model(&self) -> &Model { &self.model } @@ -786,6 +3450,17 @@ impl DeepSeekExecutor { pub(super) fn reset(&mut self) -> Result<(), String> { self.session = Session::new(&self.model, self.session.context, self.session.prefill_cap)?; + if let Some(mtp) = &mut self.legacy_mtp { + mtp.layer = + LayerState::allocate(&self.model, 1, self.session.context, self.session.raw_cap)?; + mtp.draft_token = None; + mtp.raw_rows = 0; + } + if let Some(dspark) = &mut self.dspark { + dspark.capture_mask = 0; + dspark.cache_start = 0; + dspark.cache_len = 0; + } self.tokens.clear(); self.checkpoint_tag = [0; 32]; Ok(()) @@ -845,7 +3520,12 @@ impl DeepSeekExecutor { self.session.position, token, self.session.raw_cap, + self.steering.as_ref(), + self.ssd.as_ref(), )?; + if let Some(dspark) = &mut self.dspark { + dspark.capture_decode(index as u32, &scratch.current_hc, shape)?; + } } encode_output(scratch, &self.weights, shape, map, size) } @@ -870,6 +3550,17 @@ impl Executor { context, quality, prefill_chunk, + 100, + EngineSpeculativeSettings { + mtp_draft_tokens: 1, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark: false, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }, crate::settings::EngineSsdSettings { enabled: false, cold: false, @@ -879,20 +3570,38 @@ impl Executor { full_layers_set: false, preload_experts: 0, }, + EngineSteeringSettings { + file: None, + ffn_scale: 0.0, + attention_scale: 0.0, + }, ) } + #[allow(clippy::too_many_arguments)] pub(super) fn open_configured( model: Model, context: u32, quality: bool, prefill_chunk: u32, - ssd: crate::settings::EngineSsdSettings, + power_percent: u8, + speculative: EngineSpeculativeSettings, + ssd: EngineSsdSettings, + steering: EngineSteeringSettings, ) -> Result { match model.shape.family { - ModelFamily::DeepSeek => DeepSeekExecutor::open(model, context, quality, prefill_chunk) - .map(Box::new) - .map(Self::DeepSeek), + ModelFamily::DeepSeek => DeepSeekExecutor::open( + model, + context, + quality, + prefill_chunk, + power_percent, + speculative, + ssd, + steering, + ) + .map(Box::new) + .map(Self::DeepSeek), ModelFamily::Glm => GlmExecutor::open(model, context, quality, ssd) .map(Box::new) .map(Self::Glm), @@ -906,6 +3615,24 @@ impl Executor { } } + pub(super) fn eval_speculative_greedy( + &mut self, + token: i32, + max_tokens: u32, + reasoning: ReasoningMode, + cancelled: &std::sync::atomic::AtomicBool, + ) -> Result, String> { + match self { + Self::DeepSeek(executor) => { + executor.eval_speculative_greedy(token, max_tokens, reasoning, cancelled) + } + Self::Glm(executor) => { + executor.eval(token)?; + Ok(vec![token]) + } + } + } + pub(super) fn prefill( &mut self, tokens: &[i32], @@ -924,6 +3651,13 @@ impl Executor { } } + pub(super) fn execution_stats(&self) -> ExecutionStats { + match self { + Self::DeepSeek(executor) => executor.execution_stats(), + Self::Glm(_) => ExecutionStats::default(), + } + } + pub(super) fn model(&self) -> &Model { match self { Self::DeepSeek(executor) => executor.model(), @@ -1412,6 +4146,8 @@ fn encode_batch_layer( pos: u32, rows: u32, raw_cap: u32, + steering: Option<&Steering>, + ssd: Option<&SsdPlan>, ) -> Result<(), String> { let hc_dim = shape.hc * shape.embd; let mix_hc = 2 * shape.hc + shape.hc * shape.hc; @@ -2037,22 +4773,24 @@ fn encode_batch_layer( "batch inverse attention RoPE", )?; let group_dim = shape.head_dim * (shape.heads / shape.out_groups); - let half_output = unsafe { - ds4_gpu_attention_output_q8_batch_f16_tensor( - s.q_half.raw(), - s.attention_low.raw(), - map, - size, - w.attn_output_a.offset, - w.attn_output_b.offset, - group_dim, - shape.lora_o, - shape.out_groups as u32, - shape.embd, - s.heads.raw(), - rows, - ) - } != 0; + let attention_steering = steering.is_some_and(|value| value.attention_scale != 0.0); + let half_output = !attention_steering + && unsafe { + ds4_gpu_attention_output_q8_batch_f16_tensor( + s.q_half.raw(), + s.attention_low.raw(), + map, + size, + w.attn_output_a.offset, + w.attn_output_b.offset, + group_dim, + shape.lora_o, + shape.out_groups as u32, + shape.embd, + s.heads.raw(), + rows, + ) + } != 0; if half_output { call( unsafe { @@ -2089,6 +4827,9 @@ fn encode_batch_layer( }, "batch attention output", )?; + if let Some(steering) = steering { + steering.apply(&s.attention_out, layer, rows, true)?; + } call( unsafe { ds4_gpu_hc_expand_split_tensor( @@ -2180,43 +4921,24 @@ fn encode_batch_layer( )?; let gate_row = w.expert_gate.bytes / (w.expert_gate.dims[1] * w.expert_gate.dims[2]); let down_row = w.expert_down.bytes / (w.expert_down.dims[1] * w.expert_down.dims[2]); - let mut mid_f16 = false; - call( - unsafe { - ds4_gpu_routed_moe_batch_tensor( - s.routed_out.raw(), - s.routed_gate.raw(), - s.routed_up.raw(), - s.routed_mid.raw(), - s.routed_experts.raw(), - map, - size, - w.expert_gate.offset, - w.expert_up.offset, - w.expert_down.offset, - w.expert_gate.kind, - w.expert_down.kind, - w.expert_gate.dims[1] * gate_row, - gate_row, - w.expert_down.dims[1] * down_row, - down_row, - w.expert_gate.dims[0] as u32, - w.expert_down.dims[0] as u32, - w.expert_down.dims[1] as u32, - s.router_selected.raw(), - s.router_weights.raw(), - shape.experts as u32, - shape.experts_used as u32, - shape.swiglu_clamp, - s.norm.raw(), - layer, - rows, - &mut mid_f16, - false, - ) - }, - "batch routed experts", - )?; + if let Some(ssd) = ssd { + let table = StreamExpertTable { + model_map: map, + model_size: size, + layer, + total_experts: shape.experts as u32, + gate_offset: w.expert_gate.offset, + up_offset: w.expert_up.offset, + down_offset: w.expert_down.offset, + gate_expert_bytes: w.expert_gate.dims[1] * gate_row, + down_expert_bytes: w.expert_down.dims[1] * down_row, + }; + ssd.begin_selected( + &s.router_selected, + table, + rows.saturating_mul(shape.experts_used as u32), + )?; + } q8_rows( &s.shared_gate, w.shared_gate, @@ -2260,20 +4982,88 @@ fn encode_batch_layer( map, size, )?; + if let Some(ssd) = ssd { + ssd.finish_selected(false)?; + } + let mut mid_f16 = false; call( unsafe { - ds4_gpu_hc_expand_add_split_tensor( - s.next_hc.raw(), + ds4_gpu_routed_moe_batch_tensor( s.routed_out.raw(), - s.shared_out.raw(), - s.after_attention_hc.raw(), - s.hc_split.raw(), - shape.embd as u32, - shape.hc as u32, + s.routed_gate.raw(), + s.routed_up.raw(), + s.routed_mid.raw(), + s.routed_experts.raw(), + map, + size, + w.expert_gate.offset, + w.expert_up.offset, + w.expert_down.offset, + w.expert_gate.kind, + w.expert_down.kind, + w.expert_gate.dims[1] * gate_row, + gate_row, + w.expert_down.dims[1] * down_row, + down_row, + w.expert_gate.dims[0] as u32, + w.expert_down.dims[0] as u32, + w.expert_down.dims[1] as u32, + s.router_selected.raw(), + s.router_weights.raw(), + shape.experts as u32, + shape.experts_used as u32, + shape.swiglu_clamp, + s.norm.raw(), + layer, + rows, + &mut mid_f16, + false, ) }, - "batch FFN HC expansion", - ) + "batch routed experts", + )?; + if let Some(steering) = steering.filter(|value| value.ffn_scale != 0.0) { + call( + unsafe { + ds4_gpu_add_tensor( + s.attention_out.raw(), + s.routed_out.raw(), + s.shared_out.raw(), + rows * shape.embd as u32, + ) + }, + "combining batch FFN output", + )?; + steering.apply(&s.attention_out, layer, rows, false)?; + call( + unsafe { + ds4_gpu_hc_expand_split_tensor( + s.next_hc.raw(), + s.attention_out.raw(), + s.after_attention_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "batch steered FFN HC expansion", + ) + } else { + call( + unsafe { + ds4_gpu_hc_expand_add_split_tensor( + s.next_hc.raw(), + s.routed_out.raw(), + s.shared_out.raw(), + s.after_attention_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "batch FFN HC expansion", + ) + } } // Mirrors one fixed DS4 tape invocation; grouping these scalar dimensions would @@ -2290,6 +5080,29 @@ fn encode_layer( pos: u32, token: u32, raw_cap: u32, + steering: Option<&Steering>, + ssd: Option<&SsdPlan>, +) -> Result<(), String> { + encode_layer_with_cache_rows( + s, state, w, shape, map, size, layer, pos, token, raw_cap, steering, ssd, None, + ) +} + +#[allow(clippy::too_many_arguments)] +fn encode_layer_with_cache_rows( + s: &Scratch, + state: &mut LayerState, + w: &Layer, + shape: super::Shape, + map: *const c_void, + size: u64, + layer: u32, + pos: u32, + token: u32, + raw_cap: u32, + steering: Option<&Steering>, + ssd: Option<&SsdPlan>, + cache_rows: Option, ) -> Result<(), String> { let hc_dim = shape.hc * shape.embd; let mix_hc = 2 * shape.hc + shape.hc * shape.hc; @@ -2331,7 +5144,7 @@ fn encode_layer( }, "attention HC norm", )?; - f16( + matmul( &s.hc_mix, w.hc_attn_fn, hc_dim, @@ -2497,7 +5310,12 @@ fn encode_layer( }, "KV cache write", )?; - let (n_raw, raw_start) = raw_decode_span(pos, raw_cap, shape.sliding_window as u32); + let (n_raw, _) = raw_decode_span( + cache_rows.unwrap_or(pos), + raw_cap, + shape.sliding_window as u32, + ); + let raw_start = (pos + 1 - n_raw) % raw_cap; if let (Some(weights), Some(compression)) = (w.attn_compressor.as_ref(), state.compression.as_mut()) { @@ -2716,25 +5534,51 @@ fn encode_layer( }, "attention low projection", )?; - call( - unsafe { - ds4_gpu_matmul_q8_0_hc_expand_tensor( - s.next_hc.raw(), - s.attention_out.raw(), - map, - size, - w.attn_output_b.offset, - shape.out_groups * shape.lora_o, - shape.embd, - s.attention_low.raw(), - s.current_hc.raw(), - s.hc_split.raw(), - shape.embd as u32, - shape.hc as u32, - ) - }, - "attention output", - )?; + if let Some(steering) = steering.filter(|value| value.attention_scale != 0.0) { + q8( + &s.attention_out, + w.attn_output_b, + shape.out_groups * shape.lora_o, + shape.embd, + &s.attention_low, + map, + size, + )?; + steering.apply(&s.attention_out, layer, 1, true)?; + call( + unsafe { + ds4_gpu_hc_expand_split_tensor( + s.next_hc.raw(), + s.attention_out.raw(), + s.current_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "steered attention HC expansion", + )?; + } else { + call( + unsafe { + ds4_gpu_matmul_q8_0_hc_expand_tensor( + s.next_hc.raw(), + s.attention_out.raw(), + map, + size, + w.attn_output_b.offset, + shape.out_groups * shape.lora_o, + shape.embd, + s.attention_low.raw(), + s.current_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "attention output", + )?; + } call( unsafe { ds4_gpu_rms_norm_plain_tensor( @@ -2746,7 +5590,7 @@ fn encode_layer( }, "FFN HC norm", )?; - f16( + matmul( &s.hc_mix, w.hc_ffn_fn, hc_dim, @@ -2777,7 +5621,7 @@ fn encode_layer( }, "FFN HC mix", )?; - f16( + matmul( &s.router_logits, w.router, shape.embd, @@ -2812,6 +5656,52 @@ fn encode_layer( )?; let gate_row = w.expert_gate.bytes / (w.expert_gate.dims[1] * w.expert_gate.dims[2]); let down_row = w.expert_down.bytes / (w.expert_down.dims[1] * w.expert_down.dims[2]); + if let Some(ssd) = ssd { + let table = StreamExpertTable { + model_map: map, + model_size: size, + layer, + total_experts: shape.experts as u32, + gate_offset: w.expert_gate.offset, + up_offset: w.expert_up.offset, + down_offset: w.expert_down.offset, + gate_expert_bytes: w.expert_gate.dims[1] * gate_row, + down_expert_bytes: w.expert_down.dims[1] * down_row, + }; + ssd.begin_selected(&s.router_selected, table, shape.experts_used as u32)?; + } + call( + unsafe { + ds4_gpu_shared_gate_up_swiglu_q8_0_tensor( + s.shared_gate.raw(), + s.shared_up.raw(), + s.shared_mid.raw(), + map, + size, + w.shared_gate.offset, + w.shared_up.offset, + shape.embd, + shape.ff_expert, + s.norm.raw(), + shape.swiglu_clamp, + ) + }, + "shared expert gate/up", + )?; + if ssd.is_some() || steering.is_some_and(|value| value.ffn_scale != 0.0) { + q8( + &s.shared_out, + w.shared_down, + shape.ff_expert, + shape.embd, + &s.shared_mid, + map, + size, + )?; + } + if let Some(ssd) = ssd { + ssd.finish_selected(true)?; + } call( unsafe { ds4_gpu_routed_moe_one_tensor( @@ -2847,44 +5737,69 @@ fn encode_layer( }, "routed experts", )?; - call( - unsafe { - ds4_gpu_shared_gate_up_swiglu_q8_0_tensor( - s.shared_gate.raw(), - s.shared_up.raw(), - s.shared_mid.raw(), - map, - size, - w.shared_gate.offset, - w.shared_up.offset, - shape.embd, - shape.ff_expert, - s.norm.raw(), - shape.swiglu_clamp, - ) - }, - "shared expert gate/up", - )?; - call( - unsafe { - ds4_gpu_shared_down_hc_expand_q8_0_tensor( - s.current_hc.raw(), - s.shared_out.raw(), - map, - size, - w.shared_down.offset, - shape.ff_expert, - shape.embd, - s.shared_mid.raw(), - s.routed_out.raw(), - s.next_hc.raw(), - s.hc_split.raw(), - shape.embd as u32, - shape.hc as u32, - ) - }, - "shared expert output", - ) + if let Some(steering) = steering.filter(|value| value.ffn_scale != 0.0) { + call( + unsafe { + ds4_gpu_add_tensor( + s.attention_out.raw(), + s.routed_out.raw(), + s.shared_out.raw(), + shape.embd as u32, + ) + }, + "combining FFN output", + )?; + steering.apply(&s.attention_out, layer, 1, false)?; + call( + unsafe { + ds4_gpu_hc_expand_split_tensor( + s.current_hc.raw(), + s.attention_out.raw(), + s.next_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "steered FFN HC expansion", + ) + } else if ssd.is_some() { + call( + unsafe { + ds4_gpu_hc_expand_add_split_tensor( + s.current_hc.raw(), + s.routed_out.raw(), + s.shared_out.raw(), + s.next_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "streamed FFN HC expansion", + ) + } else { + call( + unsafe { + ds4_gpu_shared_down_hc_expand_q8_0_tensor( + s.current_hc.raw(), + s.shared_out.raw(), + map, + size, + w.shared_down.offset, + shape.ff_expert, + shape.embd, + s.shared_mid.raw(), + s.routed_out.raw(), + s.next_hc.raw(), + s.hc_split.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "shared expert output", + ) + } } #[allow(clippy::too_many_arguments)] @@ -3232,6 +6147,231 @@ fn encode_output( ) } +#[allow(clippy::too_many_arguments)] +fn encode_mtp_output( + s: &Scratch, + mtp: &LegacyMtpWeights, + base_output: Weight, + shape: super::Shape, + support_map: *const c_void, + support_size: u64, + base_map: *const c_void, + base_size: u64, +) -> Result<(), String> { + let hc_dim = shape.hc * shape.embd; + call( + unsafe { + ds4_gpu_rms_norm_plain_tensor( + s.flat_hc.raw(), + s.current_hc.raw(), + hc_dim as u32, + shape.rms_epsilon, + ) + }, + "MTP output HC norm", + )?; + matmul( + &s.output_pre, + mtp.hc_head_fn, + hc_dim, + shape.hc, + &s.flat_hc, + support_map, + support_size, + )?; + call( + unsafe { + ds4_gpu_output_hc_weights_tensor( + s.output_weights.raw(), + s.output_pre.raw(), + support_map, + support_size, + mtp.hc_head_scale.offset, + mtp.hc_head_base.offset, + shape.hc as u32, + shape.hc_epsilon, + ) + }, + "MTP output HC weights", + )?; + call( + unsafe { + ds4_gpu_hc_weighted_sum_tensor( + s.output_embedding.raw(), + s.current_hc.raw(), + s.output_weights.raw(), + shape.embd as u32, + shape.hc as u32, + ) + }, + "MTP output HC collapse", + )?; + call( + unsafe { + ds4_gpu_rms_norm_weight_tensor( + s.output_norm.raw(), + s.output_embedding.raw(), + support_map, + support_size, + mtp.norm.offset, + shape.embd as u32, + shape.rms_epsilon, + ) + }, + "MTP output norm", + )?; + q8( + &s.logits, + base_output, + shape.embd, + shape.vocab, + &s.output_norm, + base_map, + base_size, + ) +} + +fn argmax(values: &[f32]) -> i32 { + values + .iter() + .enumerate() + .max_by(|left, right| left.1.total_cmp(right.1)) + .map_or(-1, |(index, _)| index as i32) +} + +fn top2(values: &[f32]) -> (i32, f32) { + let mut first = (usize::MAX, f32::NEG_INFINITY); + let mut second = f32::NEG_INFINITY; + for (index, &value) in values.iter().enumerate() { + if value > first.1 { + second = first.1; + first = (index, value); + } else if value > second { + second = value; + } + } + (first.0 as i32, first.1 - second) +} + +fn dense_row(model: &Gguf, weight: Weight, row: u32) -> Result, String> { + if u64::from(row) >= weight.dims[1] { + return Err("dense row is outside the tensor".into()); + } + let width = weight.dims[0] as usize; + let row_bytes = match weight.kind { + F32 => width.checked_mul(4), + F16 => width.checked_mul(2), + Q8_0 => width.div_ceil(32).checked_mul(34), + _ => None, + } + .ok_or("unsupported DSpark dense tensor layout")?; + let offset = weight + .offset + .checked_add(u64::from(row) * row_bytes as u64) + .ok_or("DSpark dense row offset overflow")?; + if offset > model.len() || row_bytes as u64 > model.len() - offset { + return Err("DSpark dense row is outside the GGUF mapping".into()); + } + let bytes = + unsafe { std::slice::from_raw_parts(model.map_ptr().add(offset as usize), row_bytes) }; + let mut out = vec![0.0; width]; + match weight.kind { + F32 => { + for (value, bytes) in out.iter_mut().zip(bytes.chunks_exact(4)) { + *value = f32::from_le_bytes(bytes.try_into().expect("four bytes")); + } + } + F16 => { + for (value, bytes) in out.iter_mut().zip(bytes.chunks_exact(2)) { + *value = half_to_f32(u16::from_le_bytes(bytes.try_into().expect("two bytes"))); + } + } + Q8_0 => { + for (block, bytes) in bytes.chunks_exact(34).enumerate() { + let scale = half_to_f32(u16::from_le_bytes([bytes[0], bytes[1]])); + for (index, quantized) in bytes[2..].iter().enumerate() { + let output = block * 32 + index; + if output == width { + break; + } + out[output] = scale * f32::from(*quantized as i8); + } + } + } + _ => unreachable!(), + } + Ok(out) +} + +fn dense_dot(model: &Gguf, weight: Weight, row: u32, values: &[f32]) -> Result { + if weight.dims[0] as usize != values.len() || u64::from(row) >= weight.dims[1] { + return Err("DSpark dense dot has mismatched dimensions".into()); + } + let width = values.len(); + let row_bytes = match weight.kind { + F32 => width.checked_mul(4), + F16 => width.checked_mul(2), + Q8_0 => width.div_ceil(32).checked_mul(34), + _ => None, + } + .ok_or("unsupported DSpark dense tensor layout")?; + let offset = weight + .offset + .checked_add(u64::from(row) * row_bytes as u64) + .ok_or("DSpark dense dot offset overflow")?; + if offset > model.len() || row_bytes as u64 > model.len() - offset { + return Err("DSpark dense dot is outside the GGUF mapping".into()); + } + let bytes = + unsafe { std::slice::from_raw_parts(model.map_ptr().add(offset as usize), row_bytes) }; + let sum = match weight.kind { + F32 => bytes + .chunks_exact(4) + .zip(values) + .map(|(bytes, value)| f32::from_le_bytes(bytes.try_into().expect("four bytes")) * value) + .sum(), + F16 => bytes + .chunks_exact(2) + .zip(values) + .map(|(bytes, value)| { + half_to_f32(u16::from_le_bytes(bytes.try_into().expect("two bytes"))) * value + }) + .sum(), + Q8_0 => { + let mut sum = 0.0; + for (block, bytes) in bytes.chunks_exact(34).enumerate() { + let scale = half_to_f32(u16::from_le_bytes([bytes[0], bytes[1]])); + for (index, quantized) in bytes[2..].iter().enumerate() { + let input = block * 32 + index; + if input == width { + break; + } + sum += scale * f32::from(*quantized as i8) * values[input]; + } + } + sum + } + _ => unreachable!(), + }; + Ok(sum) +} + +fn half_to_f32(value: u16) -> f32 { + let sign = u32::from(value & 0x8000) << 16; + let exponent = u32::from((value >> 10) & 0x1f); + let fraction = u32::from(value & 0x03ff); + let bits = match exponent { + 0 if fraction == 0 => sign, + 0 => { + let shift = fraction.leading_zeros() - 21; + sign | ((127 - 14 - shift) << 23) | ((fraction << (shift + 1) & 0x03ff) << 13) + } + 31 => sign | 0x7f80_0000 | (fraction << 13), + _ => sign | ((exponent + 127 - 15) << 23) | (fraction << 13), + }; + f32::from_bits(bits) +} + fn f16( out: &Buffer, weight: Weight, @@ -3299,6 +6439,21 @@ fn matmul_rows( ) -> Result<(), String> { match weight.kind { F16 => f16_rows(out, weight, input, output, x, rows, map, size), + F32 => call( + unsafe { + ds4_gpu_matmul_f32_tensor( + out.raw(), + map, + size, + weight.offset, + input, + output, + x.raw(), + u64::from(rows), + ) + }, + "batch F32 projection", + ), Q8_0 => q8_rows(out, weight, input, output, x, rows, map, size), kind => Err(format!("unsupported Metal batch projection type {kind}")), } @@ -3315,6 +6470,21 @@ fn matmul( ) -> Result<(), String> { match weight.kind { F16 => f16(out, weight, input, output, x, map, size), + F32 => call( + unsafe { + ds4_gpu_matmul_f32_tensor( + out.raw(), + map, + size, + weight.offset, + input, + output, + x.raw(), + 1, + ) + }, + "F32 projection", + ), Q8_0 => q8(out, weight, input, output, x, map, size), kind => Err(format!("unsupported Metal projection type {kind}")), } @@ -3374,6 +6544,20 @@ fn q8_rows( ) } +fn throttle(average: &mut f64, elapsed: Duration, power_percent: u8) { + if power_percent >= 100 { + return; + } + let sample = elapsed.as_secs_f64(); + *average = if *average <= 0.0 || !average.is_finite() { + sample + } else { + *average * 0.875 + sample * 0.125 + }; + let sleep = *average * (100.0 - f64::from(power_percent)) / f64::from(power_percent); + std::thread::sleep(Duration::from_secs_f64(sleep)); +} + fn call(result: i32, operation: &str) -> Result<(), String> { if result == 0 { Err(format!("Metal failed while {operation}")) @@ -3388,7 +6572,10 @@ fn check(result: i32, operation: &str) -> Result<(), String> { #[cfg(test)] mod tests { - use super::{compression_ratio, raw_batch_span, raw_decode_span}; + use super::{ + compression_ratio, effective_prefill_cap, effective_raw_cap, + estimated_deepseek_runtime_bytes, raw_batch_span, raw_decode_span, + }; use crate::engine::{FLASH, PRO}; #[test] @@ -3414,4 +6601,566 @@ mod tests { assert_eq!(raw_batch_span(4_096, 1_925, 4_352, 128), (2_053, 3_968)); assert_eq!(raw_batch_span(4_500, 1, 4_352, 128), (128, 21)); } + + #[test] + fn half_conversion_handles_normal_and_subnormal_values() { + assert_eq!(super::half_to_f32(0x0000), 0.0); + assert_eq!(super::half_to_f32(0x3c00), 1.0); + assert_eq!(super::half_to_f32(0xc000), -2.0); + assert_eq!(super::half_to_f32(0x0400), 2.0_f32.powi(-14)); + assert_eq!(super::half_to_f32(0x0001), 2.0_f32.powi(-24)); + } + + #[test] + fn streaming_memory_plan_matches_ds4_graph_formulas() { + assert_eq!(effective_prefill_cap(16_384, 0), 4_096); + assert_eq!(effective_prefill_cap(16_384, 2_048), 2_048); + assert_eq!(effective_raw_cap(FLASH, 16_384, 4_096), 4_352); + assert_eq!(effective_raw_cap(FLASH, 1_000, 1_000), 1_024); + + let raw = 43_u64 * 4_352 * 512 * 4; + let ratio4 = 21_u64 * (16_384 / 4 + 2) * (512 * 2 + 128 * 4); + let ratio128 = 20_u64 * (16_384 / 128 + 2) * 512 * 2; + let scratch = 2_u64 * (16_384 / 4 + 2) * 4_096 * 4 + (4_096 / 4 + 2) as u64 * 512 * 4; + assert_eq!( + estimated_deepseek_runtime_bytes(FLASH, 16_384, 0), + raw + ratio4 + ratio128 + scratch + ); + } + + #[test] + #[ignore = "requires the installed 81 GiB Flash and legacy MTP GGUF fixtures"] + fn legacy_mtp_runs_a_target_owned_greedy_cycle() { + use super::{DeepSeekExecutor, argmax, configure_sources}; + use crate::engine::Model; + use crate::engine::gguf::Gguf; + use crate::engine::validation::validate_support; + use crate::model::ModelChoice; + use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, + }; + use std::path::Path; + use std::sync::atomic::AtomicBool; + + let main_path = Path::new( + "../ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf", + ); + configure_sources().unwrap(); + let support_path = Path::new("../ds4/gguf/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf"); + let mut model = Model::open_main(main_path, ModelChoice::DeepSeekV4Flash).unwrap(); + let support = Gguf::open(support_path).unwrap(); + model.support_kind = Some(validate_support(&support, &model.shape).unwrap()); + model.support = Some(support); + let prompt = model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "hi".into(), + }], + crate::settings::ReasoningMode::Direct, + ); + let mut executor = DeepSeekExecutor::open( + model, + 512, + false, + 512, + 100, + EngineSpeculativeSettings { + mtp_draft_tokens: 4, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark: false, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }, + EngineSsdSettings { + enabled: false, + cold: false, + cache_experts: 0, + cache_bytes: 0, + full_layers: 0, + full_layers_set: false, + preload_experts: 0, + }, + EngineSteeringSettings { + file: None, + ffn_scale: 0.0, + attention_scale: 0.0, + }, + ) + .unwrap(); + executor.prefill(&prompt, |_| true).unwrap(); + let mut generated = Vec::new(); + while generated.len() < 8 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + (8 - generated.len()) as u32, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + generated.extend(cycle); + } + assert_eq!(generated, [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316]); + let mtp = executor.legacy_mtp.as_ref().unwrap(); + assert!(mtp.drafted > 0); + + executor.reset().unwrap(); + let copy_prompt = executor.model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "Reproduce the following C code EXACTLY, character for character, inside a single code block and output nothing else:\n\n```c\nstatic uint32_t clamp_u32(uint32_t v, uint32_t lo, uint32_t hi) {\n if (v < lo) return lo;\n if (v > hi) return hi;\n return v;\n}\n\nstatic uint32_t ring_advance(uint32_t pos, uint32_t cap) {\n uint32_t next = pos + 1u;\n return next >= cap ? 0u : next;\n}\n\nstatic int scratch_init(scratch *s, uint32_t ctx_size) {\n if (ctx_size == 0u) ctx_size = 1u;\n s->ctx_size = ctx_size;\n s->comp_cap = ctx_size / 4u + 2u;\n s->rows = clamp_u32(s->comp_cap, 1u, 4096u);\n s->head = 0u;\n return s->rows > 0u ? 0 : -1;\n}\n```" + .into(), + }], + crate::settings::ReasoningMode::Direct, + ); + executor.prefill(©_prompt, |_| true).unwrap(); + let mut max_cycle = 0; + let mut copied = 0; + while copied < 128 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + 128 - copied, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + max_cycle = max_cycle.max(cycle.len()); + copied += cycle.len() as u32; + } + assert!(max_cycle > 1, "legacy MTP never committed a draft token"); + + executor.reset().unwrap(); + executor.legacy_mtp.as_mut().unwrap().draft_limit = 1; + executor.prefill(&prompt, |_| true).unwrap(); + let mut target_only = Vec::new(); + while target_only.len() < 8 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + (8 - target_only.len()) as u32, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + assert_eq!(cycle.len(), 1); + target_only.extend(cycle); + } + assert_eq!(target_only, [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316]); + } + + #[test] + #[ignore = "requires the installed 81 GiB Flash and DSpark GGUF fixtures"] + fn dspark_runs_a_target_owned_greedy_cycle() { + use super::{DeepSeekExecutor, argmax, configure_sources}; + use crate::engine::Model; + use crate::engine::gguf::Gguf; + use crate::engine::validation::validate_support; + use crate::model::ModelChoice; + use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, + }; + use std::path::Path; + use std::sync::atomic::AtomicBool; + + configure_sources().unwrap(); + let main_path = Path::new( + "../ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf", + ); + let support_path = Path::new("../ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf"); + let mut model = Model::open_main(main_path, ModelChoice::DeepSeekV4Flash).unwrap(); + let support = Gguf::open(support_path).unwrap(); + model.support_kind = Some(validate_support(&support, &model.shape).unwrap()); + model.support = Some(support); + let prompt = model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "hi".into(), + }], + crate::settings::ReasoningMode::Direct, + ); + let mut executor = DeepSeekExecutor::open( + model, + 64, + false, + 64, + 100, + EngineSpeculativeSettings { + mtp_draft_tokens: 1, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark: true, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }, + EngineSsdSettings { + enabled: false, + cold: false, + cache_experts: 0, + cache_bytes: 0, + full_layers: 0, + full_layers_set: false, + preload_experts: 0, + }, + EngineSteeringSettings { + file: None, + ffn_scale: 0.0, + attention_scale: 0.0, + }, + ) + .unwrap(); + executor.prefill(&prompt, |_| true).unwrap(); + let mut generated = Vec::new(); + while generated.len() < 8 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + (8 - generated.len()) as u32, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + generated.extend(cycle); + } + assert_eq!(generated, [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316]); + let dspark = executor.dspark.as_ref().unwrap(); + assert!(dspark.drafted > 0); + assert!(dspark.accepted > 0); + + executor.reset().unwrap(); + executor.dspark.as_mut().unwrap().strict = true; + executor.prefill(&prompt, |_| true).unwrap(); + let mut target_only = Vec::new(); + while target_only.len() < 8 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + (8 - target_only.len()) as u32, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + assert_eq!(cycle.len(), 1); + target_only.extend(cycle); + } + assert_eq!(target_only, [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316]); + + let mut turns = Vec::new(); + for _ in 0..4 { + for (user, content) in [(true, "ping"), (false, "pong")] { + turns.push(crate::engine::ChatTurn { + user, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: content.into(), + }); + } + } + let ping_pong = + executor + .model + .render_conversation("", &turns, crate::settings::ReasoningMode::Direct); + assert!(ping_pong.len() > executor.dspark.as_ref().unwrap().config.block_size as usize + 1); + executor.reset().unwrap(); + executor.dspark.as_mut().unwrap().strict = false; + executor.prefill(&ping_pong, |_| true).unwrap(); + let first = argmax(executor.logits()); + assert!( + !executor + .eval_speculative_greedy( + first, + 4, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap() + .is_empty() + ); + } + + #[test] + #[ignore = "requires the installed Flash, legacy MTP, and DSpark GGUF fixtures"] + fn ssd_streaming_supports_legacy_mtp_and_dspark() { + use super::{DeepSeekExecutor, argmax, configure_sources}; + use crate::engine::Model; + use crate::engine::gguf::Gguf; + use crate::engine::validation::validate_support; + use crate::model::ModelChoice; + use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, + }; + use std::path::Path; + use std::sync::atomic::AtomicBool; + + configure_sources().unwrap(); + let main_path = Path::new( + "../ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf", + ); + let cases = [ + ( + "../ds4/gguf/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf", + false, + 4, + ), + ("../ds4/gguf/DeepSeek-V4-Flash-DSpark-support.gguf", true, 1), + ]; + for (support_path, dspark, draft_tokens) in cases { + let mut model = Model::open_main(main_path, ModelChoice::DeepSeekV4Flash).unwrap(); + let support = Gguf::open(Path::new(support_path)).unwrap(); + model.support_kind = Some(validate_support(&support, &model.shape).unwrap()); + model.support = Some(support); + let prompt = model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "hi".into(), + }], + crate::settings::ReasoningMode::Direct, + ); + let mut executor = DeepSeekExecutor::open( + model, + 64, + false, + 64, + 100, + EngineSpeculativeSettings { + mtp_draft_tokens: draft_tokens, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }, + EngineSsdSettings { + enabled: true, + cold: true, + cache_experts: 256, + cache_bytes: 0, + full_layers: 0, + full_layers_set: false, + preload_experts: 0, + }, + EngineSteeringSettings { + file: None, + ffn_scale: 0.0, + attention_scale: 0.0, + }, + ) + .unwrap(); + executor.prefill(&prompt, |_| true).unwrap(); + let mut generated = Vec::new(); + while generated.len() < 8 { + let first = argmax(executor.logits()); + let cycle = executor + .eval_speculative_greedy( + first, + (8 - generated.len()) as u32, + crate::settings::ReasoningMode::Direct, + &AtomicBool::new(false), + ) + .unwrap(); + generated.extend(cycle); + } + assert_eq!( + generated, + [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316], + "SSD speculative output differed for {support_path}" + ); + } + } + + #[test] + #[ignore = "requires the installed 81 GiB Flash GGUF fixture and a Metal device"] + fn flash_resident_and_ssd_streaming_choose_the_same_tokens() { + use super::{DeepSeekExecutor, argmax, configure_sources}; + use crate::engine::Model; + use crate::model::ModelChoice; + use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, + }; + use std::path::Path; + + fn run(path: &Path, streaming: bool) -> Vec { + let model = Model::open_main(path, ModelChoice::DeepSeekV4Flash).unwrap(); + let prompt = model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "hi".into(), + }], + crate::settings::ReasoningMode::Direct, + ); + assert_eq!(prompt.len(), 5); + let speculative = EngineSpeculativeSettings { + mtp_draft_tokens: 1, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark: false, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }; + let mut executor = DeepSeekExecutor::open( + model, + 32, + false, + 32, + 100, + speculative, + EngineSsdSettings { + enabled: streaming, + cold: true, + cache_experts: if streaming { 16 } else { 0 }, + cache_bytes: 0, + full_layers: 0, + full_layers_set: false, + preload_experts: 0, + }, + EngineSteeringSettings { + file: None, + ffn_scale: 0.0, + attention_scale: 0.0, + }, + ) + .unwrap(); + executor.prefill(&prompt, |_| true).unwrap(); + let mut tokens = Vec::new(); + for _ in 0..4 { + let token = argmax(executor.logits()); + tokens.push(token); + executor.eval(token).unwrap(); + } + tokens + } + + configure_sources().unwrap(); + let path = Path::new( + "../ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf", + ); + let resident = run(path, false); + assert_eq!(resident, [19_923, 3, 1_730, 588]); + assert_eq!(resident, run(path, true)); + } + + #[test] + #[ignore = "requires the installed Flash GGUF, DS4 steering fixture, and a Metal device"] + fn directional_steering_matches_the_ds4_token_oracle() { + use super::{DeepSeekExecutor, argmax, configure_sources}; + use crate::engine::Model; + use crate::model::ModelChoice; + use crate::settings::{ + EngineSpeculativeSettings, EngineSsdSettings, EngineSteeringSettings, + }; + use std::path::Path; + + configure_sources().unwrap(); + let model_path = Path::new( + "../ds4/gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix.gguf", + ); + let steering_path = "../ds4/dir-steering/out/verbosity.f32"; + let cases = [ + (1.0, 0.0, [19_923, 3, 1_730, 588, 342, 8_233, 440, 4_316]), + (0.0, 1.0, [19_923, 3, 1_730, 588, 342, 1_694, 440, 4_316]), + ]; + for (ffn_scale, attention_scale, expected) in cases { + let model = Model::open_main(model_path, ModelChoice::DeepSeekV4Flash).unwrap(); + let prompt = model.render_conversation( + "", + &[crate::engine::ChatTurn { + user: true, + tool: false, + system: false, + skip_previous_eos: false, + reasoning: None, + reasoning_complete: true, + content: "hi".into(), + }], + crate::settings::ReasoningMode::Direct, + ); + let mut executor = DeepSeekExecutor::open( + model, + 32, + false, + 32, + 100, + EngineSpeculativeSettings { + mtp_draft_tokens: 1, + mtp_margin: 3.0, + glm_mtp: false, + glm_mtp_timing: false, + dspark: false, + dspark_confidence_threshold: 0.9, + dspark_confidence_threshold_set: false, + dspark_strict: false, + }, + EngineSsdSettings { + enabled: false, + cold: false, + cache_experts: 0, + cache_bytes: 0, + full_layers: 0, + full_layers_set: false, + preload_experts: 0, + }, + EngineSteeringSettings { + file: Some(steering_path.into()), + ffn_scale, + attention_scale, + }, + ) + .unwrap(); + executor.prefill(&prompt, |_| true).unwrap(); + let mut tokens = Vec::new(); + for _ in 0..8 { + let token = argmax(executor.logits()); + tokens.push(token); + executor.eval(token).unwrap(); + } + assert_eq!(tokens, expected); + } + } } diff --git a/src/engine/metal/checkpoint.rs b/src/engine/metal/checkpoint.rs index 84c4afb..d1ae841 100644 --- a/src/engine/metal/checkpoint.rs +++ b/src/engine/metal/checkpoint.rs @@ -301,6 +301,15 @@ impl DeepSeekExecutor { self.tokens = tokens; self.logits = logits; self.checkpoint_tag = checkpoint_tag; + if let Some(mtp) = &mut self.legacy_mtp { + mtp.draft_token = None; + mtp.raw_rows = 0; + } + if let Some(dspark) = &mut self.dspark { + dspark.capture_mask = 0; + dspark.cache_start = 0; + dspark.cache_len = 0; + } Ok(()) } } diff --git a/src/engine/metal/glm.rs b/src/engine/metal/glm.rs index 921bf32..d8cb537 100644 --- a/src/engine/metal/glm.rs +++ b/src/engine/metal/glm.rs @@ -231,7 +231,7 @@ impl GlmExecutor { } let weights = GlmWeights::bind(&model)?; let admission = admission_bytes(&model, &weights, context, ssd)?; - let context_handle = Context::open(&model, quality, ssd.enabled, admission)?; + let context_handle = Context::open(&model, quality, ssd.enabled, admission, None)?; configure_streaming(&model, &weights, ssd)?; let scratch = GlmScratch::allocate(&model, context)?; let caches = (0..weights.layers.len()) diff --git a/src/engine/metal/gpu.rs b/src/engine/metal/gpu.rs index a0a0b6a..ba54cbb 100644 --- a/src/engine/metal/gpu.rs +++ b/src/engine/metal/gpu.rs @@ -28,6 +28,14 @@ unsafe extern "C" { map_size: u64, max_tensor_bytes: u64, ) -> i32; + pub(super) fn ds4_gpu_set_model_map_spans( + model_map: *const c_void, + model_size: u64, + offsets: *const u64, + sizes: *const u64, + count: u32, + max_tensor_bytes: u64, + ) -> i32; pub(super) fn ds4_gpu_set_quality(quality: bool); pub(super) fn ds4_gpu_set_glm_model(enabled: bool); pub(super) fn ds4_gpu_set_ssd_streaming(enabled: bool); @@ -39,6 +47,19 @@ unsafe extern "C" { gate_expert_bytes: u64, down_expert_bytes: u64, ) -> u32; + pub(super) fn ds4_gpu_stream_expert_cache_seed_experts( + table: *const StreamExpertTable, + expert_ids: *const i32, + expert_priorities: *const u32, + experts: u32, + ) -> i32; + pub(super) fn ds4_gpu_stream_expert_cache_begin_selected_load( + table: *const StreamExpertTable, + selected_ids: *const i32, + selected: u32, + ) -> i32; + pub(super) fn ds4_gpu_stream_expert_cache_note_service_thread(); + pub(super) fn ds4_gpu_stream_expert_cache_reset_route_hotness(); pub(super) fn ds4_gpu_glm_stream_expert_cache_begin_selected_load_tensor( table: *const StreamExpertTable, selected: *const GpuTensor, @@ -79,8 +100,23 @@ unsafe extern "C" { src_offset: u64, count: u64, ) -> i32; + pub(super) fn ds4_gpu_pack_slot_rows_f32_tensor( + out: *mut GpuTensor, + slots: *const GpuTensor, + rows: u32, + width: u32, + slot_count: u32, + slot_stride: u32, + ) -> i32; pub(super) fn ds4_gpu_begin_commands() -> i32; pub(super) fn ds4_gpu_end_commands() -> i32; + pub(super) fn ds4_gpu_signal_selected_readback_ready(event: *mut u64) -> i32; + pub(super) fn ds4_gpu_wait_selected_readback_ready( + event: u64, + label: *const std::ffi::c_char, + ) -> i32; + pub(super) fn ds4_gpu_routed_moe_set_selected_override(selected: *const i32, count: u32) + -> i32; pub(super) fn ds4_gpu_embed_tokens_hc_tensor( out: *mut GpuTensor, @@ -110,6 +146,13 @@ unsafe extern "C" { n: u32, eps: f32, ) -> i32; + pub(super) fn ds4_gpu_rms_norm_plain_rows_tensor( + out: *mut GpuTensor, + x: *const GpuTensor, + n: u32, + rows: u32, + eps: f32, + ) -> i32; pub(super) fn ds4_gpu_rms_norm_weight_tensor( out: *mut GpuTensor, x: *const GpuTensor, @@ -119,6 +162,36 @@ unsafe extern "C" { n: u32, eps: f32, ) -> i32; + pub(super) fn ds4_gpu_rms_norm_weight_rows_tensor( + out: *mut GpuTensor, + x: *const GpuTensor, + map: *const c_void, + size: u64, + weight: u64, + n: u32, + rows: u32, + eps: f32, + ) -> i32; + pub(super) fn ds4_gpu_repeat_hc_tensor( + out: *mut GpuTensor, + x: *const GpuTensor, + embd: u32, + hc: u32, + ) -> i32; + pub(super) fn ds4_gpu_attention_noncausal_raw_batch_heads_tensor( + out: *mut GpuTensor, + map: *const c_void, + size: u64, + sinks: u64, + q: *const GpuTensor, + raw_cache: *const GpuTensor, + rows: u32, + visible_rows: u32, + cache_cap: u32, + raw_start: u32, + heads: u32, + head_dim: u32, + ) -> i32; pub(super) fn ds4_gpu_hc_rms_scale_project_f16_tensor( out: *mut GpuTensor, scale: *mut GpuTensor, @@ -865,6 +938,14 @@ unsafe extern "C" { c: *const GpuTensor, count: u32, ) -> i32; + pub(super) fn ds4_gpu_directional_steering_project_tensor( + x: *mut GpuTensor, + directions: *const GpuTensor, + layer: u32, + width: u32, + rows: u32, + scale: f32, + ) -> i32; pub(super) fn ds4_gpu_add_rms_norm_weight_tensor( norm: *mut GpuTensor, sum: *mut GpuTensor, @@ -1043,6 +1124,7 @@ impl Context { quality: bool, ssd_streaming: bool, admission_bytes: u64, + model_spans: Option<&[(u64, u64)]>, ) -> Result { check(unsafe { ds4_gpu_init() }, "Metal initialization")?; unsafe { @@ -1059,7 +1141,19 @@ impl Context { )); } let data_offset = model.main.data_offset(); - if let Err(error) = check( + let mapped = if let Some(spans) = model_spans { + let (offsets, sizes): (Vec<_>, Vec<_>) = spans.iter().copied().unzip(); + unsafe { + ds4_gpu_set_model_map_spans( + model.main.map_ptr().cast(), + model.main.len(), + offsets.as_ptr(), + sizes.as_ptr(), + spans.len() as u32, + model.main.max_tensor_bytes(), + ) + } + } else { unsafe { ds4_gpu_set_model_map_range( model.main.map_ptr().cast(), @@ -1068,12 +1162,27 @@ impl Context { model.main.len() - data_offset, model.main.max_tensor_bytes(), ) - }, - "model mapping", - ) { + } + }; + if let Err(error) = check(mapped, "model mapping") { unsafe { ds4_gpu_cleanup() }; return Err(error); } + if let Some(support) = &model.support { + let mapped = unsafe { + ds4_gpu_set_model_map_range( + support.map_ptr().cast(), + support.len(), + support.data_offset(), + support.len() - support.data_offset(), + support.max_tensor_bytes(), + ) + }; + if let Err(error) = check(mapped, "support-model mapping") { + unsafe { ds4_gpu_cleanup() }; + return Err(error); + } + } unsafe { ds4_gpu_set_quality(quality) }; let model_file = File::open(model.main.path()).map_err(|error| { unsafe { ds4_gpu_cleanup() }; @@ -1209,6 +1318,20 @@ impl Buffer { ) } + pub(super) fn write_f32(&self, values: &[f32]) -> Result<(), String> { + check( + unsafe { + ds4_gpu_tensor_write( + self.raw(), + 0, + values.as_ptr().cast(), + std::mem::size_of_val(values) as u64, + ) + }, + "uploading floats", + ) + } + pub(super) fn fill(&self, value: f32, count: u64) -> Result<(), String> { call( unsafe { ds4_gpu_tensor_fill_f32(self.raw(), value, count) }, diff --git a/src/engine/metal/hotlist.rs b/src/engine/metal/hotlist.rs new file mode 100644 index 0000000..e68fa26 --- /dev/null +++ b/src/engine/metal/hotlist.rs @@ -0,0 +1,6439 @@ +// Generated mechanically from ../ds4/ds4_streaming_hotlist.inc. +pub(super) const FLASH: &[(u16, u16)] = &[ + (25, 53), + (22, 66), + (38, 63), + (40, 104), + (34, 44), + (28, 209), + (32, 164), + (39, 139), + (28, 79), + (38, 27), + (13, 122), + (27, 25), + (36, 106), + (26, 191), + (33, 90), + (40, 108), + (26, 88), + (41, 253), + (31, 89), + (12, 127), + (7, 170), + (17, 3), + (42, 225), + (29, 226), + (8, 128), + (36, 5), + (21, 59), + (33, 213), + (5, 42), + (18, 79), + (5, 10), + (7, 233), + (9, 191), + (36, 199), + (7, 160), + (15, 246), + (4, 233), + (34, 113), + (6, 203), + (8, 28), + (8, 189), + (11, 94), + (36, 20), + (13, 146), + (37, 55), + (15, 97), + (30, 118), + (42, 67), + (31, 43), + (24, 64), + (16, 176), + (35, 143), + (37, 32), + (40, 22), + (27, 76), + (3, 72), + (4, 26), + (6, 86), + (24, 132), + (29, 43), + (10, 150), + (18, 179), + (39, 223), + (17, 187), + (7, 141), + (4, 100), + (25, 69), + (29, 70), + (37, 142), + (30, 104), + (23, 63), + (39, 218), + (42, 234), + (31, 208), + (33, 247), + (36, 76), + (29, 247), + (32, 109), + (15, 6), + (12, 195), + (13, 145), + (11, 7), + (18, 234), + (41, 236), + (4, 139), + (24, 36), + (25, 2), + (37, 69), + (38, 56), + (27, 242), + (3, 101), + (8, 194), + (11, 33), + (11, 129), + (22, 181), + (37, 27), + (24, 110), + (34, 135), + (37, 147), + (10, 15), + (13, 144), + (14, 10), + (19, 181), + (25, 5), + (8, 200), + (3, 188), + (36, 63), + (19, 44), + (14, 1), + (20, 169), + (42, 15), + (14, 81), + (14, 120), + (40, 59), + (21, 130), + (33, 187), + (16, 14), + (20, 60), + (19, 254), + (27, 134), + (28, 126), + (18, 131), + (9, 32), + (15, 26), + (8, 106), + (26, 202), + (18, 166), + (31, 74), + (5, 149), + (13, 129), + (24, 195), + (32, 204), + (26, 253), + (14, 112), + (3, 237), + (14, 136), + (24, 180), + (28, 191), + (40, 109), + (6, 138), + (9, 106), + (12, 177), + (30, 155), + (3, 203), + (9, 186), + (14, 56), + (23, 57), + (23, 175), + (27, 10), + (28, 227), + (18, 238), + (11, 185), + (14, 118), + (26, 157), + (15, 104), + (5, 5), + (6, 233), + (10, 30), + (13, 184), + (14, 126), + (9, 149), + (16, 222), + (38, 133), + (20, 206), + (29, 96), + (3, 10), + (15, 53), + (30, 192), + (23, 253), + (26, 154), + (32, 99), + (24, 174), + (10, 89), + (11, 221), + (12, 39), + (25, 157), + (34, 176), + (7, 154), + (27, 180), + (41, 234), + (4, 97), + (17, 133), + (23, 190), + (30, 238), + (41, 209), + (9, 253), + (16, 0), + (16, 96), + (17, 228), + (22, 244), + (28, 1), + (35, 188), + (35, 162), + (36, 169), + (40, 161), + (15, 40), + (15, 173), + (26, 117), + (26, 177), + (34, 241), + (38, 210), + (28, 176), + (33, 25), + (6, 16), + (21, 170), + (24, 176), + (41, 82), + (42, 18), + (4, 43), + (8, 115), + (16, 179), + (25, 205), + (30, 248), + (34, 53), + (34, 240), + (4, 91), + (17, 120), + (27, 145), + (35, 84), + (40, 127), + (30, 199), + (20, 73), + (31, 178), + (3, 71), + (9, 12), + (35, 105), + (42, 50), + (5, 112), + (15, 215), + (28, 85), + (40, 199), + (41, 247), + (7, 86), + (17, 61), + (26, 196), + (30, 183), + (42, 242), + (6, 51), + (6, 195), + (12, 122), + (12, 229), + (12, 235), + (29, 167), + (39, 100), + (10, 19), + (14, 227), + (32, 15), + (11, 207), + (13, 175), + (15, 7), + (17, 20), + (18, 100), + (25, 143), + (5, 56), + (6, 111), + (9, 221), + (11, 58), + (14, 132), + (20, 91), + (22, 233), + (32, 79), + (3, 50), + (16, 85), + (38, 94), + (38, 213), + (12, 59), + (14, 222), + (28, 243), + (32, 215), + (41, 241), + (12, 29), + (16, 240), + (23, 36), + (23, 134), + (25, 222), + (29, 105), + (40, 121), + (42, 4), + (6, 224), + (15, 85), + (16, 229), + (17, 1), + (18, 195), + (21, 47), + (31, 32), + (32, 120), + (6, 50), + (10, 229), + (12, 90), + (13, 111), + (22, 80), + (22, 122), + (31, 137), + (31, 255), + (33, 251), + (36, 191), + (41, 91), + (15, 139), + (20, 53), + (23, 54), + (25, 81), + (26, 127), + (29, 78), + (34, 1), + (35, 247), + (37, 207), + (5, 181), + (7, 200), + (9, 161), + (24, 125), + (26, 130), + (27, 138), + (33, 56), + (7, 122), + (9, 78), + (11, 162), + (14, 187), + (16, 48), + (20, 163), + (37, 122), + (4, 38), + (6, 134), + (12, 171), + (13, 41), + (21, 201), + (27, 233), + (34, 136), + (3, 241), + (17, 125), + (22, 50), + (23, 100), + (23, 224), + (30, 163), + (34, 178), + (21, 176), + (23, 39), + (25, 17), + (34, 209), + (39, 60), + (40, 208), + (3, 176), + (10, 248), + (12, 100), + (16, 29), + (16, 193), + (19, 60), + (20, 144), + (32, 53), + (35, 0), + (37, 70), + (41, 32), + (5, 126), + (8, 168), + (12, 248), + (15, 174), + (17, 50), + (23, 71), + (23, 132), + (29, 109), + (33, 160), + (41, 235), + (42, 80), + (6, 80), + (7, 214), + (12, 68), + (13, 173), + (17, 26), + (19, 134), + (20, 88), + (21, 49), + (25, 174), + (30, 184), + (32, 68), + (32, 254), + (33, 132), + (35, 158), + (35, 240), + (41, 0), + (4, 32), + (8, 15), + (9, 122), + (11, 191), + (12, 131), + (13, 172), + (15, 249), + (16, 13), + (17, 41), + (21, 190), + (22, 197), + (23, 140), + (26, 218), + (31, 84), + (33, 147), + (41, 190), + (42, 244), + (12, 108), + (13, 165), + (18, 37), + (18, 182), + (21, 173), + (22, 16), + (27, 236), + (28, 198), + (30, 0), + (31, 55), + (33, 233), + (35, 35), + (37, 186), + (38, 255), + (39, 194), + (41, 23), + (41, 102), + (10, 216), + (12, 140), + (18, 213), + (19, 13), + (21, 72), + (21, 208), + (21, 228), + (25, 183), + (29, 235), + (30, 247), + (32, 114), + (35, 156), + (39, 70), + (41, 44), + (9, 60), + (12, 26), + (14, 153), + (15, 148), + (21, 111), + (21, 247), + (23, 2), + (23, 7), + (23, 103), + (27, 248), + (30, 170), + (32, 246), + (35, 252), + (36, 69), + (37, 6), + (38, 244), + (39, 9), + (42, 125), + (3, 102), + (6, 125), + (8, 231), + (14, 196), + (19, 222), + (21, 65), + (23, 160), + (23, 165), + (27, 77), + (28, 25), + (31, 63), + (31, 142), + (35, 48), + (35, 182), + (35, 254), + (38, 178), + (39, 180), + (41, 36), + (42, 131), + (42, 233), + (4, 137), + (7, 54), + (7, 194), + (8, 29), + (12, 8), + (13, 230), + (16, 129), + (18, 232), + (19, 202), + (20, 103), + (21, 42), + (22, 214), + (23, 119), + (30, 33), + (32, 96), + (35, 71), + (35, 110), + (3, 193), + (10, 33), + (10, 125), + (16, 16), + (19, 207), + (20, 94), + (20, 181), + (21, 151), + (21, 155), + (21, 198), + (33, 69), + (35, 206), + (41, 95), + (41, 170), + (42, 239), + (4, 39), + (5, 40), + (5, 164), + (10, 24), + (11, 59), + (12, 128), + (12, 168), + (13, 185), + (14, 97), + (16, 52), + (16, 74), + (16, 177), + (17, 112), + (17, 138), + (17, 252), + (18, 9), + (18, 60), + (19, 168), + (20, 173), + (20, 242), + (21, 91), + (21, 123), + (21, 251), + (22, 23), + (23, 240), + (24, 170), + (24, 223), + (29, 76), + (29, 159), + (29, 180), + (30, 59), + (31, 112), + (32, 231), + (34, 5), + (40, 30), + (11, 41), + (14, 82), + (14, 131), + (15, 134), + (16, 104), + (19, 212), + (19, 249), + (21, 138), + (30, 19), + (31, 161), + (31, 229), + (35, 142), + (37, 148), + (37, 253), + (39, 54), + (39, 170), + (41, 171), + (3, 108), + (4, 177), + (9, 24), + (9, 226), + (9, 242), + (11, 182), + (13, 42), + (16, 171), + (17, 84), + (17, 109), + (18, 107), + (20, 22), + (20, 200), + (20, 255), + (22, 2), + (22, 215), + (23, 172), + (29, 88), + (29, 115), + (29, 238), + (32, 136), + (33, 43), + (39, 11), + (39, 108), + (39, 221), + (40, 146), + (0, 112), + (4, 33), + (6, 253), + (11, 176), + (11, 232), + (16, 37), + (17, 204), + (18, 57), + (19, 28), + (19, 174), + (20, 116), + (22, 144), + (22, 170), + (23, 75), + (24, 198), + (28, 107), + (30, 36), + (30, 243), + (34, 180), + (35, 6), + (35, 85), + (39, 144), + (40, 12), + (41, 208), + (42, 7), + (0, 106), + (5, 186), + (7, 196), + (8, 74), + (9, 240), + (10, 84), + (11, 93), + (11, 113), + (15, 8), + (16, 209), + (17, 241), + (18, 46), + (19, 251), + (21, 121), + (26, 94), + (28, 8), + (28, 117), + (28, 151), + (30, 177), + (32, 233), + (32, 240), + (34, 93), + (34, 175), + (35, 108), + (36, 159), + (37, 13), + (38, 149), + (39, 115), + (4, 53), + (7, 43), + (7, 242), + (10, 81), + (10, 197), + (11, 23), + (14, 198), + (18, 242), + (19, 5), + (19, 47), + (19, 210), + (20, 145), + (21, 48), + (21, 135), + (22, 78), + (22, 201), + (24, 108), + (24, 231), + (25, 54), + (29, 17), + (31, 65), + (35, 179), + (36, 165), + (36, 237), + (38, 61), + (38, 162), + (40, 107), + (41, 52), + (41, 78), + (41, 239), + (4, 104), + (5, 217), + (6, 170), + (7, 76), + (8, 91), + (11, 117), + (12, 152), + (13, 14), + (14, 210), + (17, 6), + (17, 75), + (18, 160), + (19, 131), + (20, 0), + (22, 26), + (22, 91), + (25, 16), + (25, 51), + (25, 116), + (25, 195), + (27, 93), + (27, 104), + (27, 250), + (29, 205), + (30, 117), + (32, 25), + (33, 45), + (34, 6), + (39, 117), + (40, 24), + (40, 111), + (40, 145), + (41, 167), + (42, 79), + (42, 240), + (0, 86), + (1, 2), + (1, 179), + (2, 5), + (2, 68), + (2, 134), + (4, 27), + (5, 208), + (6, 29), + (6, 162), + (7, 125), + (8, 195), + (9, 90), + (10, 92), + (10, 135), + (10, 164), + (11, 105), + (14, 201), + (15, 137), + (15, 244), + (19, 182), + (19, 228), + (21, 28), + (22, 49), + (23, 35), + (24, 6), + (24, 106), + (28, 13), + (29, 24), + (29, 246), + (30, 12), + (30, 101), + (32, 147), + (33, 177), + (34, 225), + (35, 170), + (37, 48), + (38, 227), + (0, 132), + (2, 198), + (3, 61), + (3, 185), + (5, 93), + (5, 204), + (8, 141), + (8, 239), + (9, 131), + (10, 98), + (11, 6), + (14, 147), + (16, 107), + (16, 115), + (16, 120), + (17, 179), + (19, 3), + (19, 64), + (19, 211), + (21, 45), + (21, 188), + (22, 105), + (26, 233), + (28, 93), + (28, 249), + (29, 215), + (30, 111), + (33, 131), + (33, 228), + (35, 50), + (35, 74), + (36, 249), + (37, 136), + (39, 154), + (40, 132), + (41, 58), + (41, 107), + (42, 162), + (0, 115), + (4, 128), + (5, 107), + (5, 246), + (6, 3), + (6, 28), + (8, 196), + (10, 46), + (10, 140), + (10, 172), + (10, 215), + (10, 235), + (12, 49), + (15, 122), + (16, 151), + (17, 101), + (18, 45), + (18, 151), + (19, 10), + (19, 73), + (19, 110), + (20, 29), + (20, 187), + (20, 213), + (20, 229), + (21, 89), + (21, 99), + (22, 115), + (24, 11), + (24, 126), + (25, 32), + (25, 151), + (26, 35), + (26, 141), + (27, 105), + (28, 188), + (30, 194), + (30, 221), + (31, 120), + (32, 95), + (32, 213), + (34, 58), + (37, 170), + (37, 173), + (38, 221), + (39, 49), + (39, 101), + (39, 175), + (1, 96), + (2, 236), + (2, 245), + (3, 172), + (3, 173), + (4, 220), + (5, 22), + (6, 62), + (7, 71), + (8, 1), + (9, 155), + (9, 245), + (9, 246), + (10, 18), + (10, 49), + (10, 203), + (11, 19), + (13, 57), + (16, 166), + (17, 202), + (18, 98), + (20, 182), + (20, 199), + (21, 179), + (22, 34), + (22, 46), + (22, 75), + (23, 44), + (23, 108), + (26, 169), + (27, 174), + (31, 193), + (32, 154), + (32, 175), + (33, 47), + (34, 238), + (34, 246), + (35, 49), + (36, 77), + (36, 158), + (37, 114), + (37, 233), + (39, 214), + (39, 226), + (41, 117), + (41, 128), + (41, 161), + (42, 250), + (0, 44), + (0, 76), + (0, 240), + (1, 77), + (1, 245), + (2, 21), + (2, 66), + (2, 184), + (3, 15), + (4, 235), + (5, 49), + (5, 203), + (6, 1), + (7, 252), + (9, 2), + (9, 55), + (9, 251), + (11, 20), + (11, 102), + (11, 244), + (12, 25), + (12, 169), + (13, 40), + (13, 157), + (13, 159), + (14, 158), + (15, 167), + (16, 24), + (16, 147), + (17, 66), + (17, 195), + (19, 25), + (20, 251), + (21, 156), + (21, 219), + (22, 24), + (22, 83), + (22, 89), + (23, 168), + (24, 220), + (25, 63), + (26, 19), + (26, 129), + (27, 56), + (29, 55), + (29, 222), + (30, 136), + (30, 145), + (30, 228), + (31, 6), + (31, 52), + (31, 252), + (32, 121), + (32, 239), + (33, 152), + (33, 203), + (34, 68), + (34, 109), + (34, 184), + (35, 117), + (35, 137), + (35, 243), + (36, 127), + (36, 220), + (38, 47), + (42, 70), + (42, 85), + (42, 133), + (42, 159), + (42, 161), + (0, 100), + (0, 212), + (1, 20), + (1, 157), + (1, 162), + (1, 182), + (1, 216), + (2, 48), + (3, 68), + (3, 84), + (3, 142), + (3, 148), + (3, 240), + (4, 248), + (5, 220), + (5, 245), + (5, 254), + (6, 92), + (6, 168), + (7, 139), + (9, 179), + (10, 71), + (10, 149), + (11, 187), + (11, 227), + (12, 16), + (12, 123), + (12, 209), + (13, 66), + (14, 13), + (14, 184), + (16, 228), + (17, 215), + (18, 90), + (18, 143), + (19, 220), + (19, 233), + (20, 160), + (21, 248), + (22, 18), + (22, 117), + (23, 118), + (23, 208), + (24, 31), + (24, 41), + (24, 155), + (25, 118), + (25, 175), + (25, 189), + (25, 241), + (27, 141), + (27, 152), + (29, 35), + (29, 153), + (30, 75), + (31, 2), + (31, 38), + (31, 129), + (32, 104), + (32, 173), + (33, 28), + (33, 196), + (34, 8), + (34, 97), + (35, 224), + (36, 144), + (36, 154), + (37, 242), + (38, 84), + (38, 89), + (39, 59), + (39, 87), + (39, 229), + (40, 130), + (0, 165), + (1, 19), + (1, 49), + (1, 112), + (1, 197), + (2, 89), + (2, 183), + (2, 185), + (2, 207), + (5, 62), + (5, 228), + (6, 157), + (6, 205), + (6, 239), + (7, 80), + (8, 80), + (9, 80), + (9, 187), + (10, 116), + (11, 125), + (12, 15), + (12, 104), + (12, 223), + (13, 75), + (13, 116), + (14, 46), + (14, 91), + (14, 133), + (14, 245), + (16, 198), + (16, 201), + (16, 211), + (17, 219), + (18, 194), + (19, 54), + (19, 112), + (19, 218), + (20, 95), + (20, 166), + (20, 167), + (20, 178), + (20, 232), + (20, 247), + (21, 4), + (21, 34), + (22, 10), + (22, 251), + (23, 66), + (24, 62), + (24, 80), + (25, 114), + (25, 120), + (27, 211), + (28, 138), + (29, 67), + (30, 84), + (30, 106), + (30, 156), + (31, 88), + (31, 166), + (32, 118), + (32, 212), + (34, 149), + (34, 183), + (35, 201), + (36, 115), + (37, 33), + (38, 59), + (38, 218), + (38, 248), + (39, 15), + (40, 151), + (40, 166), + (40, 255), + (41, 232), + (42, 0), + (42, 149), + (0, 13), + (0, 118), + (0, 159), + (0, 178), + (1, 6), + (1, 127), + (2, 33), + (2, 141), + (2, 164), + (3, 92), + (3, 133), + (4, 249), + (5, 76), + (6, 95), + (8, 249), + (9, 18), + (9, 206), + (10, 31), + (10, 42), + (10, 47), + (10, 73), + (10, 146), + (10, 176), + (11, 3), + (11, 144), + (12, 110), + (12, 125), + (13, 28), + (14, 124), + (16, 64), + (16, 189), + (17, 38), + (18, 80), + (18, 163), + (18, 217), + (19, 43), + (19, 45), + (19, 141), + (19, 161), + (19, 180), + (21, 38), + (21, 85), + (22, 55), + (22, 74), + (22, 156), + (23, 193), + (24, 56), + (24, 156), + (24, 167), + (24, 246), + (25, 201), + (26, 28), + (26, 188), + (27, 27), + (27, 95), + (28, 125), + (29, 9), + (29, 196), + (30, 30), + (30, 31), + (30, 50), + (31, 190), + (31, 246), + (32, 252), + (33, 77), + (33, 91), + (34, 127), + (35, 146), + (35, 195), + (35, 212), + (36, 70), + (37, 115), + (37, 157), + (37, 225), + (38, 86), + (39, 21), + (39, 168), + (39, 182), + (40, 105), + (41, 166), + (42, 182), + (0, 15), + (0, 66), + (0, 224), + (1, 26), + (1, 36), + (1, 114), + (1, 151), + (1, 212), + (2, 20), + (2, 74), + (2, 108), + (2, 129), + (2, 209), + (2, 223), + (3, 4), + (3, 23), + (3, 89), + (3, 165), + (3, 198), + (3, 247), + (4, 80), + (4, 124), + (5, 101), + (5, 239), + (6, 130), + (6, 174), + (7, 15), + (7, 253), + (8, 105), + (9, 10), + (9, 49), + (9, 126), + (10, 0), + (10, 12), + (10, 108), + (10, 168), + (10, 211), + (10, 230), + (10, 250), + (10, 251), + (12, 200), + (12, 202), + (13, 217), + (14, 47), + (14, 223), + (15, 9), + (15, 241), + (16, 39), + (16, 169), + (16, 230), + (17, 121), + (17, 157), + (18, 2), + (18, 215), + (19, 11), + (19, 31), + (19, 130), + (19, 140), + (19, 223), + (19, 238), + (19, 246), + (20, 37), + (20, 64), + (20, 69), + (20, 96), + (20, 115), + (20, 118), + (20, 192), + (22, 48), + (22, 73), + (22, 107), + (22, 134), + (23, 68), + (23, 104), + (23, 242), + (24, 0), + (24, 133), + (25, 108), + (26, 192), + (27, 26), + (27, 31), + (27, 175), + (27, 219), + (29, 132), + (30, 82), + (31, 254), + (32, 22), + (32, 221), + (33, 50), + (33, 144), + (33, 146), + (34, 94), + (35, 21), + (35, 109), + (37, 91), + (37, 171), + (37, 240), + (38, 167), + (38, 168), + (38, 201), + (38, 231), + (39, 86), + (39, 129), + (39, 199), + (40, 91), + (41, 160), + (42, 36), + (42, 192), + (0, 154), + (0, 174), + (0, 211), + (0, 225), + (0, 232), + (1, 13), + (1, 75), + (1, 97), + (1, 103), + (1, 137), + (1, 163), + (1, 201), + (1, 229), + (1, 250), + (2, 40), + (2, 49), + (2, 62), + (2, 208), + (2, 247), + (3, 7), + (3, 57), + (3, 83), + (3, 244), + (4, 37), + (4, 83), + (4, 138), + (4, 192), + (4, 250), + (5, 79), + (5, 114), + (5, 175), + (6, 9), + (6, 83), + (6, 192), + (7, 61), + (7, 121), + (7, 140), + (7, 143), + (8, 67), + (9, 48), + (9, 83), + (9, 167), + (9, 172), + (9, 174), + (9, 218), + (10, 2), + (10, 41), + (10, 99), + (10, 101), + (10, 147), + (11, 2), + (11, 11), + (11, 80), + (11, 89), + (11, 110), + (11, 151), + (11, 174), + (12, 252), + (13, 104), + (13, 190), + (13, 204), + (14, 86), + (14, 94), + (15, 87), + (16, 43), + (16, 236), + (16, 245), + (17, 16), + (17, 43), + (17, 108), + (17, 188), + (19, 22), + (19, 74), + (19, 97), + (19, 160), + (20, 55), + (20, 235), + (21, 1), + (21, 11), + (21, 46), + (21, 238), + (22, 209), + (22, 220), + (22, 253), + (23, 17), + (23, 182), + (24, 63), + (24, 122), + (24, 212), + (25, 43), + (26, 18), + (26, 197), + (27, 100), + (27, 203), + (27, 227), + (28, 28), + (29, 31), + (30, 35), + (30, 123), + (30, 222), + (30, 236), + (31, 54), + (31, 62), + (31, 143), + (31, 181), + (31, 226), + (32, 24), + (32, 160), + (32, 237), + (33, 54), + (34, 29), + (34, 85), + (37, 102), + (37, 155), + (37, 239), + (38, 165), + (39, 85), + (39, 150), + (39, 234), + (39, 242), + (40, 76), + (40, 114), + (41, 97), + (41, 141), + (41, 192), + (41, 238), + (42, 34), + (42, 81), + (42, 140), + (0, 1), + (0, 62), + (0, 116), + (0, 117), + (0, 126), + (0, 139), + (0, 218), + (0, 222), + (0, 230), + (0, 231), + (0, 242), + (1, 76), + (1, 153), + (1, 155), + (1, 178), + (1, 192), + (1, 193), + (1, 249), + (2, 15), + (2, 59), + (2, 87), + (2, 93), + (2, 112), + (2, 115), + (2, 153), + (2, 167), + (2, 172), + (4, 15), + (4, 73), + (5, 29), + (5, 115), + (5, 160), + (5, 209), + (5, 224), + (6, 31), + (6, 58), + (6, 129), + (6, 165), + (6, 166), + (7, 51), + (7, 168), + (7, 226), + (9, 44), + (9, 96), + (9, 166), + (10, 130), + (10, 156), + (10, 210), + (10, 217), + (11, 24), + (11, 79), + (11, 138), + (11, 224), + (12, 142), + (12, 196), + (12, 246), + (13, 96), + (13, 139), + (14, 11), + (14, 138), + (14, 235), + (15, 60), + (15, 176), + (16, 119), + (17, 69), + (17, 127), + (17, 128), + (17, 134), + (17, 208), + (17, 224), + (18, 85), + (18, 88), + (18, 187), + (19, 83), + (19, 102), + (19, 193), + (19, 197), + (19, 253), + (20, 32), + (20, 49), + (20, 98), + (20, 112), + (20, 124), + (20, 129), + (20, 130), + (20, 134), + (20, 184), + (21, 6), + (21, 249), + (22, 99), + (22, 250), + (23, 95), + (24, 33), + (24, 118), + (24, 240), + (25, 26), + (25, 228), + (26, 178), + (26, 209), + (26, 215), + (27, 200), + (28, 29), + (28, 71), + (28, 158), + (28, 162), + (29, 243), + (30, 94), + (30, 141), + (30, 173), + (31, 27), + (31, 57), + (31, 76), + (32, 12), + (32, 170), + (32, 188), + (32, 219), + (33, 112), + (33, 128), + (34, 13), + (34, 107), + (34, 164), + (34, 250), + (35, 2), + (35, 63), + (35, 111), + (35, 126), + (35, 133), + (35, 216), + (36, 18), + (37, 94), + (38, 105), + (39, 90), + (39, 207), + (39, 220), + (42, 30), + (42, 98), + (42, 232), + (0, 69), + (0, 89), + (0, 107), + (0, 111), + (0, 113), + (0, 123), + (0, 129), + (0, 210), + (0, 250), + (1, 7), + (1, 51), + (1, 68), + (1, 69), + (1, 84), + (1, 118), + (1, 187), + (1, 224), + (1, 230), + (1, 231), + (1, 232), + (1, 251), + (2, 7), + (2, 12), + (2, 22), + (2, 55), + (2, 119), + (2, 128), + (2, 137), + (2, 146), + (2, 173), + (2, 193), + (2, 212), + (3, 103), + (3, 105), + (3, 107), + (3, 223), + (4, 45), + (4, 94), + (4, 95), + (4, 164), + (4, 226), + (5, 45), + (5, 85), + (5, 132), + (6, 15), + (6, 91), + (6, 112), + (6, 121), + (6, 132), + (6, 194), + (7, 74), + (7, 221), + (7, 239), + (8, 24), + (8, 45), + (8, 146), + (9, 13), + (9, 105), + (9, 198), + (10, 8), + (10, 234), + (11, 84), + (11, 169), + (12, 215), + (13, 94), + (13, 95), + (13, 115), + (13, 123), + (15, 43), + (15, 105), + (15, 217), + (15, 218), + (16, 46), + (16, 173), + (16, 192), + (16, 231), + (17, 123), + (17, 166), + (17, 168), + (18, 141), + (19, 8), + (19, 93), + (19, 209), + (19, 216), + (20, 77), + (20, 93), + (20, 122), + (20, 219), + (20, 220), + (20, 252), + (22, 51), + (22, 52), + (22, 61), + (22, 64), + (22, 157), + (22, 190), + (23, 21), + (23, 23), + (23, 64), + (23, 73), + (23, 125), + (23, 142), + (23, 159), + (23, 221), + (23, 249), + (24, 218), + (25, 71), + (25, 212), + (26, 54), + (26, 131), + (26, 179), + (26, 228), + (26, 245), + (27, 37), + (27, 73), + (27, 80), + (27, 133), + (27, 163), + (27, 224), + (27, 247), + (27, 254), + (29, 4), + (29, 60), + (29, 85), + (29, 93), + (29, 240), + (30, 7), + (31, 21), + (31, 80), + (31, 102), + (31, 144), + (31, 156), + (31, 169), + (31, 223), + (32, 2), + (33, 29), + (33, 38), + (33, 116), + (33, 181), + (33, 216), + (33, 249), + (34, 242), + (34, 247), + (35, 29), + (35, 57), + (35, 61), + (35, 113), + (35, 144), + (36, 153), + (37, 162), + (38, 34), + (38, 80), + (38, 148), + (38, 253), + (40, 54), + (41, 63), + (42, 73), + (42, 112), + (42, 117), + (42, 189), + (0, 2), + (0, 29), + (0, 40), + (0, 51), + (0, 77), + (0, 168), + (0, 190), + (0, 195), + (0, 200), + (0, 204), + (0, 205), + (0, 248), + (1, 32), + (1, 52), + (1, 78), + (1, 85), + (1, 99), + (1, 122), + (1, 133), + (1, 140), + (1, 154), + (1, 175), + (1, 185), + (1, 210), + (2, 13), + (2, 19), + (2, 32), + (2, 39), + (2, 61), + (2, 76), + (2, 113), + (2, 120), + (2, 174), + (2, 181), + (2, 195), + (2, 231), + (2, 238), + (3, 0), + (3, 14), + (3, 17), + (3, 26), + (3, 139), + (4, 56), + (4, 68), + (4, 112), + (4, 114), + (4, 209), + (4, 216), + (5, 1), + (5, 47), + (5, 242), + (6, 254), + (7, 30), + (7, 32), + (7, 181), + (7, 188), + (7, 235), + (8, 13), + (8, 161), + (9, 53), + (9, 81), + (9, 118), + (9, 133), + (9, 181), + (9, 219), + (10, 7), + (10, 40), + (10, 44), + (10, 142), + (10, 162), + (10, 178), + (10, 214), + (10, 232), + (11, 30), + (11, 88), + (11, 243), + (12, 64), + (12, 220), + (12, 239), + (13, 63), + (13, 127), + (13, 197), + (13, 252), + (14, 6), + (14, 73), + (14, 205), + (15, 10), + (15, 66), + (15, 82), + (15, 130), + (15, 196), + (15, 213), + (16, 102), + (16, 105), + (16, 137), + (16, 139), + (16, 181), + (16, 191), + (17, 117), + (18, 6), + (18, 32), + (18, 128), + (18, 157), + (18, 173), + (18, 233), + (19, 103), + (19, 108), + (19, 123), + (19, 148), + (19, 158), + (19, 170), + (20, 12), + (20, 48), + (20, 162), + (20, 202), + (21, 26), + (21, 78), + (21, 137), + (22, 27), + (22, 62), + (23, 43), + (23, 60), + (23, 115), + (23, 173), + (24, 48), + (24, 59), + (24, 77), + (24, 96), + (24, 134), + (24, 166), + (24, 222), + (25, 65), + (25, 103), + (25, 197), + (26, 71), + (26, 213), + (27, 14), + (27, 102), + (27, 235), + (28, 19), + (28, 51), + (28, 67), + (29, 12), + (29, 44), + (29, 94), + (29, 126), + (29, 165), + (30, 41), + (30, 105), + (30, 200), + (30, 251), + (31, 34), + (32, 140), + (32, 158), + (32, 172), + (33, 7), + (33, 104), + (33, 118), + (33, 136), + (33, 209), + (34, 69), + (34, 249), + (35, 51), + (35, 124), + (35, 127), + (35, 229), + (36, 126), + (37, 2), + (37, 112), + (37, 152), + (37, 159), + (37, 175), + (38, 30), + (38, 33), + (38, 145), + (38, 216), + (39, 45), + (39, 89), + (39, 140), + (39, 146), + (39, 165), + (39, 177), + (39, 181), + (39, 208), + (40, 135), + (41, 140), + (41, 197), + (41, 214), + (42, 14), + (42, 107), + (42, 165), + (0, 4), + (0, 6), + (0, 14), + (0, 18), + (0, 19), + (0, 25), + (0, 27), + (0, 54), + (0, 58), + (0, 60), + (0, 63), + (0, 74), + (0, 75), + (0, 79), + (0, 85), + (0, 91), + (0, 98), + (0, 105), + (0, 122), + (0, 131), + (0, 150), + (0, 152), + (0, 177), + (0, 184), + (0, 189), + (0, 194), + (0, 202), + (0, 229), + (0, 236), + (1, 4), + (1, 5), + (1, 21), + (1, 25), + (1, 29), + (1, 30), + (1, 47), + (1, 58), + (1, 65), + (1, 79), + (1, 82), + (1, 86), + (1, 91), + (1, 110), + (1, 116), + (1, 120), + (1, 136), + (1, 141), + (1, 161), + (1, 164), + (1, 169), + (1, 183), + (1, 217), + (2, 6), + (2, 10), + (2, 36), + (2, 58), + (2, 63), + (2, 77), + (2, 101), + (2, 133), + (2, 199), + (2, 227), + (2, 233), + (2, 239), + (2, 251), + (3, 3), + (3, 109), + (4, 21), + (4, 69), + (4, 136), + (4, 150), + (4, 165), + (4, 194), + (4, 242), + (4, 251), + (5, 83), + (5, 133), + (5, 150), + (5, 155), + (5, 161), + (5, 176), + (5, 183), + (6, 103), + (6, 120), + (6, 137), + (7, 23), + (7, 28), + (7, 90), + (7, 182), + (7, 185), + (7, 187), + (8, 51), + (8, 81), + (8, 103), + (8, 118), + (8, 180), + (8, 210), + (8, 213), + (9, 58), + (9, 140), + (9, 162), + (11, 26), + (11, 96), + (11, 141), + (11, 193), + (11, 204), + (12, 188), + (13, 12), + (13, 222), + (13, 225), + (14, 15), + (14, 139), + (14, 206), + (15, 45), + (15, 59), + (15, 64), + (15, 119), + (15, 141), + (15, 151), + (15, 240), + (16, 59), + (16, 110), + (17, 71), + (17, 76), + (17, 124), + (17, 177), + (17, 205), + (17, 234), + (17, 243), + (18, 39), + (18, 198), + (19, 68), + (19, 149), + (19, 189), + (19, 203), + (20, 7), + (20, 74), + (20, 110), + (20, 216), + (21, 50), + (21, 234), + (21, 239), + (21, 245), + (22, 68), + (22, 102), + (22, 169), + (23, 107), + (23, 167), + (24, 40), + (24, 58), + (24, 150), + (24, 153), + (25, 4), + (25, 135), + (26, 3), + (26, 16), + (26, 142), + (26, 204), + (27, 94), + (27, 106), + (27, 143), + (27, 193), + (27, 209), + (28, 146), + (29, 28), + (29, 68), + (29, 182), + (29, 186), + (29, 195), + (29, 239), + (29, 249), + (30, 18), + (30, 45), + (31, 109), + (31, 133), + (31, 238), + (32, 33), + (32, 86), + (32, 113), + (32, 131), + (32, 146), + (32, 168), + (33, 19), + (33, 75), + (33, 166), + (34, 233), + (35, 119), + (35, 138), + (35, 150), + (35, 198), + (35, 202), + (35, 210), + (36, 172), + (37, 1), + (37, 12), + (37, 15), + (37, 57), + (37, 82), + (37, 123), + (37, 191), + (37, 201), + (37, 213), + (37, 254), + (38, 3), + (38, 13), + (39, 55), + (39, 149), + (39, 200), + (39, 203), + (40, 242), + (41, 5), + (41, 54), + (41, 59), + (41, 81), + (41, 127), + (41, 129), + (41, 153), + (42, 21), + (42, 64), + (42, 78), + (42, 224), + (0, 16), + (0, 24), + (0, 30), + (0, 39), + (0, 42), + (0, 46), + (0, 70), + (0, 95), + (0, 96), + (0, 97), + (0, 120), + (0, 127), + (0, 135), + (0, 166), + (0, 186), + (0, 188), + (0, 245), + (1, 18), + (1, 31), + (1, 35), + (1, 48), + (1, 64), + (1, 72), + (1, 80), + (1, 100), + (1, 111), + (1, 138), + (1, 142), + (1, 172), + (1, 186), + (1, 196), + (1, 255), + (2, 0), + (2, 4), + (2, 8), + (2, 11), + (2, 31), + (2, 34), + (2, 44), + (2, 52), + (2, 60), + (2, 79), + (2, 80), + (2, 106), + (2, 122), + (2, 131), + (2, 135), + (2, 140), + (2, 156), + (2, 160), + (2, 182), + (2, 214), + (2, 219), + (2, 230), + (2, 242), + (3, 13), + (3, 18), + (3, 36), + (3, 123), + (3, 200), + (3, 227), + (3, 233), + (3, 252), + (4, 81), + (4, 127), + (4, 131), + (4, 154), + (4, 175), + (4, 186), + (4, 189), + (4, 213), + (4, 236), + (5, 43), + (5, 64), + (5, 94), + (5, 154), + (5, 159), + (5, 171), + (5, 193), + (5, 222), + (5, 237), + (6, 117), + (6, 139), + (6, 144), + (6, 149), + (6, 164), + (6, 221), + (7, 78), + (7, 103), + (7, 209), + (8, 63), + (8, 134), + (8, 218), + (8, 219), + (8, 225), + (8, 228), + (9, 36), + (9, 42), + (9, 99), + (9, 100), + (9, 130), + (9, 176), + (9, 196), + (10, 21), + (10, 70), + (10, 72), + (10, 138), + (10, 196), + (11, 98), + (11, 168), + (11, 194), + (12, 92), + (12, 97), + (13, 10), + (13, 24), + (13, 151), + (14, 27), + (14, 49), + (14, 123), + (14, 162), + (14, 178), + (15, 37), + (15, 165), + (16, 70), + (16, 77), + (16, 135), + (16, 145), + (16, 170), + (17, 12), + (17, 183), + (18, 25), + (18, 28), + (18, 84), + (18, 110), + (18, 189), + (18, 199), + (18, 224), + (19, 16), + (19, 91), + (19, 100), + (19, 194), + (19, 215), + (20, 4), + (20, 8), + (20, 15), + (20, 27), + (20, 75), + (20, 85), + (20, 86), + (20, 154), + (20, 157), + (20, 164), + (20, 171), + (21, 115), + (21, 129), + (21, 183), + (21, 202), + (21, 203), + (21, 254), + (22, 135), + (22, 241), + (23, 31), + (23, 225), + (24, 2), + (24, 15), + (24, 60), + (24, 121), + (24, 138), + (25, 36), + (25, 38), + (25, 244), + (26, 136), + (26, 247), + (26, 255), + (27, 24), + (27, 131), + (27, 155), + (27, 218), + (27, 232), + (28, 31), + (28, 94), + (28, 157), + (28, 167), + (28, 177), + (28, 232), + (28, 234), + (29, 69), + (29, 98), + (29, 157), + (29, 251), + (30, 39), + (30, 74), + (30, 214), + (30, 229), + (31, 67), + (31, 125), + (31, 211), + (32, 8), + (32, 37), + (32, 102), + (32, 142), + (32, 162), + (33, 86), + (33, 102), + (33, 122), + (33, 161), + (33, 218), + (34, 33), + (34, 37), + (34, 83), + (34, 98), + (34, 114), + (34, 144), + (34, 253), + (35, 86), + (35, 115), + (35, 134), + (35, 177), + (35, 178), + (35, 185), + (35, 207), + (35, 217), + (35, 245), + (36, 118), + (36, 198), + (37, 89), + (37, 120), + (37, 181), + (38, 10), + (38, 38), + (38, 41), + (38, 67), + (38, 230), + (39, 95), + (39, 142), + (39, 178), + (39, 230), + (40, 4), + (40, 37), + (40, 93), + (40, 194), + (40, 237), + (40, 239), + (41, 12), + (41, 60), + (41, 134), + (41, 138), + (42, 122), + (42, 136), + (42, 137), + (42, 172), + (42, 202), + (42, 236), + (0, 5), + (0, 8), + (0, 10), + (0, 41), + (0, 48), + (0, 49), + (0, 52), + (0, 68), + (0, 71), + (0, 99), + (0, 102), + (0, 114), + (0, 121), + (0, 133), + (0, 147), + (0, 151), + (0, 153), + (0, 164), + (0, 179), + (0, 180), + (0, 181), + (0, 185), + (0, 207), + (0, 239), + (1, 8), + (1, 12), + (1, 34), + (1, 38), + (1, 54), + (1, 56), + (1, 95), + (1, 105), + (1, 107), + (1, 113), + (1, 117), + (1, 143), + (1, 160), + (1, 184), + (1, 207), + (1, 211), + (1, 226), + (1, 236), + (2, 17), + (2, 30), + (2, 41), + (2, 43), + (2, 46), + (2, 64), + (2, 85), + (2, 86), + (2, 92), + (2, 107), + (2, 109), + (2, 114), + (2, 118), + (2, 125), + (2, 166), + (2, 178), + (2, 202), + (2, 224), + (3, 11), + (3, 20), + (3, 31), + (3, 34), + (3, 114), + (3, 121), + (3, 152), + (3, 174), + (3, 205), + (3, 212), + (3, 234), + (3, 248), + (4, 55), + (4, 106), + (4, 118), + (4, 153), + (5, 48), + (5, 145), + (5, 169), + (5, 180), + (5, 205), + (5, 207), + (5, 226), + (6, 13), + (6, 20), + (6, 55), + (6, 127), + (6, 198), + (6, 207), + (6, 236), + (7, 47), + (7, 136), + (8, 2), + (8, 46), + (8, 85), + (8, 88), + (8, 90), + (8, 172), + (8, 191), + (8, 253), + (8, 254), + (9, 14), + (9, 52), + (9, 112), + (9, 148), + (9, 180), + (9, 192), + (9, 252), + (10, 9), + (10, 74), + (10, 85), + (10, 94), + (10, 103), + (10, 109), + (10, 245), + (11, 1), + (11, 40), + (11, 101), + (11, 165), + (11, 205), + (11, 217), + (11, 223), + (12, 109), + (12, 167), + (12, 185), + (12, 210), + (13, 33), + (13, 180), + (13, 201), + (13, 237), + (14, 99), + (14, 100), + (14, 137), + (14, 141), + (14, 236), + (15, 25), + (15, 63), + (15, 121), + (15, 189), + (16, 5), + (16, 34), + (16, 109), + (16, 239), + (16, 255), + (17, 165), + (17, 171), + (17, 175), + (17, 198), + (17, 236), + (18, 27), + (18, 124), + (18, 223), + (19, 30), + (19, 77), + (19, 86), + (19, 98), + (19, 154), + (20, 36), + (20, 43), + (20, 109), + (20, 114), + (21, 124), + (22, 3), + (22, 31), + (22, 33), + (22, 176), + (22, 218), + (22, 219), + (22, 232), + (23, 13), + (23, 25), + (23, 81), + (23, 96), + (23, 149), + (23, 177), + (24, 28), + (24, 29), + (24, 86), + (24, 120), + (24, 173), + (24, 193), + (24, 225), + (25, 31), + (25, 89), + (25, 137), + (25, 171), + (25, 194), + (25, 206), + (25, 209), + (26, 77), + (26, 91), + (26, 162), + (26, 254), + (27, 39), + (27, 88), + (27, 92), + (27, 114), + (27, 116), + (28, 14), + (28, 112), + (28, 121), + (29, 32), + (29, 101), + (29, 110), + (29, 130), + (29, 136), + (29, 255), + (30, 8), + (30, 21), + (30, 95), + (30, 211), + (30, 212), + (30, 240), + (31, 94), + (31, 99), + (31, 160), + (32, 116), + (32, 205), + (33, 12), + (33, 18), + (33, 21), + (33, 191), + (33, 229), + (34, 110), + (34, 188), + (34, 203), + (34, 204), + (35, 47), + (35, 53), + (35, 64), + (35, 90), + (35, 130), + (35, 253), + (36, 4), + (36, 17), + (36, 67), + (36, 116), + (36, 182), + (36, 203), + (37, 108), + (37, 221), + (38, 32), + (38, 124), + (38, 143), + (38, 179), + (38, 184), + (39, 33), + (39, 37), + (39, 38), + (39, 78), + (39, 80), + (39, 227), + (40, 41), + (40, 224), + (41, 3), + (41, 21), + (42, 58), + (42, 92), + (42, 209), + (0, 20), + (0, 28), + (0, 31), + (0, 38), + (0, 50), + (0, 57), + (0, 61), + (0, 83), + (0, 119), + (0, 124), + (0, 138), + (0, 145), + (0, 156), + (0, 162), + (0, 169), + (0, 173), + (0, 183), + (0, 187), + (0, 192), + (0, 199), + (0, 208), + (0, 217), + (0, 241), + (0, 249), + (0, 252), + (1, 0), + (1, 3), + (1, 9), + (1, 11), + (1, 15), + (1, 22), + (1, 27), + (1, 39), + (1, 59), + (1, 93), + (1, 102), + (1, 135), + (1, 149), + (1, 150), + (1, 167), + (1, 190), + (1, 203), + (1, 205), + (1, 214), + (1, 220), + (1, 221), + (1, 223), + (1, 227), + (1, 239), + (1, 252), + (1, 253), + (2, 3), + (2, 14), + (2, 18), + (2, 24), + (2, 28), + (2, 37), + (2, 83), + (2, 94), + (2, 97), + (2, 99), + (2, 105), + (2, 110), + (2, 117), + (2, 124), + (2, 126), + (2, 144), + (2, 148), + (2, 149), + (2, 152), + (2, 168), + (2, 175), + (2, 176), + (2, 204), + (2, 206), + (2, 235), + (2, 248), + (2, 252), + (2, 255), + (3, 54), + (3, 99), + (3, 106), + (3, 115), + (3, 145), + (3, 187), + (3, 218), + (3, 230), + (3, 232), + (4, 63), + (4, 82), + (4, 199), + (4, 231), + (4, 237), + (4, 255), + (5, 2), + (5, 31), + (5, 38), + (5, 95), + (5, 125), + (5, 148), + (5, 230), + (5, 243), + (5, 250), + (6, 4), + (6, 27), + (6, 147), + (6, 190), + (6, 208), + (6, 226), + (6, 229), + (6, 238), + (7, 7), + (7, 39), + (7, 129), + (7, 135), + (7, 156), + (8, 10), + (8, 42), + (8, 82), + (8, 135), + (8, 229), + (8, 248), + (9, 108), + (9, 135), + (9, 141), + (9, 150), + (9, 178), + (9, 193), + (9, 212), + (9, 227), + (10, 23), + (10, 51), + (10, 68), + (10, 119), + (10, 129), + (10, 180), + (10, 182), + (10, 207), + (10, 239), + (10, 255), + (11, 91), + (11, 119), + (11, 179), + (12, 23), + (12, 144), + (12, 162), + (12, 211), + (12, 222), + (13, 15), + (13, 20), + (13, 114), + (13, 171), + (13, 245), + (14, 55), + (14, 76), + (14, 113), + (14, 134), + (14, 173), + (14, 209), + (15, 17), + (15, 21), + (15, 50), + (15, 94), + (15, 123), + (15, 142), + (16, 69), + (16, 132), + (16, 152), + (16, 190), + (16, 206), + (16, 224), + (16, 251), + (17, 77), + (17, 81), + (17, 83), + (17, 87), + (17, 106), + (17, 126), + (17, 164), + (17, 235), + (18, 26), + (18, 38), + (18, 58), + (18, 61), + (18, 67), + (18, 97), + (18, 126), + (18, 145), + (18, 150), + (18, 167), + (18, 171), + (19, 40), + (19, 53), + (19, 229), + (20, 65), + (20, 105), + (20, 147), + (20, 174), + (20, 205), + (20, 207), + (20, 236), + (20, 254), + (21, 60), + (21, 71), + (21, 94), + (21, 101), + (21, 108), + (21, 159), + (21, 168), + (21, 211), + (22, 21), + (22, 145), + (22, 162), + (22, 174), + (23, 12), + (23, 70), + (23, 161), + (23, 197), + (23, 220), + (24, 14), + (24, 19), + (24, 26), + (24, 101), + (24, 112), + (24, 139), + (24, 140), + (24, 154), + (24, 171), + (24, 184), + (24, 248), + (24, 250), + (25, 1), + (25, 8), + (25, 167), + (25, 187), + (25, 198), + (25, 214), + (25, 226), + (26, 56), + (26, 79), + (26, 102), + (26, 112), + (26, 214), + (26, 223), + (27, 63), + (27, 140), + (27, 153), + (27, 168), + (27, 179), + (27, 221), + (28, 37), + (28, 88), + (28, 129), + (28, 179), + (28, 219), + (28, 241), + (29, 75), + (29, 158), + (29, 197), + (29, 223), + (30, 58), + (30, 140), + (30, 146), + (30, 167), + (31, 33), + (31, 39), + (31, 47), + (31, 73), + (31, 113), + (31, 114), + (31, 134), + (31, 212), + (31, 216), + (31, 235), + (31, 240), + (31, 249), + (32, 5), + (32, 45), + (32, 71), + (32, 124), + (32, 196), + (32, 220), + (33, 27), + (33, 64), + (33, 111), + (33, 117), + (33, 186), + (33, 253), + (34, 159), + (34, 179), + (34, 181), + (34, 195), + (34, 244), + (35, 5), + (35, 8), + (35, 96), + (35, 116), + (35, 120), + (35, 129), + (35, 136), + (35, 204), + (35, 244), + (36, 100), + (36, 194), + (36, 248), + (37, 43), + (37, 45), + (38, 21), + (38, 72), + (38, 73), + (38, 97), + (38, 107), + (38, 110), + (38, 173), + (38, 174), + (38, 188), + (38, 190), + (38, 203), + (38, 224), + (38, 234), + (38, 249), + (39, 6), + (39, 43), + (39, 84), + (39, 138), + (39, 163), + (40, 92), + (40, 113), + (40, 174), + (40, 175), + (40, 180), + (40, 213), + (41, 48), + (41, 66), + (41, 70), + (41, 186), + (41, 205), + (42, 46), + (42, 71), + (42, 138), + (42, 200), + (42, 203), + (42, 211), + (42, 227), + (0, 3), + (0, 9), + (0, 17), + (0, 35), + (0, 36), + (0, 55), + (0, 78), + (0, 88), + (0, 104), + (0, 108), + (0, 134), + (0, 137), + (0, 144), + (0, 146), + (0, 198), + (0, 206), + (0, 209), + (0, 213), + (0, 219), + (0, 251), + (0, 255), + (1, 17), + (1, 33), + (1, 37), + (1, 43), + (1, 44), + (1, 53), + (1, 57), + (1, 66), + (1, 70), + (1, 73), + (1, 74), + (1, 83), + (1, 87), + (1, 123), + (1, 128), + (1, 132), + (1, 146), + (1, 158), + (1, 176), + (1, 180), + (1, 181), + (1, 195), + (1, 206), + (1, 209), + (1, 218), + (1, 219), + (1, 225), + (1, 234), + (1, 238), + (1, 240), + (2, 1), + (2, 23), + (2, 26), + (2, 29), + (2, 38), + (2, 45), + (2, 57), + (2, 67), + (2, 75), + (2, 84), + (2, 96), + (2, 116), + (2, 132), + (2, 139), + (2, 150), + (2, 151), + (2, 158), + (2, 170), + (2, 177), + (2, 186), + (2, 197), + (2, 200), + (2, 203), + (2, 215), + (2, 225), + (2, 234), + (2, 254), + (3, 27), + (3, 32), + (3, 90), + (3, 127), + (3, 144), + (3, 196), + (3, 201), + (3, 204), + (3, 221), + (3, 246), + (3, 253), + (4, 29), + (4, 98), + (4, 102), + (4, 144), + (4, 147), + (4, 156), + (4, 163), + (4, 181), + (4, 210), + (4, 230), + (4, 252), + (5, 19), + (5, 55), + (5, 58), + (5, 65), + (5, 67), + (5, 90), + (5, 172), + (5, 177), + (5, 201), + (5, 232), + (6, 64), + (6, 90), + (6, 177), + (6, 200), + (6, 219), + (6, 228), + (6, 230), + (6, 248), + (7, 21), + (7, 59), + (7, 69), + (7, 110), + (7, 157), + (7, 174), + (7, 177), + (7, 230), + (7, 234), + (7, 245), + (8, 101), + (8, 131), + (8, 152), + (8, 177), + (8, 202), + (8, 241), + (8, 247), + (9, 9), + (9, 102), + (9, 128), + (9, 145), + (9, 200), + (10, 11), + (10, 37), + (10, 50), + (10, 112), + (10, 124), + (10, 133), + (10, 154), + (10, 166), + (11, 64), + (11, 121), + (11, 135), + (11, 199), + (12, 21), + (12, 63), + (12, 65), + (12, 82), + (12, 136), + (12, 137), + (12, 183), + (12, 236), + (13, 37), + (13, 83), + (13, 103), + (13, 124), + (13, 195), + (14, 8), + (14, 23), + (14, 26), + (14, 78), + (15, 27), + (15, 69), + (15, 153), + (15, 201), + (15, 203), + (15, 223), + (15, 225), + (15, 237), + (15, 248), + (16, 57), + (17, 70), + (17, 80), + (17, 96), + (17, 137), + (17, 144), + (17, 152), + (17, 207), + (17, 217), + (18, 99), + (18, 135), + (18, 178), + (18, 183), + (18, 216), + (18, 252), + (19, 17), + (19, 19), + (19, 38), + (19, 85), + (19, 101), + (19, 105), + (19, 143), + (19, 155), + (19, 178), + (19, 208), + (19, 224), + (20, 101), + (20, 128), + (20, 140), + (20, 159), + (20, 190), + (20, 243), + (20, 250), + (21, 44), + (21, 58), + (21, 75), + (21, 83), + (21, 93), + (21, 172), + (21, 193), + (21, 194), + (21, 217), + (21, 221), + (21, 225), + (21, 252), + (22, 42), + (22, 149), + (22, 159), + (22, 193), + (22, 206), + (22, 208), + (22, 229), + (22, 230), + (22, 239), + (22, 254), + (23, 10), + (23, 45), + (23, 65), + (23, 92), + (23, 235), + (24, 57), + (24, 65), + (24, 87), + (24, 161), + (24, 196), + (24, 209), + (25, 68), + (25, 83), + (25, 88), + (25, 104), + (25, 124), + (25, 128), + (25, 148), + (25, 176), + (25, 207), + (26, 32), + (26, 113), + (26, 132), + (26, 167), + (26, 211), + (26, 219), + (27, 35), + (27, 126), + (27, 167), + (27, 196), + (27, 228), + (28, 4), + (28, 9), + (28, 20), + (28, 34), + (28, 142), + (28, 197), + (28, 201), + (28, 211), + (28, 222), + (29, 46), + (29, 74), + (29, 87), + (29, 171), + (29, 200), + (29, 241), + (30, 2), + (30, 6), + (30, 23), + (30, 66), + (30, 125), + (30, 132), + (30, 168), + (30, 195), + (30, 213), + (30, 233), + (30, 253), + (31, 19), + (31, 44), + (31, 92), + (31, 189), + (31, 244), + (32, 6), + (32, 100), + (32, 112), + (32, 180), + (32, 183), + (32, 206), + (32, 209), + (32, 234), + (32, 250), + (33, 61), + (33, 65), + (33, 115), + (33, 120), + (33, 140), + (33, 156), + (33, 190), + (33, 226), + (34, 18), + (34, 19), + (34, 54), + (34, 134), + (34, 160), + (34, 214), + (35, 28), + (35, 31), + (35, 88), + (35, 95), + (35, 160), + (35, 161), + (35, 166), + (35, 168), + (35, 175), + (36, 33), + (36, 47), + (36, 48), + (36, 92), + (36, 102), + (36, 162), + (36, 174), + (36, 187), + (36, 197), + (36, 201), + (36, 251), + (37, 60), + (37, 76), + (37, 80), + (37, 87), + (37, 153), + (37, 160), + (37, 205), + (37, 216), + (38, 76), + (38, 108), + (38, 117), + (38, 126), + (38, 144), + (38, 169), + (38, 176), + (38, 187), + (38, 197), + (38, 208), + (38, 242), + (39, 14), + (39, 26), + (39, 132), + (39, 162), + (39, 237), + (40, 7), + (40, 197), + (40, 205), + (40, 225), + (40, 238), + (41, 131), + (41, 143), + (41, 151), + (41, 251), + (41, 255), + (42, 135), + (42, 178), + (42, 197), + (42, 219), + (0, 0), + (0, 7), + (0, 23), + (0, 37), + (0, 43), + (0, 56), + (0, 65), + (0, 81), + (0, 82), + (0, 84), + (0, 87), + (0, 93), + (0, 94), + (0, 130), + (0, 141), + (0, 143), + (0, 149), + (0, 155), + (0, 157), + (0, 163), + (0, 170), + (0, 171), + (0, 175), + (0, 176), + (0, 182), + (0, 203), + (0, 214), + (0, 215), + (0, 253), + (1, 41), + (1, 45), + (1, 50), + (1, 60), + (1, 62), + (1, 67), + (1, 81), + (1, 106), + (1, 109), + (1, 115), + (1, 119), + (1, 126), + (1, 139), + (1, 144), + (1, 145), + (1, 147), + (1, 156), + (1, 168), + (1, 170), + (1, 173), + (1, 204), + (1, 208), + (1, 213), + (1, 228), + (1, 235), + (1, 242), + (1, 243), + (1, 247), + (1, 248), + (1, 254), + (2, 2), + (2, 27), + (2, 35), + (2, 42), + (2, 50), + (2, 51), + (2, 54), + (2, 73), + (2, 81), + (2, 82), + (2, 91), + (2, 95), + (2, 100), + (2, 102), + (2, 121), + (2, 130), + (2, 145), + (2, 155), + (2, 157), + (2, 159), + (2, 161), + (2, 163), + (2, 169), + (2, 179), + (2, 180), + (2, 187), + (2, 188), + (2, 191), + (2, 192), + (2, 196), + (2, 210), + (2, 229), + (3, 77), + (3, 98), + (3, 116), + (3, 122), + (3, 136), + (3, 140), + (3, 175), + (3, 209), + (3, 236), + (3, 251), + (4, 13), + (4, 22), + (4, 75), + (4, 120), + (4, 148), + (5, 15), + (5, 21), + (5, 32), + (5, 61), + (5, 80), + (5, 89), + (5, 104), + (5, 111), + (5, 130), + (5, 191), + (5, 197), + (5, 200), + (5, 241), + (5, 249), + (6, 5), + (6, 25), + (6, 67), + (6, 124), + (6, 136), + (6, 196), + (6, 201), + (6, 209), + (6, 212), + (6, 218), + (6, 222), + (6, 223), + (6, 241), + (6, 244), + (6, 246), + (7, 117), + (7, 166), + (7, 216), + (7, 236), + (7, 248), + (8, 35), + (8, 36), + (8, 38), + (8, 58), + (8, 111), + (8, 112), + (8, 137), + (8, 158), + (8, 192), + (9, 28), + (9, 54), + (9, 175), + (9, 209), + (9, 213), + (9, 228), + (9, 239), + (10, 22), + (10, 25), + (10, 48), + (10, 75), + (10, 82), + (10, 88), + (10, 111), + (10, 152), + (10, 187), + (10, 201), + (10, 231), + (10, 246), + (10, 252), + (11, 9), + (11, 47), + (11, 61), + (11, 66), + (11, 77), + (11, 127), + (11, 132), + (11, 150), + (11, 160), + (11, 208), + (11, 226), + (11, 241), + (12, 34), + (12, 40), + (12, 70), + (12, 86), + (12, 88), + (12, 106), + (12, 148), + (12, 173), + (12, 192), + (12, 207), + (12, 238), + (12, 245), + (13, 6), + (13, 45), + (13, 97), + (13, 132), + (13, 192), + (14, 16), + (14, 61), + (14, 142), + (14, 171), + (14, 176), + (14, 229), + (14, 240), + (15, 2), + (15, 5), + (15, 34), + (15, 42), + (15, 68), + (15, 114), + (15, 220), + (16, 17), + (16, 42), + (16, 45), + (16, 125), + (16, 130), + (16, 172), + (16, 187), + (16, 199), + (16, 212), + (16, 249), + (17, 10), + (17, 40), + (17, 57), + (17, 74), + (17, 78), + (17, 100), + (17, 105), + (17, 111), + (17, 170), + (18, 36), + (18, 47), + (18, 63), + (18, 123), + (18, 132), + (18, 136), + (18, 185), + (18, 188), + (18, 196), + (18, 228), + (18, 239), + (19, 7), + (19, 37), + (19, 62), + (19, 113), + (19, 124), + (19, 139), + (19, 159), + (19, 187), + (19, 196), + (19, 225), + (19, 244), + (20, 1), + (20, 3), + (20, 135), + (20, 188), + (20, 194), + (20, 215), + (20, 217), + (21, 7), + (21, 17), + (21, 27), + (21, 73), + (21, 102), + (21, 113), + (21, 157), + (21, 166), + (21, 185), + (21, 222), + (22, 11), + (22, 17), + (22, 19), + (22, 25), + (22, 103), + (22, 125), + (22, 133), + (22, 212), + (22, 228), + (23, 0), + (23, 11), + (23, 14), + (23, 40), + (23, 41), + (23, 42), + (23, 48), + (23, 51), + (23, 58), + (23, 67), + (23, 72), + (23, 88), + (23, 99), + (23, 127), + (23, 136), + (23, 153), + (23, 158), + (23, 199), + (23, 203), + (23, 209), + (23, 210), + (23, 215), + (23, 228), + (24, 47), + (24, 91), + (24, 146), + (24, 172), + (24, 191), + (24, 202), + (25, 29), + (25, 82), + (25, 90), + (25, 101), + (25, 121), + (25, 129), + (25, 141), + (25, 161), + (25, 162), + (25, 200), + (25, 232), + (25, 233), + (26, 8), + (26, 10), + (26, 58), + (26, 66), + (26, 68), + (26, 80), + (26, 116), + (26, 133), + (26, 212), + (26, 236), + (27, 29), + (27, 41), + (27, 65), + (27, 135), + (27, 139), + (28, 11), + (28, 33), + (28, 59), + (28, 65), + (28, 66), + (28, 82), + (28, 136), + (28, 139), + (28, 178), + (28, 193), + (28, 228), + (28, 239), + (29, 20), + (29, 30), + (29, 73), + (29, 83), + (29, 95), + (29, 118), + (29, 131), + (30, 11), + (30, 34), + (30, 51), + (30, 53), + (30, 54), + (30, 80), + (30, 91), + (30, 151), + (30, 154), + (30, 227), + (30, 237), + (30, 239), + (30, 245), + (31, 10), + (31, 30), + (31, 116), + (31, 152), + (31, 159), + (31, 197), + (31, 204), + (32, 39), + (32, 74), + (32, 92), + (32, 119), + (32, 141), + (32, 193), + (33, 5), + (33, 81), + (33, 127), + (33, 145), + (33, 175), + (33, 189), + (33, 206), + (34, 59), + (34, 60), + (34, 82), + (34, 92), + (34, 124), + (34, 158), + (34, 173), + (34, 193), + (35, 14), + (35, 17), + (35, 54), + (35, 60), + (35, 81), + (35, 100), + (35, 131), + (35, 132), + (35, 148), + (35, 187), + (35, 193), + (35, 214), + (35, 249), + (36, 3), + (36, 24), + (36, 57), + (36, 62), + (36, 82), + (36, 94), + (36, 123), + (36, 132), + (36, 190), + (36, 226), + (36, 228), + (36, 244), + (37, 10), + (37, 17), + (37, 50), + (37, 67), + (37, 81), + (37, 83), + (37, 158), + (37, 194), + (37, 202), + (37, 209), + (37, 224), + (37, 250), + (38, 8), + (38, 20), + (38, 88), + (38, 93), + (38, 175), + (38, 211), + (39, 1), + (39, 22), + (39, 25), + (39, 40), + (39, 61), + (39, 82), + (39, 119), + (39, 147), + (39, 219), + (39, 235), + (39, 239), + (40, 36), + (40, 52), + (40, 62), + (40, 71), + (40, 90), + (40, 100), + (40, 215), + (41, 7), + (41, 10), + (41, 80), + (41, 93), + (41, 109), + (41, 155), + (41, 164), + (41, 168), + (41, 185), + (41, 189), + (41, 221), + (42, 44), + (42, 54), + (42, 99), + (42, 160), + (42, 190), + (42, 191), + (42, 222), + (0, 11), + (0, 22), + (0, 26), + (0, 32), + (0, 34), + (0, 53), + (0, 64), + (0, 67), + (0, 72), + (0, 92), + (0, 101), + (0, 103), + (0, 109), + (0, 140), + (0, 142), + (0, 148), + (0, 158), + (0, 172), + (0, 191), + (0, 196), + (0, 201), + (0, 216), + (0, 223), + (0, 227), + (0, 228), + (0, 234), + (0, 237), + (0, 243), + (1, 10), + (1, 14), + (1, 24), + (1, 40), + (1, 88), + (1, 89), + (1, 94), + (1, 98), + (1, 101), + (1, 121), + (1, 124), + (1, 148), + (1, 165), + (1, 166), + (1, 171), + (1, 188), + (1, 189), + (1, 222), + (1, 237), + (1, 241), + (1, 246), + (2, 16), + (2, 25), + (2, 47), + (2, 53), + (2, 65), + (2, 70), + (2, 71), + (2, 72), + (2, 88), + (2, 90), + (2, 123), + (2, 138), + (2, 147), + (2, 154), + (2, 165), + (2, 189), + (2, 190), + (2, 194), + (2, 205), + (2, 217), + (2, 220), + (2, 221), + (2, 226), + (2, 228), + (2, 237), + (2, 240), + (2, 243), + (3, 37), + (3, 45), + (3, 48), + (3, 60), + (3, 75), + (3, 85), + (3, 86), + (3, 93), + (3, 113), + (3, 120), + (3, 137), + (3, 141), + (3, 163), + (3, 170), + (3, 178), + (3, 180), + (3, 217), + (3, 228), + (3, 229), + (3, 231), + (3, 235), + (4, 41), + (4, 46), + (4, 66), + (4, 78), + (4, 90), + (4, 122), + (4, 184), + (4, 185), + (4, 198), + (4, 204), + (4, 221), + (4, 244), + (5, 8), + (5, 18), + (5, 35), + (5, 44), + (5, 66), + (5, 81), + (5, 92), + (5, 96), + (5, 110), + (5, 113), + (5, 129), + (5, 134), + (5, 179), + (5, 235), + (5, 236), + (5, 247), + (6, 14), + (6, 30), + (6, 35), + (6, 47), + (6, 54), + (6, 102), + (6, 109), + (6, 131), + (6, 142), + (6, 176), + (6, 242), + (6, 243), + (6, 251), + (6, 255), + (7, 5), + (7, 27), + (7, 40), + (7, 50), + (7, 52), + (7, 72), + (7, 84), + (7, 163), + (7, 172), + (7, 175), + (7, 189), + (7, 201), + (7, 220), + (7, 247), + (7, 254), + (8, 7), + (8, 12), + (8, 16), + (8, 18), + (8, 33), + (8, 34), + (8, 54), + (8, 56), + (8, 73), + (8, 76), + (8, 86), + (8, 98), + (8, 129), + (8, 133), + (8, 147), + (8, 166), + (8, 170), + (8, 179), + (8, 184), + (8, 193), + (8, 237), + (9, 33), + (9, 56), + (9, 73), + (9, 114), + (9, 182), + (9, 188), + (9, 194), + (9, 223), + (9, 229), + (9, 243), + (10, 4), + (10, 106), + (10, 118), + (10, 120), + (10, 132), + (10, 160), + (10, 169), + (10, 189), + (10, 195), + (10, 209), + (10, 241), + (10, 242), + (11, 5), + (11, 32), + (11, 43), + (11, 52), + (11, 99), + (11, 153), + (11, 233), + (11, 237), + (12, 11), + (12, 14), + (12, 72), + (12, 83), + (12, 91), + (12, 95), + (12, 99), + (12, 134), + (12, 138), + (12, 139), + (12, 194), + (12, 230), + (13, 3), + (13, 18), + (13, 52), + (13, 61), + (13, 67), + (13, 93), + (13, 117), + (13, 134), + (13, 148), + (13, 161), + (13, 162), + (13, 207), + (14, 24), + (14, 60), + (14, 66), + (14, 67), + (14, 156), + (14, 159), + (14, 191), + (14, 211), + (14, 230), + (15, 44), + (15, 72), + (15, 79), + (15, 103), + (15, 118), + (15, 140), + (15, 184), + (15, 192), + (16, 27), + (16, 87), + (16, 113), + (16, 117), + (16, 122), + (16, 175), + (16, 185), + (16, 194), + (16, 196), + (16, 203), + (16, 238), + (16, 243), + (17, 4), + (17, 32), + (17, 52), + (17, 104), + (17, 140), + (17, 143), + (17, 163), + (17, 178), + (17, 182), + (17, 190), + (17, 220), + (17, 242), + (18, 0), + (18, 10), + (18, 19), + (18, 20), + (18, 56), + (18, 156), + (18, 190), + (18, 202), + (18, 209), + (18, 222), + (18, 231), + (18, 237), + (18, 247), + (18, 253), + (19, 9), + (19, 36), + (19, 39), + (19, 46), + (19, 56), + (19, 59), + (19, 65), + (19, 117), + (19, 150), + (19, 164), + (19, 221), + (19, 242), + (19, 247), + (19, 252), + (20, 11), + (20, 31), + (20, 44), + (20, 63), + (20, 92), + (20, 191), + (20, 197), + (20, 226), + (20, 237), + (21, 23), + (21, 55), + (21, 61), + (21, 92), + (21, 117), + (21, 119), + (21, 142), + (21, 143), + (21, 158), + (21, 180), + (21, 230), + (22, 4), + (22, 29), + (22, 43), + (22, 45), + (22, 53), + (22, 60), + (22, 97), + (22, 100), + (22, 141), + (22, 148), + (22, 153), + (22, 158), + (22, 161), + (22, 163), + (22, 187), + (22, 199), + (22, 200), + (22, 213), + (22, 234), + (22, 243), + (22, 252), + (23, 1), + (23, 28), + (23, 30), + (23, 37), + (23, 49), + (23, 61), + (23, 87), + (23, 94), + (23, 113), + (23, 129), + (23, 133), + (23, 135), + (23, 156), + (23, 180), + (23, 185), + (23, 186), + (23, 192), + (23, 206), + (23, 216), + (23, 217), + (23, 231), + (23, 234), + (23, 236), + (23, 255), + (24, 30), + (24, 54), + (24, 127), + (24, 179), + (24, 183), + (24, 204), + (24, 228), + (24, 241), + (24, 255), + (25, 20), + (25, 40), + (25, 80), + (25, 98), + (25, 111), + (25, 119), + (25, 122), + (25, 140), + (25, 149), + (25, 186), + (25, 227), + (26, 4), + (26, 15), + (26, 38), + (26, 44), + (26, 62), + (26, 148), + (26, 150), + (26, 189), + (26, 193), + (26, 230), + (26, 250), + (27, 69), + (27, 70), + (27, 85), + (27, 137), + (27, 149), + (27, 160), + (27, 165), + (27, 173), + (27, 176), + (27, 195), + (27, 201), + (27, 213), + (27, 240), + (27, 243), + (27, 245), + (28, 21), + (28, 54), + (28, 62), + (28, 72), + (28, 76), + (28, 81), + (28, 83), + (28, 101), + (28, 103), + (28, 109), + (28, 135), + (28, 148), + (28, 149), + (28, 156), + (28, 189), + (28, 231), + (28, 235), + (28, 245), + (28, 253), + (28, 254), + (29, 29), + (29, 57), + (29, 89), + (29, 106), + (29, 156), + (29, 207), + (29, 227), + (30, 13), + (30, 20), + (30, 120), + (30, 162), + (30, 181), + (30, 205), + (30, 234), + (30, 255), + (31, 9), + (31, 13), + (31, 22), + (31, 45), + (31, 72), + (31, 118), + (31, 128), + (31, 130), + (31, 135), + (31, 183), + (31, 198), + (31, 206), + (31, 215), + (31, 219), + (31, 225), + (31, 242), + (32, 1), + (32, 3), + (32, 49), + (32, 123), + (32, 137), + (32, 166), + (32, 181), + (32, 216), + (32, 244), + (32, 255), + (33, 3), + (33, 37), + (33, 40), + (33, 51), + (33, 68), + (33, 78), + (33, 87), + (33, 98), + (33, 103), + (33, 143), + (33, 151), + (33, 222), + (33, 230), + (33, 234), + (33, 238), + (33, 254), + (34, 12), + (34, 32), + (34, 34), + (34, 51), + (34, 57), + (34, 118), + (34, 196), + (34, 210), + (34, 211), + (34, 248), + (35, 13), + (35, 16), + (35, 25), + (35, 32), + (35, 190), + (35, 194), + (35, 234), + (35, 236), + (35, 251), + (36, 36), + (36, 52), + (36, 72), + (36, 206), + (36, 212), + (36, 236), + (37, 40), + (37, 47), + (37, 62), + (37, 72), + (37, 78), + (37, 96), + (37, 139), + (37, 144), + (37, 161), + (37, 168), + (37, 203), + (37, 215), + (37, 230), + (37, 243), + (37, 248), + (37, 251), + (38, 0), + (38, 6), + (38, 16), + (38, 40), + (38, 48), + (38, 49), + (38, 55), + (38, 109), + (38, 114), + (38, 141), + (38, 147), + (38, 155), + (38, 194), + (38, 196), + (38, 217), + (38, 228), + (38, 245), + (38, 247), + (39, 7), + (39, 13), + (39, 34), + (39, 36), + (39, 39), + (39, 52), + (39, 75), + (39, 98), + (39, 134), + (39, 155), + (39, 185), + (39, 198), + (39, 224), + (39, 225), + (39, 233), + (39, 249), + (40, 1), + (40, 43), + (40, 57), + (40, 63), + (40, 87), + (40, 89), + (40, 112), + (40, 131), + (40, 154), + (40, 157), + (40, 165), + (40, 182), + (40, 193), + (40, 211), + (40, 227), + (40, 235), + (41, 27), + (41, 46), + (41, 57), + (41, 103), + (41, 120), + (41, 136), + (41, 156), + (41, 165), + (41, 182), + (41, 202), + (41, 207), + (41, 212), + (41, 213), + (41, 246), + (41, 250), + (42, 29), + (42, 35), + (42, 51), + (42, 75), + (42, 90), + (42, 104), + (42, 120), + (42, 175), + (42, 186), + (42, 201), + (0, 45), + (0, 59), + (0, 73), + (0, 80), + (0, 90), + (0, 110), + (0, 125), + (0, 161), + (0, 193), + (0, 197), + (0, 220), + (0, 221), + (0, 244), + (0, 246), + (0, 247), + (1, 23), + (1, 28), + (1, 42), + (1, 46), + (1, 55), + (1, 61), + (1, 63), + (1, 71), + (1, 90), + (1, 92), + (1, 104), + (1, 125), + (1, 129), + (1, 131), + (1, 134), + (1, 159), + (1, 174), + (1, 191), + (1, 198), + (1, 200), + (1, 202), + (1, 233), + (1, 244), + (2, 9), + (2, 69), + (2, 98), + (2, 103), + (2, 111), + (2, 136), + (2, 142), + (2, 162), + (2, 171), + (2, 211), + (2, 213), + (2, 218), + (2, 222), + (2, 246), + (2, 249), + (2, 250), + (2, 253), + (3, 16), + (3, 19), + (3, 29), + (3, 38), + (3, 52), + (3, 66), + (3, 69), + (3, 91), + (3, 119), + (3, 131), + (3, 147), + (3, 157), + (3, 169), + (3, 194), + (3, 206), + (3, 245), + (4, 1), + (4, 8), + (4, 9), + (4, 34), + (4, 50), + (4, 54), + (4, 60), + (4, 64), + (4, 84), + (4, 85), + (4, 116), + (4, 125), + (4, 129), + (4, 141), + (4, 157), + (4, 158), + (4, 162), + (4, 166), + (4, 167), + (4, 169), + (4, 190), + (4, 195), + (4, 206), + (4, 218), + (4, 219), + (4, 232), + (4, 238), + (4, 240), + (5, 9), + (5, 13), + (5, 46), + (5, 71), + (5, 88), + (5, 105), + (5, 117), + (5, 118), + (5, 119), + (5, 127), + (5, 131), + (5, 141), + (5, 151), + (5, 185), + (5, 202), + (5, 211), + (5, 216), + (5, 221), + (6, 8), + (6, 38), + (6, 43), + (6, 56), + (6, 57), + (6, 61), + (6, 108), + (6, 119), + (6, 123), + (6, 140), + (6, 148), + (6, 173), + (6, 184), + (6, 193), + (6, 197), + (6, 214), + (6, 227), + (6, 231), + (7, 22), + (7, 70), + (7, 81), + (7, 93), + (7, 96), + (7, 130), + (7, 132), + (7, 150), + (7, 169), + (7, 171), + (7, 192), + (7, 198), + (7, 202), + (7, 205), + (7, 210), + (8, 0), + (8, 22), + (8, 41), + (8, 47), + (8, 75), + (8, 94), + (8, 97), + (8, 110), + (8, 114), + (8, 120), + (8, 122), + (8, 124), + (8, 126), + (8, 139), + (8, 157), + (8, 164), + (8, 176), + (8, 217), + (8, 220), + (8, 250), + (8, 255), + (9, 4), + (9, 25), + (9, 31), + (9, 40), + (9, 41), + (9, 62), + (9, 79), + (9, 82), + (9, 93), + (9, 104), + (9, 117), + (9, 125), + (9, 137), + (9, 138), + (9, 146), + (9, 147), + (9, 157), + (9, 204), + (9, 222), + (9, 230), + (9, 238), + (9, 244), + (10, 20), + (10, 34), + (10, 39), + (10, 45), + (10, 59), + (10, 65), + (10, 69), + (10, 90), + (10, 96), + (10, 100), + (10, 127), + (10, 137), + (10, 139), + (10, 153), + (10, 167), + (10, 177), + (10, 183), + (10, 184), + (10, 186), + (10, 202), + (10, 224), + (11, 4), + (11, 15), + (11, 16), + (11, 25), + (11, 38), + (11, 46), + (11, 53), + (11, 55), + (11, 65), + (11, 90), + (11, 104), + (11, 124), + (11, 128), + (11, 134), + (11, 140), + (11, 152), + (11, 202), + (11, 210), + (11, 214), + (11, 225), + (12, 4), + (12, 9), + (12, 10), + (12, 43), + (12, 50), + (12, 58), + (12, 115), + (12, 119), + (12, 121), + (12, 133), + (12, 180), + (12, 203), + (12, 208), + (12, 213), + (13, 4), + (13, 29), + (13, 32), + (13, 34), + (13, 53), + (13, 72), + (13, 85), + (13, 91), + (13, 99), + (13, 100), + (13, 107), + (13, 126), + (13, 135), + (13, 189), + (13, 203), + (13, 224), + (13, 231), + (13, 242), + (13, 247), + (14, 33), + (14, 38), + (14, 107), + (14, 122), + (14, 149), + (14, 154), + (14, 179), + (14, 185), + (14, 228), + (14, 233), + (14, 237), + (14, 239), + (14, 244), + (14, 246), + (14, 248), + (14, 251), + (15, 1), + (15, 16), + (15, 30), + (15, 46), + (15, 62), + (15, 65), + (15, 78), + (15, 81), + (15, 90), + (15, 93), + (15, 125), + (15, 129), + (15, 156), + (15, 166), + (15, 175), + (15, 187), + (15, 232), + (15, 253), + (15, 254), + (16, 2), + (16, 28), + (16, 36), + (16, 62), + (16, 75), + (16, 121), + (16, 124), + (16, 141), + (16, 162), + (16, 174), + (16, 200), + (16, 219), + (16, 225), + (17, 18), + (17, 21), + (17, 45), + (17, 58), + (17, 63), + (17, 93), + (17, 95), + (17, 118), + (17, 154), + (17, 158), + (17, 160), + (17, 169), + (17, 173), + (17, 174), + (17, 254), + (18, 8), + (18, 21), + (18, 24), + (18, 30), + (18, 52), + (18, 54), + (18, 65), + (18, 68), + (18, 81), + (18, 86), + (18, 108), + (18, 138), + (18, 169), + (18, 176), + (18, 180), + (18, 204), + (18, 208), + (18, 214), + (19, 4), + (19, 6), + (19, 15), + (19, 20), + (19, 23), + (19, 33), + (19, 78), + (19, 81), + (19, 94), + (19, 96), + (19, 109), + (19, 121), + (19, 125), + (19, 144), + (19, 157), + (19, 162), + (19, 163), + (19, 167), + (19, 175), + (19, 176), + (19, 177), + (19, 183), + (19, 232), + (20, 9), + (20, 14), + (20, 28), + (20, 38), + (20, 61), + (20, 79), + (20, 117), + (20, 121), + (20, 142), + (20, 152), + (20, 168), + (20, 186), + (20, 227), + (20, 228), + (20, 238), + (20, 244), + (20, 253), + (21, 5), + (21, 9), + (21, 18), + (21, 57), + (21, 66), + (21, 105), + (21, 122), + (21, 127), + (21, 148), + (21, 163), + (21, 186), + (21, 214), + (21, 255), + (22, 1), + (22, 20), + (22, 37), + (22, 71), + (22, 76), + (22, 79), + (22, 126), + (22, 127), + (22, 154), + (22, 186), + (22, 237), + (23, 8), + (23, 19), + (23, 20), + (23, 29), + (23, 76), + (23, 98), + (23, 101), + (23, 111), + (23, 117), + (23, 130), + (23, 143), + (23, 251), + (24, 7), + (24, 46), + (24, 51), + (24, 79), + (24, 114), + (24, 130), + (24, 157), + (24, 169), + (24, 185), + (24, 186), + (24, 224), + (24, 242), + (24, 245), + (24, 251), + (25, 13), + (25, 37), + (25, 52), + (25, 74), + (25, 91), + (25, 95), + (25, 102), + (25, 112), + (25, 138), + (25, 142), + (25, 150), + (25, 154), + (25, 164), + (25, 169), + (25, 178), + (25, 180), + (25, 184), + (25, 192), + (25, 216), + (25, 223), + (25, 231), + (25, 239), + (26, 2), + (26, 11), + (26, 53), + (26, 60), + (26, 61), + (26, 81), + (26, 82), + (26, 128), + (26, 138), + (26, 163), + (26, 187), + (26, 210), + (26, 217), + (26, 251), + (27, 6), + (27, 12), + (27, 16), + (27, 21), + (27, 48), + (27, 49), + (27, 75), + (27, 87), + (27, 109), + (27, 110), + (27, 121), + (27, 122), + (27, 128), + (27, 132), + (27, 144), + (27, 147), + (27, 182), + (27, 184), + (27, 192), + (27, 197), + (27, 229), + (27, 244), + (28, 16), + (28, 22), + (28, 35), + (28, 45), + (28, 47), + (28, 48), + (28, 49), + (28, 55), + (28, 95), + (28, 102), + (28, 120), + (28, 131), + (28, 132), + (28, 134), + (28, 147), + (28, 182), + (28, 195), + (28, 202), + (28, 207), + (28, 212), + (28, 225), + (28, 240), + (29, 3), + (29, 15), + (29, 18), + (29, 135), + (29, 143), + (29, 148), + (29, 151), + (29, 161), + (29, 201), + (29, 209), + (29, 216), + (29, 237), + (30, 3), + (30, 5), + (30, 40), + (30, 62), + (30, 85), + (30, 119), + (30, 147), + (30, 148), + (30, 204), + (30, 225), + (30, 246), + (31, 20), + (31, 28), + (31, 49), + (31, 58), + (31, 70), + (31, 91), + (31, 110), + (31, 141), + (31, 151), + (31, 175), + (31, 177), + (31, 207), + (31, 227), + (31, 247), + (31, 248), + (32, 0), + (32, 26), + (32, 27), + (32, 56), + (32, 65), + (32, 126), + (32, 182), + (32, 197), + (32, 201), + (32, 217), + (32, 226), + (33, 49), + (33, 59), + (33, 72), + (33, 109), + (33, 134), + (33, 139), + (33, 149), + (33, 154), + (33, 155), + (33, 164), + (33, 165), + (33, 173), + (33, 174), + (33, 185), + (33, 200), + (33, 227), + (33, 231), + (33, 237), + (33, 244), + (33, 252), + (34, 9), + (34, 11), + (34, 14), + (34, 15), + (34, 67), + (34, 74), + (34, 90), + (34, 104), + (34, 121), + (34, 147), + (34, 152), + (34, 153), + (34, 169), + (34, 186), + (34, 189), + (34, 190), + (34, 192), + (34, 199), + (34, 206), + (34, 207), + (34, 213), + (35, 39), + (35, 56), + (35, 80), + (35, 102), + (35, 107), + (35, 118), + (35, 145), + (35, 151), + (35, 167), + (35, 215), + (35, 227), + (35, 232), + (36, 41), + (36, 44), + (36, 51), + (36, 60), + (36, 105), + (36, 138), + (36, 161), + (36, 195), + (36, 209), + (36, 222), + (36, 232), + (36, 239), + (37, 20), + (37, 22), + (37, 34), + (37, 63), + (37, 68), + (37, 101), + (37, 106), + (37, 107), + (37, 113), + (37, 135), + (37, 138), + (37, 145), + (37, 146), + (37, 151), + (37, 163), + (37, 176), + (37, 177), + (37, 204), + (37, 237), + (38, 12), + (38, 22), + (38, 31), + (38, 35), + (38, 42), + (38, 46), + (38, 75), + (38, 81), + (38, 90), + (38, 95), + (38, 101), + (38, 131), + (38, 157), + (38, 171), + (38, 199), + (38, 214), + (38, 219), + (38, 254), + (39, 10), + (39, 28), + (39, 41), + (39, 56), + (39, 65), + (39, 79), + (39, 93), + (39, 104), + (39, 112), + (39, 131), + (39, 156), + (39, 157), + (39, 166), + (39, 169), + (39, 171), + (39, 187), + (39, 202), + (39, 204), + (39, 211), + (39, 216), + (39, 228), + (39, 232), + (39, 240), + (39, 250), + (39, 255), + (40, 5), + (40, 10), + (40, 11), + (40, 14), + (40, 29), + (40, 35), + (40, 38), + (40, 73), + (40, 78), + (40, 85), + (40, 95), + (40, 98), + (40, 99), + (40, 136), + (40, 143), + (40, 150), + (40, 164), + (40, 192), + (40, 233), + (40, 245), + (40, 250), + (41, 39), + (41, 42), + (41, 43), + (41, 45), + (41, 47), + (41, 53), + (41, 64), + (41, 69), + (41, 76), + (41, 77), + (41, 88), + (41, 110), + (41, 125), + (41, 146), + (41, 179), + (41, 180), + (41, 188), + (41, 199), + (41, 204), + (41, 226), + (41, 240), + (41, 243), + (41, 248), + (42, 25), + (42, 33), + (42, 41), + (42, 86), + (42, 105), + (42, 119), + (42, 130), + (42, 132), + (42, 171), + (42, 184), + (42, 188), + (42, 193), + (42, 194), + (42, 195), + (42, 198), + (42, 205), + (42, 213), + (42, 243), + (42, 252), + (0, 21), + (0, 128), + (0, 136), + (0, 167), + (0, 226), + (0, 233), + (0, 235), + (0, 238), + (1, 1), + (1, 108), + (1, 130), + (1, 152), + (1, 215), + (2, 56), + (2, 78), + (2, 104), + (2, 143), + (2, 201), + (2, 216), + (2, 241), + (2, 244), + (3, 2), + (3, 5), + (3, 8), + (3, 30), + (3, 33), + (3, 43), + (3, 44), + (3, 46), + (3, 47), + (3, 49), + (3, 53), + (3, 62), + (3, 64), + (3, 67), + (3, 74), + (3, 78), + (3, 81), + (3, 82), + (3, 94), + (3, 96), + (3, 111), + (3, 112), + (3, 126), + (3, 128), + (3, 130), + (3, 132), + (3, 134), + (3, 149), + (3, 156), + (3, 160), + (3, 166), + (3, 171), + (3, 181), + (3, 183), + (3, 184), + (3, 195), + (3, 208), + (3, 213), + (3, 219), + (3, 224), + (3, 226), + (3, 239), + (3, 242), + (3, 254), + (4, 3), + (4, 7), + (4, 20), + (4, 23), + (4, 28), + (4, 47), + (4, 59), + (4, 72), + (4, 86), + (4, 87), + (4, 93), + (4, 101), + (4, 109), + (4, 110), + (4, 111), + (4, 117), + (4, 145), + (4, 170), + (4, 178), + (4, 187), + (4, 191), + (4, 193), + (4, 196), + (4, 200), + (4, 201), + (4, 205), + (4, 207), + (4, 215), + (4, 222), + (4, 223), + (4, 225), + (4, 227), + (4, 241), + (4, 254), + (5, 0), + (5, 3), + (5, 7), + (5, 14), + (5, 20), + (5, 26), + (5, 27), + (5, 50), + (5, 52), + (5, 53), + (5, 60), + (5, 68), + (5, 72), + (5, 73), + (5, 91), + (5, 97), + (5, 98), + (5, 100), + (5, 102), + (5, 108), + (5, 136), + (5, 138), + (5, 142), + (5, 144), + (5, 152), + (5, 153), + (5, 157), + (5, 158), + (5, 163), + (5, 178), + (5, 182), + (5, 184), + (5, 189), + (5, 190), + (5, 212), + (5, 214), + (5, 227), + (5, 231), + (5, 234), + (5, 251), + (6, 0), + (6, 6), + (6, 17), + (6, 19), + (6, 21), + (6, 22), + (6, 26), + (6, 32), + (6, 36), + (6, 45), + (6, 49), + (6, 59), + (6, 68), + (6, 69), + (6, 70), + (6, 73), + (6, 81), + (6, 101), + (6, 104), + (6, 107), + (6, 114), + (6, 141), + (6, 145), + (6, 150), + (6, 153), + (6, 156), + (6, 175), + (6, 178), + (6, 179), + (6, 199), + (6, 204), + (6, 211), + (6, 215), + (6, 216), + (6, 232), + (6, 237), + (6, 247), + (6, 250), + (7, 1), + (7, 4), + (7, 6), + (7, 11), + (7, 12), + (7, 14), + (7, 16), + (7, 20), + (7, 25), + (7, 34), + (7, 41), + (7, 57), + (7, 83), + (7, 88), + (7, 100), + (7, 113), + (7, 115), + (7, 127), + (7, 128), + (7, 142), + (7, 145), + (7, 190), + (7, 193), + (7, 208), + (7, 211), + (7, 212), + (7, 213), + (7, 215), + (7, 218), + (7, 219), + (7, 222), + (7, 224), + (7, 227), + (7, 228), + (7, 229), + (7, 232), + (7, 237), + (7, 240), + (7, 249), + (7, 250), + (8, 5), + (8, 17), + (8, 23), + (8, 32), + (8, 49), + (8, 52), + (8, 70), + (8, 71), + (8, 77), + (8, 78), + (8, 84), + (8, 89), + (8, 92), + (8, 102), + (8, 116), + (8, 117), + (8, 132), + (8, 151), + (8, 155), + (8, 171), + (8, 187), + (8, 201), + (8, 203), + (8, 224), + (8, 235), + (8, 242), + (9, 6), + (9, 7), + (9, 20), + (9, 27), + (9, 37), + (9, 50), + (9, 71), + (9, 75), + (9, 91), + (9, 95), + (9, 101), + (9, 103), + (9, 111), + (9, 115), + (9, 136), + (9, 139), + (9, 142), + (9, 143), + (9, 152), + (9, 165), + (9, 170), + (9, 177), + (9, 185), + (9, 195), + (9, 202), + (9, 215), + (9, 217), + (9, 224), + (9, 235), + (10, 36), + (10, 62), + (10, 63), + (10, 102), + (10, 105), + (10, 114), + (10, 122), + (10, 136), + (10, 141), + (10, 144), + (10, 171), + (10, 181), + (10, 193), + (10, 204), + (10, 220), + (10, 221), + (10, 222), + (10, 226), + (10, 233), + (10, 240), + (10, 244), + (10, 253), + (11, 21), + (11, 27), + (11, 34), + (11, 36), + (11, 44), + (11, 45), + (11, 57), + (11, 68), + (11, 69), + (11, 72), + (11, 76), + (11, 85), + (11, 86), + (11, 97), + (11, 111), + (11, 122), + (11, 130), + (11, 145), + (11, 149), + (11, 156), + (11, 161), + (11, 170), + (11, 184), + (11, 186), + (11, 188), + (11, 190), + (11, 212), + (11, 215), + (11, 216), + (11, 222), + (11, 238), + (11, 240), + (11, 249), + (11, 251), + (12, 19), + (12, 38), + (12, 48), + (12, 52), + (12, 74), + (12, 84), + (12, 113), + (12, 120), + (12, 124), + (12, 146), + (12, 155), + (12, 163), + (12, 170), + (12, 184), + (12, 187), + (12, 191), + (12, 199), + (12, 204), + (12, 216), + (12, 217), + (12, 242), + (12, 244), + (12, 247), + (12, 249), + (12, 255), + (13, 2), + (13, 5), + (13, 9), + (13, 16), + (13, 21), + (13, 26), + (13, 27), + (13, 30), + (13, 46), + (13, 47), + (13, 59), + (13, 79), + (13, 84), + (13, 109), + (13, 118), + (13, 120), + (13, 137), + (13, 138), + (13, 152), + (13, 153), + (13, 154), + (13, 160), + (13, 168), + (13, 177), + (13, 178), + (13, 183), + (13, 186), + (13, 199), + (13, 200), + (13, 215), + (13, 219), + (13, 220), + (13, 228), + (13, 229), + (13, 235), + (13, 236), + (13, 238), + (13, 241), + (13, 243), + (13, 244), + (13, 253), + (14, 2), + (14, 29), + (14, 40), + (14, 42), + (14, 50), + (14, 54), + (14, 62), + (14, 63), + (14, 64), + (14, 79), + (14, 80), + (14, 88), + (14, 101), + (14, 106), + (14, 114), + (14, 127), + (14, 130), + (14, 144), + (14, 150), + (14, 194), + (14, 226), + (14, 252), + (14, 255), + (15, 12), + (15, 14), + (15, 18), + (15, 35), + (15, 39), + (15, 52), + (15, 57), + (15, 71), + (15, 73), + (15, 76), + (15, 83), + (15, 84), + (15, 86), + (15, 89), + (15, 98), + (15, 108), + (15, 110), + (15, 115), + (15, 127), + (15, 132), + (15, 133), + (15, 143), + (15, 149), + (15, 158), + (15, 160), + (15, 178), + (15, 179), + (15, 183), + (15, 197), + (15, 226), + (15, 228), + (15, 235), + (15, 245), + (16, 3), + (16, 10), + (16, 19), + (16, 25), + (16, 50), + (16, 55), + (16, 58), + (16, 63), + (16, 68), + (16, 78), + (16, 79), + (16, 81), + (16, 88), + (16, 91), + (16, 95), + (16, 101), + (16, 103), + (16, 111), + (16, 126), + (16, 127), + (16, 142), + (16, 148), + (16, 155), + (16, 158), + (16, 180), + (16, 197), + (16, 204), + (16, 227), + (16, 233), + (16, 235), + (16, 253), + (17, 5), + (17, 28), + (17, 29), + (17, 33), + (17, 42), + (17, 48), + (17, 68), + (17, 72), + (17, 86), + (17, 91), + (17, 99), + (17, 115), + (17, 129), + (17, 132), + (17, 135), + (17, 146), + (17, 148), + (17, 185), + (17, 186), + (17, 194), + (17, 216), + (17, 231), + (17, 237), + (17, 240), + (17, 246), + (17, 249), + (17, 255), + (18, 3), + (18, 11), + (18, 23), + (18, 29), + (18, 40), + (18, 48), + (18, 66), + (18, 82), + (18, 87), + (18, 89), + (18, 112), + (18, 118), + (18, 121), + (18, 125), + (18, 127), + (18, 137), + (18, 148), + (18, 172), + (18, 220), + (18, 225), + (18, 227), + (18, 240), + (18, 245), + (18, 249), + (19, 1), + (19, 12), + (19, 18), + (19, 24), + (19, 29), + (19, 41), + (19, 48), + (19, 49), + (19, 52), + (19, 61), + (19, 66), + (19, 69), + (19, 70), + (19, 71), + (19, 79), + (19, 87), + (19, 88), + (19, 90), + (19, 99), + (19, 107), + (19, 111), + (19, 114), + (19, 120), + (19, 132), + (19, 133), + (19, 137), + (19, 152), + (19, 165), + (19, 171), + (19, 172), + (19, 199), + (19, 201), + (19, 205), + (19, 206), + (19, 219), + (19, 234), + (19, 235), + (19, 236), + (19, 237), + (19, 239), + (19, 240), + (19, 245), + (19, 250), + (20, 5), + (20, 6), + (20, 13), + (20, 16), + (20, 24), + (20, 50), + (20, 59), + (20, 100), + (20, 108), + (20, 125), + (20, 155), + (20, 156), + (20, 158), + (20, 165), + (20, 175), + (20, 177), + (20, 193), + (20, 196), + (20, 204), + (20, 212), + (20, 218), + (20, 240), + (20, 246), + (20, 248), + (20, 249), + (21, 12), + (21, 13), + (21, 16), + (21, 30), + (21, 35), + (21, 52), + (21, 80), + (21, 96), + (21, 100), + (21, 106), + (21, 109), + (21, 136), + (21, 141), + (21, 146), + (21, 165), + (21, 175), + (21, 178), + (21, 199), + (21, 205), + (21, 210), + (21, 213), + (21, 227), + (21, 231), + (21, 243), + (21, 253), + (22, 0), + (22, 6), + (22, 14), + (22, 57), + (22, 59), + (22, 63), + (22, 65), + (22, 67), + (22, 69), + (22, 72), + (22, 81), + (22, 92), + (22, 96), + (22, 118), + (22, 119), + (22, 121), + (22, 150), + (22, 155), + (22, 164), + (22, 165), + (22, 168), + (22, 175), + (22, 189), + (22, 194), + (22, 203), + (22, 204), + (22, 211), + (22, 240), + (22, 246), + (23, 5), + (23, 6), + (23, 15), + (23, 27), + (23, 32), + (23, 56), + (23, 78), + (23, 105), + (23, 106), + (23, 114), + (23, 124), + (23, 128), + (23, 131), + (23, 146), + (23, 148), + (23, 179), + (23, 181), + (23, 187), + (23, 201), + (23, 219), + (23, 226), + (23, 227), + (23, 232), + (23, 237), + (23, 248), + (24, 1), + (24, 20), + (24, 22), + (24, 24), + (24, 42), + (24, 45), + (24, 49), + (24, 84), + (24, 93), + (24, 103), + (24, 124), + (24, 128), + (24, 142), + (24, 152), + (24, 188), + (24, 217), + (24, 244), + (24, 247), + (25, 9), + (25, 10), + (25, 11), + (25, 15), + (25, 19), + (25, 22), + (25, 24), + (25, 33), + (25, 39), + (25, 49), + (25, 56), + (25, 57), + (25, 58), + (25, 61), + (25, 72), + (25, 75), + (25, 85), + (25, 92), + (25, 93), + (25, 94), + (25, 109), + (25, 117), + (25, 125), + (25, 134), + (25, 147), + (25, 152), + (25, 159), + (25, 168), + (25, 188), + (25, 190), + (25, 193), + (25, 203), + (25, 215), + (25, 229), + (25, 234), + (25, 236), + (25, 240), + (25, 242), + (25, 247), + (25, 250), + (25, 251), + (26, 0), + (26, 5), + (26, 6), + (26, 12), + (26, 17), + (26, 25), + (26, 52), + (26, 72), + (26, 76), + (26, 87), + (26, 92), + (26, 96), + (26, 101), + (26, 108), + (26, 118), + (26, 121), + (26, 122), + (26, 126), + (26, 135), + (26, 140), + (26, 158), + (26, 168), + (26, 183), + (26, 184), + (26, 186), + (26, 190), + (26, 195), + (26, 206), + (26, 221), + (26, 232), + (26, 238), + (26, 239), + (26, 241), + (27, 5), + (27, 19), + (27, 30), + (27, 33), + (27, 34), + (27, 36), + (27, 46), + (27, 58), + (27, 74), + (27, 83), + (27, 97), + (27, 107), + (27, 117), + (27, 119), + (27, 136), + (27, 170), + (27, 186), + (27, 199), + (27, 202), + (27, 220), + (27, 222), + (27, 238), + (27, 239), + (27, 241), + (28, 3), + (28, 10), + (28, 24), + (28, 41), + (28, 42), + (28, 53), + (28, 58), + (28, 61), + (28, 69), + (28, 110), + (28, 130), + (28, 137), + (28, 165), + (28, 174), + (28, 186), + (28, 203), + (28, 204), + (28, 206), + (28, 214), + (28, 218), + (28, 238), + (28, 242), + (28, 247), + (28, 251), + (28, 255), + (29, 7), + (29, 11), + (29, 13), + (29, 23), + (29, 37), + (29, 53), + (29, 56), + (29, 59), + (29, 61), + (29, 62), + (29, 66), + (29, 72), + (29, 79), + (29, 81), + (29, 91), + (29, 92), + (29, 100), + (29, 103), + (29, 111), + (29, 120), + (29, 121), + (29, 125), + (29, 129), + (29, 134), + (29, 140), + (29, 141), + (29, 147), + (29, 149), + (29, 164), + (29, 166), + (29, 172), + (29, 181), + (29, 187), + (29, 203), + (29, 208), + (29, 212), + (29, 224), + (29, 225), + (29, 232), + (29, 236), + (29, 245), + (30, 15), + (30, 16), + (30, 26), + (30, 38), + (30, 70), + (30, 78), + (30, 79), + (30, 89), + (30, 96), + (30, 100), + (30, 108), + (30, 115), + (30, 131), + (30, 143), + (30, 150), + (30, 160), + (30, 171), + (30, 174), + (30, 179), + (30, 180), + (30, 190), + (30, 193), + (30, 198), + (30, 201), + (30, 203), + (30, 206), + (30, 218), + (30, 223), + (30, 252), + (31, 15), + (31, 16), + (31, 40), + (31, 42), + (31, 46), + (31, 53), + (31, 61), + (31, 75), + (31, 79), + (31, 81), + (31, 82), + (31, 83), + (31, 87), + (31, 93), + (31, 104), + (31, 119), + (31, 122), + (31, 124), + (31, 131), + (31, 165), + (31, 171), + (31, 172), + (31, 173), + (31, 185), + (31, 187), + (31, 194), + (31, 218), + (31, 222), + (31, 224), + (31, 234), + (31, 250), + (31, 251), + (32, 11), + (32, 19), + (32, 30), + (32, 32), + (32, 35), + (32, 46), + (32, 51), + (32, 57), + (32, 64), + (32, 72), + (32, 77), + (32, 83), + (32, 84), + (32, 90), + (32, 106), + (32, 111), + (32, 127), + (32, 128), + (32, 135), + (32, 150), + (32, 155), + (32, 161), + (32, 171), + (32, 185), + (32, 190), + (32, 198), + (32, 203), + (32, 211), + (32, 223), + (32, 224), + (32, 228), + (32, 230), + (32, 242), + (32, 249), + (33, 1), + (33, 4), + (33, 6), + (33, 11), + (33, 13), + (33, 15), + (33, 22), + (33, 30), + (33, 60), + (33, 71), + (33, 76), + (33, 113), + (33, 137), + (33, 159), + (33, 172), + (33, 180), + (33, 192), + (33, 195), + (33, 221), + (33, 223), + (33, 236), + (33, 250), + (33, 255), + (34, 17), + (34, 20), + (34, 25), + (34, 26), + (34, 30), + (34, 39), + (34, 45), + (34, 47), + (34, 49), + (34, 56), + (34, 64), + (34, 75), + (34, 84), + (34, 88), + (34, 95), + (34, 96), + (34, 102), + (34, 108), + (34, 115), + (34, 119), + (34, 122), + (34, 129), + (34, 133), + (34, 138), + (34, 145), + (34, 151), + (34, 157), + (34, 171), + (34, 177), + (34, 182), + (34, 191), + (34, 198), + (34, 201), + (34, 208), + (34, 212), + (34, 216), + (34, 219), + (34, 221), + (34, 228), + (34, 235), + (34, 255), + (35, 7), + (35, 11), + (35, 18), + (35, 23), + (35, 30), + (35, 40), + (35, 45), + (35, 58), + (35, 59), + (35, 65), + (35, 66), + (35, 67), + (35, 70), + (35, 73), + (35, 89), + (35, 92), + (35, 122), + (35, 125), + (35, 139), + (35, 152), + (35, 163), + (35, 171), + (35, 186), + (35, 189), + (35, 191), + (35, 218), + (35, 220), + (35, 238), + (35, 246), + (36, 9), + (36, 45), + (36, 50), + (36, 61), + (36, 71), + (36, 80), + (36, 89), + (36, 91), + (36, 93), + (36, 96), + (36, 121), + (36, 122), + (36, 135), + (36, 180), + (36, 181), + (36, 186), + (36, 192), + (36, 204), + (36, 208), + (36, 211), + (36, 216), + (36, 224), + (36, 230), + (36, 234), + (36, 235), + (36, 254), + (37, 0), + (37, 25), + (37, 29), + (37, 38), + (37, 39), + (37, 42), + (37, 53), + (37, 54), + (37, 56), + (37, 64), + (37, 73), + (37, 74), + (37, 77), + (37, 100), + (37, 116), + (37, 117), + (37, 132), + (37, 133), + (37, 137), + (37, 143), + (37, 154), + (37, 211), + (37, 219), + (37, 226), + (37, 235), + (37, 236), + (37, 238), + (38, 4), + (38, 11), + (38, 14), + (38, 69), + (38, 71), + (38, 78), + (38, 82), + (38, 85), + (38, 104), + (38, 106), + (38, 120), + (38, 121), + (38, 129), + (38, 140), + (38, 142), + (38, 150), + (38, 172), + (38, 192), + (38, 212), + (38, 226), + (38, 237), + (38, 239), + (39, 3), + (39, 8), + (39, 23), + (39, 27), + (39, 31), + (39, 32), + (39, 46), + (39, 63), + (39, 66), + (39, 69), + (39, 73), + (39, 74), + (39, 94), + (39, 99), + (39, 109), + (39, 110), + (39, 111), + (39, 114), + (39, 118), + (39, 120), + (39, 136), + (39, 137), + (39, 143), + (39, 148), + (39, 176), + (39, 184), + (39, 190), + (39, 192), + (39, 193), + (39, 205), + (39, 206), + (39, 217), + (39, 222), + (39, 236), + (39, 243), + (40, 2), + (40, 9), + (40, 23), + (40, 25), + (40, 60), + (40, 61), + (40, 80), + (40, 81), + (40, 94), + (40, 118), + (40, 119), + (40, 137), + (40, 149), + (40, 155), + (40, 156), + (40, 169), + (40, 172), + (40, 178), + (40, 189), + (40, 195), + (40, 198), + (40, 200), + (40, 202), + (40, 220), + (40, 223), + (40, 230), + (40, 243), + (40, 252), + (40, 253), + (41, 1), + (41, 8), + (41, 11), + (41, 13), + (41, 22), + (41, 26), + (41, 29), + (41, 40), + (41, 55), + (41, 56), + (41, 72), + (41, 73), + (41, 86), + (41, 89), + (41, 99), + (41, 105), + (41, 108), + (41, 113), + (41, 132), + (41, 137), + (41, 144), + (41, 147), + (41, 158), + (41, 174), + (41, 181), + (41, 183), + (41, 187), + (41, 200), + (41, 223), + (41, 233), + (41, 244), + (41, 249), + (42, 11), + (42, 12), + (42, 16), + (42, 20), + (42, 26), + (42, 32), + (42, 39), + (42, 53), + (42, 60), + (42, 84), + (42, 103), + (42, 114), + (42, 118), + (42, 121), + (42, 123), + (42, 124), + (42, 126), + (42, 142), + (42, 143), + (42, 147), + (42, 150), + (42, 152), + (42, 158), + (42, 164), + (42, 168), + (42, 176), + (42, 196), + (42, 208), + (42, 212), + (42, 214), + (42, 217), + (42, 226), + (42, 228), + (42, 235), + (42, 241), + (42, 246), + (42, 247), + (42, 251), + (42, 254), +]; diff --git a/src/engine/validation.rs b/src/engine/validation.rs index 484b6f0..a23f7d2 100644 --- a/src/engine/validation.rs +++ b/src/engine/validation.rs @@ -7,7 +7,7 @@ pub(crate) fn validate_model_artifact( ) -> Result<(), String> { if support { let model = Gguf::open(path)?; - validate_dspark(&model, &FLASH) + validate_support(&model, &FLASH).map(|_| ()) } else { let model = Model::open_main(path, expected)?; let summary = model.summary(); @@ -29,6 +29,192 @@ pub(crate) fn validate_model_artifact( } } +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub(super) enum SupportKind { + LegacyMtp, + DSpark, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub(super) struct DsparkConfig { + pub(super) block_size: u32, + pub(super) markov_rank: u32, + pub(super) noise_token: u32, + pub(super) target_layers: Vec, + pub(super) stages: u32, +} + +pub(super) fn dspark_config(model: &Gguf) -> Result { + let block_size = first_u32( + model, + &[ + "deepseek4.dspark.block_size", + "deepseek4.dspark_block_size", + "dspark.block_size", + ], + )?; + let markov_rank = first_u32( + model, + &[ + "deepseek4.dspark.markov_rank", + "deepseek4.dspark_markov_rank", + "dspark.markov_rank", + ], + )?; + let noise_token = first_u32( + model, + &[ + "deepseek4.dspark.noise_token_id", + "deepseek4.dspark_noise_token_id", + "dspark.noise_token_id", + ], + )?; + let target_layers = first_u32s( + model, + &[ + "deepseek4.dspark.target_layer_ids", + "deepseek4.dspark_target_layer_ids", + "dspark.target_layer_ids", + ], + )?; + let stages = model + .tensors + .keys() + .filter_map(|name| { + name.strip_prefix("mtp.")? + .split('.') + .next()? + .parse::() + .ok() + }) + .max() + .map_or(0, |stage| stage + 1); + Ok(DsparkConfig { + block_size, + markov_rank, + noise_token, + target_layers: target_layers.to_vec(), + stages, + }) +} + +pub(super) fn validate_support(model: &Gguf, shape: &Shape) -> Result { + if model.tensors.contains_key("mtp.0.e_proj.weight") + && model.tensors.contains_key("mtp.0.h_proj.weight") + && model.tensors.contains_key("mtp.0.hc_head_base.weight") + { + validate_legacy_mtp(model, shape)?; + Ok(SupportKind::LegacyMtp) + } else if model.metadata.contains_key("deepseek4.dspark.block_size") + || model.metadata.contains_key("deepseek4.dspark_block_size") + || model.metadata.contains_key("dspark.block_size") + { + validate_dspark(model, shape)?; + Ok(SupportKind::DSpark) + } else { + Err("support GGUF is neither legacy MTP nor DSpark".into()) + } +} + +fn validate_legacy_mtp(model: &Gguf, shape: &Shape) -> Result<(), String> { + if shape.model != ModelChoice::DeepSeekV4Flash { + return Err("legacy MTP support is available only for DeepSeek V4 Flash".into()); + } + let prefix = "mtp.0"; + let hc_dim = shape.embd * shape.hc; + let hc_mix = 2 * shape.hc + shape.hc * shape.hc; + let q_dim = shape.heads * shape.head_dim; + let output_low = shape.out_groups * shape.lora_o; + for (suffix, types, dims) in [ + ("hc_head_base.weight", &[F32][..], vec![shape.hc]), + ("hc_head_fn.weight", PLAIN, vec![hc_dim, shape.hc]), + ("hc_head_scale.weight", &[F32][..], vec![1]), + ("e_proj.weight", &[Q8_0][..], vec![shape.embd, shape.embd]), + ("h_proj.weight", &[Q8_0][..], vec![shape.embd, shape.embd]), + ("enorm.weight", &[F32][..], vec![shape.embd]), + ("hnorm.weight", &[F32][..], vec![shape.embd]), + ("norm.weight", &[F32][..], vec![shape.embd]), + ("hc_attn_fn.weight", PLAIN, vec![hc_dim, hc_mix]), + ("hc_attn_scale.weight", &[F32][..], vec![3]), + ("hc_attn_base.weight", &[F32][..], vec![hc_mix]), + ("attn_norm.weight", &[F32][..], vec![shape.embd]), + ( + "attn_q_a.weight", + &[Q8_0][..], + vec![shape.embd, shape.lora_q], + ), + ("attn_q_a_norm.weight", &[F32][..], vec![shape.lora_q]), + ("attn_q_b.weight", &[Q8_0][..], vec![shape.lora_q, q_dim]), + ( + "attn_kv.weight", + &[Q8_0][..], + vec![shape.embd, shape.head_dim], + ), + ("attn_kv_a_norm.weight", &[F32][..], vec![shape.head_dim]), + ("attn_sinks.weight", &[F32][..], vec![shape.heads]), + ( + "attn_output_a.weight", + &[Q8_0][..], + vec![ + shape.head_dim * (shape.heads / shape.out_groups), + output_low, + ], + ), + ( + "attn_output_b.weight", + &[Q8_0][..], + vec![output_low, shape.embd], + ), + ("hc_ffn_fn.weight", PLAIN, vec![hc_dim, hc_mix]), + ("hc_ffn_scale.weight", &[F32][..], vec![3]), + ("hc_ffn_base.weight", &[F32][..], vec![hc_mix]), + ("ffn_norm.weight", &[F32][..], vec![shape.embd]), + ( + "ffn_gate_inp.weight", + PLAIN, + vec![shape.embd, shape.experts], + ), + ("exp_probs_b.bias", &[F32][..], vec![shape.experts]), + ( + "ffn_gate_exps.weight", + ROUTED, + vec![shape.embd, shape.ff_expert, shape.experts], + ), + ( + "ffn_up_exps.weight", + ROUTED, + vec![shape.embd, shape.ff_expert, shape.experts], + ), + ( + "ffn_down_exps.weight", + ROUTED, + vec![shape.ff_expert, shape.embd, shape.experts], + ), + ( + "ffn_gate_shexp.weight", + &[Q8_0][..], + vec![shape.embd, shape.ff_expert], + ), + ( + "ffn_up_shexp.weight", + &[Q8_0][..], + vec![shape.embd, shape.ff_expert], + ), + ( + "ffn_down_shexp.weight", + &[Q8_0][..], + vec![shape.ff_expert, shape.embd], + ), + ] { + expect(model, &format!("{prefix}.{suffix}"), types, &dims)?; + } + same_type( + model, + "mtp.0.ffn_gate_exps.weight", + "mtp.0.ffn_up_exps.weight", + ) +} + pub(super) fn validate_main(model: &Gguf, expected: ModelChoice) -> Result { let family = if model.bytes("general.architecture").ok() == Some(b"glm-dsa") { ModelFamily::Glm @@ -575,38 +761,13 @@ pub(super) fn validate_dspark(model: &Gguf, shape: &Shape) -> Result<(), String> if shape.model != ModelChoice::DeepSeekV4Flash { return Err("DSpark support is available only for DeepSeek V4 Flash".into()); } - let block_size = first_u32( - model, - &[ - "deepseek4.dspark.block_size", - "deepseek4.dspark_block_size", - "dspark.block_size", - ], - )?; - let markov_rank = first_u32( - model, - &[ - "deepseek4.dspark.markov_rank", - "deepseek4.dspark_markov_rank", - "dspark.markov_rank", - ], - )?; - let noise_token = first_u32( - model, - &[ - "deepseek4.dspark.noise_token_id", - "deepseek4.dspark_noise_token_id", - "dspark.noise_token_id", - ], - )?; - let targets = first_u32s( - model, - &[ - "deepseek4.dspark.target_layer_ids", - "deepseek4.dspark_target_layer_ids", - "dspark.target_layer_ids", - ], - )?; + let DsparkConfig { + block_size, + markov_rank, + noise_token, + target_layers: targets, + stages, + } = dspark_config(model)?; if !(1..=16).contains(&block_size) || markov_rank == 0 || noise_token >= shape.vocab as u32 { return Err("invalid DSpark block, Markov, or noise-token metadata".into()); } @@ -617,18 +778,6 @@ pub(super) fn validate_dspark(model: &Gguf, shape: &Shape) -> Result<(), String> { return Err("invalid DSpark target-layer metadata".into()); } - let stages = model - .tensors - .keys() - .filter_map(|name| { - name.strip_prefix("mtp.")? - .split('.') - .next()? - .parse::() - .ok() - }) - .max() - .map_or(0, |stage| stage + 1); if !(1..=8).contains(&stages) { return Err(format!("invalid DSpark stage count: {stages}")); } @@ -1079,4 +1228,15 @@ mod tests { validate_model_artifact(path, ModelChoice::DeepSeekV4Flash, true).unwrap(); } } + + #[test] + fn installed_legacy_mtp_fixture_passes_the_target_layout() { + let path = Path::new("../ds4/gguf/DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf"); + if path.exists() { + assert_eq!( + validate_support(&Gguf::open(path).unwrap(), &FLASH).unwrap(), + SupportKind::LegacyMtp + ); + } + } } diff --git a/src/metrics.rs b/src/metrics.rs index f84376b..4dc95bc 100644 --- a/src/metrics.rs +++ b/src/metrics.rs @@ -86,6 +86,20 @@ pub(crate) struct MetricsSnapshot { pub(crate) prompt_tokens: u64, pub(crate) cached_tokens: u64, pub(crate) completion_tokens: u64, + pub(crate) speculative_mode: u8, + pub(crate) speculative_cycles: u64, + pub(crate) drafted_tokens: u64, + pub(crate) accepted_draft_tokens: u64, + pub(crate) verifier_passes: u64, + pub(crate) verifier_ms: u64, + pub(crate) ssd_enabled: bool, + pub(crate) ssd_resident_bytes: u64, + pub(crate) ssd_cache_bytes: u64, + pub(crate) ssd_cache_experts: u64, + pub(crate) ssd_preloaded_experts: u64, + pub(crate) ssd_selected_requests: u64, + pub(crate) ssd_requested_bytes: u64, + pub(crate) ssd_wait_ms: u64, pub(crate) kv_lookups: u64, pub(crate) kv_hits: u64, pub(crate) kv_memory_hits: u64, @@ -154,6 +168,20 @@ pub(crate) struct Metrics { prompt_tokens: AtomicU64, cached_tokens: AtomicU64, completion_tokens: AtomicU64, + speculative_mode: AtomicU8, + speculative_cycles: AtomicU64, + drafted_tokens: AtomicU64, + accepted_draft_tokens: AtomicU64, + verifier_passes: AtomicU64, + verifier_ms: AtomicU64, + ssd_enabled: AtomicBool, + ssd_resident_bytes: AtomicU64, + ssd_cache_bytes: AtomicU64, + ssd_cache_experts: AtomicU64, + ssd_preloaded_experts: AtomicU64, + ssd_selected_requests: AtomicU64, + ssd_requested_bytes: AtomicU64, + ssd_wait_ms: AtomicU64, kv_lookups: AtomicU64, kv_hits: AtomicU64, kv_memory_hits: AtomicU64, @@ -227,6 +255,20 @@ impl Metrics { prompt_tokens: AtomicU64::new(0), cached_tokens: AtomicU64::new(0), completion_tokens: AtomicU64::new(0), + speculative_mode: AtomicU8::new(0), + speculative_cycles: AtomicU64::new(0), + drafted_tokens: AtomicU64::new(0), + accepted_draft_tokens: AtomicU64::new(0), + verifier_passes: AtomicU64::new(0), + verifier_ms: AtomicU64::new(0), + ssd_enabled: AtomicBool::new(false), + ssd_resident_bytes: AtomicU64::new(0), + ssd_cache_bytes: AtomicU64::new(0), + ssd_cache_experts: AtomicU64::new(0), + ssd_preloaded_experts: AtomicU64::new(0), + ssd_selected_requests: AtomicU64::new(0), + ssd_requested_bytes: AtomicU64::new(0), + ssd_wait_ms: AtomicU64::new(0), kv_lookups: AtomicU64::new(0), kv_hits: AtomicU64::new(0), kv_memory_hits: AtomicU64::new(0), @@ -374,6 +416,52 @@ impl Metrics { self.source.store(WorkSource::None as u8, Ordering::Relaxed); } + pub(crate) fn speculative_stats( + &self, + mode: u8, + cycles: u64, + drafted: u64, + accepted: u64, + verifier_passes: u64, + verifier_ms: u64, + ) { + self.speculative_mode.store(mode, Ordering::Relaxed); + self.speculative_cycles.store(cycles, Ordering::Relaxed); + self.drafted_tokens.store(drafted, Ordering::Relaxed); + self.accepted_draft_tokens + .store(accepted, Ordering::Relaxed); + self.verifier_passes + .store(verifier_passes, Ordering::Relaxed); + self.verifier_ms.store(verifier_ms, Ordering::Relaxed); + } + + #[allow(clippy::too_many_arguments)] + pub(crate) fn ssd_stats( + &self, + enabled: bool, + resident_bytes: u64, + cache_bytes: u64, + cache_experts: u64, + preloaded_experts: u64, + selected_requests: u64, + requested_bytes: u64, + wait_ms: u64, + ) { + self.ssd_enabled.store(enabled, Ordering::Relaxed); + self.ssd_resident_bytes + .store(resident_bytes, Ordering::Relaxed); + self.ssd_cache_bytes.store(cache_bytes, Ordering::Relaxed); + self.ssd_cache_experts + .store(cache_experts, Ordering::Relaxed); + self.ssd_preloaded_experts + .store(preloaded_experts, Ordering::Relaxed); + self.ssd_selected_requests + .store(selected_requests, Ordering::Relaxed); + self.ssd_requested_bytes + .store(requested_bytes, Ordering::Relaxed); + self.ssd_wait_ms.store(wait_ms, Ordering::Relaxed); + } + pub(crate) fn unloaded(&self) { self.phase .store(RuntimePhase::Unloaded as u8, Ordering::Relaxed); @@ -385,6 +473,8 @@ impl Metrics { self.decode_tps.store(0, Ordering::Relaxed); self.prefill_tps.store(0, Ordering::Relaxed); self.prefill_sample.store(0, Ordering::Relaxed); + self.speculative_stats(0, 0, 0, 0, 0, 0); + self.ssd_stats(false, 0, 0, 0, 0, 0, 0, 0); self.model_unloads.fetch_add(1, Ordering::Relaxed); } @@ -534,6 +624,20 @@ impl Metrics { prompt_tokens: self.prompt_tokens.load(Ordering::Relaxed), cached_tokens: self.cached_tokens.load(Ordering::Relaxed), completion_tokens: self.completion_tokens.load(Ordering::Relaxed), + speculative_mode: self.speculative_mode.load(Ordering::Relaxed), + speculative_cycles: self.speculative_cycles.load(Ordering::Relaxed), + drafted_tokens: self.drafted_tokens.load(Ordering::Relaxed), + accepted_draft_tokens: self.accepted_draft_tokens.load(Ordering::Relaxed), + verifier_passes: self.verifier_passes.load(Ordering::Relaxed), + verifier_ms: self.verifier_ms.load(Ordering::Relaxed), + ssd_enabled: self.ssd_enabled.load(Ordering::Relaxed), + ssd_resident_bytes: self.ssd_resident_bytes.load(Ordering::Relaxed), + ssd_cache_bytes: self.ssd_cache_bytes.load(Ordering::Relaxed), + ssd_cache_experts: self.ssd_cache_experts.load(Ordering::Relaxed), + ssd_preloaded_experts: self.ssd_preloaded_experts.load(Ordering::Relaxed), + ssd_selected_requests: self.ssd_selected_requests.load(Ordering::Relaxed), + ssd_requested_bytes: self.ssd_requested_bytes.load(Ordering::Relaxed), + ssd_wait_ms: self.ssd_wait_ms.load(Ordering::Relaxed), kv_lookups: self.kv_lookups.load(Ordering::Relaxed), kv_hits: self.kv_hits.load(Ordering::Relaxed), kv_memory_hits: self.kv_memory_hits.load(Ordering::Relaxed), @@ -866,6 +970,7 @@ mod tests { metrics.kv_write_finished(Duration::from_millis(20), false); assert_eq!(metrics.take_kv_io_sample(), (2_048, 4_096)); assert_eq!(metrics.take_kv_io_sample(), (0, 0)); + metrics.ssd_stats(true, 1_024, 2_048, 16, 4, 12, 8_192, 30); metrics.request_finished( WorkSource::LocalChat, Duration::from_millis(250), @@ -890,6 +995,10 @@ mod tests { assert_eq!(snapshot.kv_prefix_hits, 1); assert_eq!(snapshot.kv_read_bytes, 2_048); assert_eq!(snapshot.kv_write_bytes, 4_096); + assert!(snapshot.ssd_enabled); + assert_eq!(snapshot.ssd_selected_requests, 12); + assert_eq!(snapshot.ssd_requested_bytes, 8_192); + assert_eq!(snapshot.ssd_wait_ms, 30); assert_eq!(snapshot.local_kv_bytes, 4_096); assert_eq!(snapshot.http_model_requests, 1); } diff --git a/src/model.rs b/src/model.rs index ec7c750..5704ada 100644 --- a/src/model.rs +++ b/src/model.rs @@ -14,8 +14,9 @@ pub(crate) const MODEL_CHOICES: [ModelChoice; 3] = [ ModelChoice::DeepSeekV4Pro, ModelChoice::Glm52, ]; -pub(crate) const MANAGED_ARTIFACTS: [ManagedArtifactId; 4] = [ +pub(crate) const MANAGED_ARTIFACTS: [ManagedArtifactId; 5] = [ ManagedArtifactId::DeepSeekV4Flash, + ManagedArtifactId::DeepSeekV4FlashMtp, ManagedArtifactId::DeepSeekV4FlashDspark, ManagedArtifactId::DeepSeekV4Pro, ManagedArtifactId::Glm52, @@ -40,6 +41,14 @@ const FLASH_DSPARK: Artifact = Artifact { sha256: "8b3adf5942bec22ae2ea867cd7079cf13530ba83ffcffaf00f5de48664a1a34e", support: Some(true), }; +const FLASH_MTP: Artifact = Artifact { + label: "Legacy MTP support", + file_name: "DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf", + repository: DEEPSEEK_REPOSITORY, + size: 3_807_602_400, + sha256: "afd481ee689dce9037f70f39085fcdae5a5b096d521cdad43b19fa52bf8f4083", + support: Some(true), +}; const PRO: Artifact = Artifact { label: "DeepSeek V4 Pro model", file_name: "DeepSeek-V4-Pro-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-Instruct-imatrix.gguf", @@ -94,9 +103,14 @@ impl ModelChoice { } #[cfg(test)] - fn artifacts(self, dspark_enabled: bool) -> impl Iterator { + fn artifacts( + self, + legacy_mtp_enabled: bool, + dspark_enabled: bool, + ) -> impl Iterator { [ Some(self.main_artifact()), + (self.supports_dspark() && legacy_mtp_enabled).then_some(&FLASH_MTP), (self.supports_dspark() && dspark_enabled).then_some(&FLASH_DSPARK), ] .into_iter() @@ -112,19 +126,26 @@ pub(crate) struct EngineArtifacts { pub(crate) fn engine_artifacts( model: ModelChoice, + legacy_mtp_enabled: bool, dspark_enabled: bool, models_path: &Path, ) -> EngineArtifacts { EngineArtifacts { model: model.main_artifact().path(model, models_path), - mtp: (model.supports_dspark() && dspark_enabled) - .then(|| FLASH_DSPARK.path(model, models_path)), + mtp: if model.supports_dspark() && legacy_mtp_enabled { + Some(FLASH_MTP.path(model, models_path)) + } else if model.supports_dspark() && dspark_enabled { + Some(FLASH_DSPARK.path(model, models_path)) + } else { + None + }, } } #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub(crate) enum ManagedArtifactId { DeepSeekV4Flash, + DeepSeekV4FlashMtp, DeepSeekV4FlashDspark, DeepSeekV4Pro, Glm52, @@ -133,7 +154,9 @@ pub(crate) enum ManagedArtifactId { impl ManagedArtifactId { pub(crate) fn model(self) -> ModelChoice { match self { - Self::DeepSeekV4Flash | Self::DeepSeekV4FlashDspark => ModelChoice::DeepSeekV4Flash, + Self::DeepSeekV4Flash | Self::DeepSeekV4FlashMtp | Self::DeepSeekV4FlashDspark => { + ModelChoice::DeepSeekV4Flash + } Self::DeepSeekV4Pro => ModelChoice::DeepSeekV4Pro, Self::Glm52 => ModelChoice::Glm52, } @@ -142,6 +165,7 @@ impl ManagedArtifactId { fn artifact(self) -> &'static Artifact { match self { Self::DeepSeekV4Flash => &FLASH, + Self::DeepSeekV4FlashMtp => &FLASH_MTP, Self::DeepSeekV4FlashDspark => &FLASH_DSPARK, Self::DeepSeekV4Pro => &PRO, Self::Glm52 => &GLM, diff --git a/src/model/transfer.rs b/src/model/transfer.rs index 4f311e1..25e25f7 100644 --- a/src/model/transfer.rs +++ b/src/model/transfer.rs @@ -331,15 +331,18 @@ mod tests { 86_720_111_488 ); assert_eq!(ModelChoice::Glm52.main_artifact().size, 211_075_856_448); - assert_eq!(ModelChoice::DeepSeekV4Flash.artifacts(true).count(), 2); - assert_eq!(ModelChoice::Glm52.artifacts(true).count(), 1); + assert_eq!( + ModelChoice::DeepSeekV4Flash.artifacts(true, true).count(), + 3 + ); + assert_eq!(ModelChoice::Glm52.artifacts(true, true).count(), 1); let id = SystemTime::now() .duration_since(UNIX_EPOCH) .unwrap() .as_nanos(); let models_path = std::env::temp_dir().join(format!("ds4-server-models-{id}")); - let engine = engine_artifacts(ModelChoice::DeepSeekV4Flash, true, &models_path); + let engine = engine_artifacts(ModelChoice::DeepSeekV4Flash, false, true, &models_path); assert_eq!( engine.model.file_name(), Some(std::ffi::OsStr::new(FLASH.file_name)) diff --git a/src/settings.rs b/src/settings.rs index f4daec6..ea5c25e 100644 --- a/src/settings.rs +++ b/src/settings.rs @@ -23,6 +23,7 @@ const DEFAULT_KV_CONTINUED_INTERVAL_TOKENS: u32 = 10_000; pub(crate) struct SpeculativePreferences { pub(crate) mtp_draft_tokens: i32, pub(crate) mtp_margin: f32, + pub(crate) legacy_mtp_enabled: bool, pub(crate) glm_mtp: bool, pub(crate) glm_mtp_timing: bool, pub(crate) dspark_enabled: bool, @@ -35,6 +36,7 @@ impl Default for SpeculativePreferences { Self { mtp_draft_tokens: 1, mtp_margin: 3.0, + legacy_mtp_enabled: false, glm_mtp: false, glm_mtp_timing: false, dspark_enabled: false, @@ -59,6 +61,12 @@ impl SpeculativePreferences { if self.dspark_enabled && !model.supports_dspark() { return Err("DSpark is not available for the selected model.".into()); } + if self.legacy_mtp_enabled && !model.supports_dspark() { + return Err("Legacy MTP is not available for the selected model.".into()); + } + if self.legacy_mtp_enabled && self.dspark_enabled { + return Err("Legacy MTP and DSpark use different support artifacts.".into()); + } if (self.dspark_confidence_threshold.is_some() || self.dspark_strict) && !self.dspark_enabled { @@ -343,6 +351,9 @@ impl DiagnosticPreferences { if let Some(gib) = self.simulated_used_memory_gib { validate_gib("Simulated used memory", gib)?; } + if self.expert_profile_path.is_some() { + return Err("Expert profiling is not available in the Rust Metal executor yet.".into()); + } Ok(()) } @@ -379,9 +390,6 @@ impl RuntimePreferences { self.steering.validate(model)?; self.diagnostics.validate()?; self.kv_cache.validate()?; - if self.ssd.enabled && self.speculative.dspark_enabled { - return Err("SSD streaming is not compatible with DSpark support.".into()); - } Ok(()) } @@ -394,7 +402,12 @@ impl RuntimePreferences { self.validate(model)?; Ok(EngineSettings { model, - artifacts: model::engine_artifacts(model, self.speculative.dspark_enabled, models_path), + artifacts: model::engine_artifacts( + model, + self.speculative.legacy_mtp_enabled, + self.speculative.dspark_enabled, + models_path, + ), context_tokens, execution: self.execution.engine_settings(), speculative: self.speculative.engine_settings(), @@ -438,6 +451,9 @@ impl ExecutionPreferences { { return Err("CPU helper threads is too large.".into()); } + if self.cpu_threads.is_some() { + return Err("CPU helper threads do not apply to the Rust Metal executor.".into()); + } if self .power_percent .is_some_and(|power| !(1..=100).contains(&power)) @@ -680,15 +696,23 @@ mod tests { (0, 0, 0) ); - let tuned = ExecutionPreferences { + let unsupported_threads = ExecutionPreferences { cpu_threads: Some(100), + ..ExecutionPreferences::default() + }; + assert!( + unsupported_threads + .validate(ModelChoice::DeepSeekV4Flash) + .is_err() + ); + + let tuned = ExecutionPreferences { power_percent: Some(50), prefill_chunk: Some(4096), ..ExecutionPreferences::default() }; assert!(tuned.validate(ModelChoice::DeepSeekV4Flash).is_ok()); assert!(tuned.validate(ModelChoice::Glm52).is_err()); - assert_eq!(tuned.engine_settings().cpu_threads, MAX_CPU_THREADS); } #[test] @@ -717,6 +741,21 @@ mod tests { }; assert!(glm.validate(ModelChoice::Glm52).is_ok()); assert!(glm.validate(ModelChoice::DeepSeekV4Pro).is_err()); + + let legacy = SpeculativePreferences { + legacy_mtp_enabled: true, + ..SpeculativePreferences::default() + }; + assert!(legacy.validate(ModelChoice::DeepSeekV4Flash).is_ok()); + assert!(legacy.validate(ModelChoice::DeepSeekV4Pro).is_err()); + assert!( + SpeculativePreferences { + dspark_enabled: true, + ..legacy + } + .validate(ModelChoice::DeepSeekV4Flash) + .is_err() + ); } #[test] @@ -787,14 +826,14 @@ mod tests { assert_eq!(engine.steering.ffn_scale, 1.0); assert_eq!(engine.diagnostics.simulated_used_memory_bytes, 8 * GIB); - let incompatible = RuntimePreferences { + let combined = RuntimePreferences { speculative: SpeculativePreferences { dspark_enabled: true, ..SpeculativePreferences::default() }, ..runtime }; - assert!(incompatible.validate(ModelChoice::DeepSeekV4Flash).is_err()); + assert!(combined.validate(ModelChoice::DeepSeekV4Flash).is_ok()); } #[test]