Add GLM 5.3 Flash support

This commit is contained in:
Georg Bauer
2026-09-01 19:06:20 +02:00
parent 9a33c61ea6
commit 46d6a976a5
31 changed files with 7806 additions and 447 deletions

View File

@@ -1,5 +1,69 @@
# Apple Metal parity for issue #78
## GLM 5.3 Flash parity target
- Previous DS4 oracle: `8db89fe083ae4d17c9a2428ccd29803d3ae8f577`
- Current DS4 oracle: `ec7642cdd9ec81d01ad4b1fd8f8a3d1511533748`
- Main artifact: `GLM-5.3-Flash-Q2.gguf`, SHA-256
`e81fd6241c6e55a64e1e14e47a3eab61a173fa8d7e4b5c1d1848827119705b32`
- Vision artifact: `GLM-5.3-Flash-Vision-Encoder.gguf`, SHA-256
`ae23e14c6979e889051b2e4a39351abcdafb161e18e606fae4d8c40095a4bf3a`
- Machine: MacBook Pro `Mac17,7`, Apple M5 Max, 128 GiB unified memory,
resident Q2 weights
The applicable commits after the previous oracle are:
| DS4 commit(s) | Scope carried into DS4Server |
| --- | --- |
| `147109a` | GLM 5.3 shape, GGUF binding, KDA/DSA attention, hyperconnections, routed MoE, BF16 tensors, output head, integrated MTP, and Metal kernels |
| `4771329`, `9f95d9f` | Vision encoder, image preprocessing, prompt overlays, and compact-prefill correctness |
| `b0c31af` | Pooled indexer cache, bounded attention memory, and batched prefill |
| `747eafc`, `1df8ff0` | Fixed-image encoder oracle, replay/logit checks, and multimodal quality gates |
| `8adb221` | Directional attention and FFN steering at the DS4 residual points |
| `639d4eb` | Continued-prefill cache and position behavior |
| `6cf658a` | BF16 decode projection schedule and Metal kernel specialization |
| `01f3316`, `81c8c39`, `5da3206` | Current Metal compact-cache ABI, Q2/full-checkpoint binding, and GLM streaming compatibility |
CUDA- and ROCm-only commits in the range are excluded. The Rust release gates
compare three-run medians against this exact DS4 build, require identical
greedy token hashes, and fail only when Rust is more than 2% slower than DS4;
there is no upper bound when Rust is faster. Text measures an
8192-token frontier plus steady decode, both with and without integrated MTP.
Vision compares the fixed encoder embedding (cosine at least 0.995, mean
absolute error at most 0.001, maximum error at most 0.06), encoder latency,
multimodal prefill, steady decode, image-conditioned logits, and replay.
### GLM 5.3 Flash acceptance results
The text fixture is `ds4.c`, rendered as one no-system chat turn (`-sys ''` in
DS4), truncated to exactly 8192 tokens, with a 32768-token allocation and 64
greedy output tokens. Same-process three-run medians are used because both DS4
and DS4Server exhibit a load-order/thermal artifact with this 96 GB model.
| Capability | Metric | DS4 | DS4Server | Result |
| --- | --- | ---: | ---: | --- |
| Text | Prefill tok/s | 288.91 | 295.21 | Rust 2.18% faster |
| Text | Decode tok/s | 22.58 | 22.78 | Rust 0.89% faster |
| Integrated MTP | Decode tok/s | 26.55 | 26.80 | Rust 0.93% faster |
| Vision encoder | Fixed-image seconds | 2.6255 | 2.4354 | Rust 7.24% faster |
| Vision prompt | Prefill tok/s | 185.81 | 201.51 | Rust 8.45% faster |
| Vision prompt | Decode tok/s | 26.40 | 27.21 | Rust 3.09% faster |
Text and MTP produced the same token SHA-256,
`0b61e613708d8367fc87f5a32fbfd4ba098b40ee920645b1e38381e867131108`.
The 1094-token fixed-image prompt produced the same vision token SHA-256,
`424fdadc10c0a2bd95b405a3985151e84aa844a209d2a2ca72cd01c39dfa53db`.
The Rust vision embedding compared with the DS4 oracle at cosine
`0.999958726`, mean absolute error `0.000036702`, and maximum error
`0.006153620`.
The checkpoint advertises a 1048576-token maximum. On this 128 GiB machine,
32768 is the default manageable context and 50000 is the extended option: the
50000-token run used 3.51 GiB of live runtime tensors in addition to 90.61 GiB
of mapped model wrappers and retained the exact output hash. Larger allocations
leave too little operating-system and application headroom for a reliable
default.
## Frozen comparison
- DS4 oracle: `8db89fe083ae4d17c9a2428ccd29803d3ae8f577`