Achievable 253 t/s - unsloth/Muse Glimmer 30B UD-Q5_K_M on a 5090

Benchmarked Muse Glimmer 30B on my RTX 5090 (32GB), 262k context, UD-Q5_K_M + dflash-kquant + mmproj. Workload Stock master + DFlash ngram-simple PR #26842 + DFlash Code patch 78 t/s 57 t/s 220-253 t/s Mixed agent turn 77 t/s 68 t/s 188-213 t/s Tool-call JSON 71 t/s 75 t/s 155-181 t/s Heavy reasoning 52 t/s 58 t/s 120-130 t/s PR #26842 moves the DFlash draft argmax from CPU to GPU, which was the bottleneck. I cherry-picked it onto master (it branched before the Muse merge, one conflict to resolve manually) and it builds clean. Code generation now matches Meta's published 233 t/s, which I could not reproduce on stock master. Notes: ngram-simple loses to DFlash on every coding workload. Server caps context at the model's metadata context_length, use --override-kv for 262k. The reasoning budget flags do not work with this template. This is verified: with the budget set to 64, the model still burned 2000+ chars thinking and the budget message never appeared. Leave max_tokens headroom for the reasoning block. Flags: llama-server ^ --model Muse-Glimmer-30B-UD-Q5_K_M.gguf ^ --mmproj mmproj-kquant.gguf ^ -c 262144 --parallel 1 ^ --override-kv "muse-glimmer.context_length=int:262144,dflash.context_length=int:262144" ^ --kv-unified -ngl all -ngld all -fa on --warmup ^ -b 4096 -ub 512 --cache-prompt --cache-reuse 256 --slot-prompt-similarity 0.5 ^ -t 8 -tb 8 -td 4 -tbd 4 -fit off --load-mode mmap ^ --spec-draft-model dflash-kquant.gguf --spec-draft-ngl 999 --spec-draft-n-max 15 ^ --spec-type draft-dflash --spec-draft-type-k f16 --spec-draft-type-v f16 ^ --reasoning-preserve ^ --temp 1.0 --top-p 0.95 --min-p 0.0 --top-k 64 --jinja --port 9090 Caveat: #26842 is a draft PR, use at your own risk. Rebuild from master once merged. submitted by /u/patricious [link] [comments]

Source: r/LocalLLaMA | 2026-08-10

Related