Qwen3.8-27B at 262K context on a Strix Halo + RTX 3090 Ti: 9.5 -> 153 tok/s, and it beats a dual-3090 vLLM box on HumanEval

Spent a while treating layer placement, KV format and llama.cpp itself as experimental variables. 159 logged experiments. Numbers first, caveats after. Hardware: AMD Ryzen AI MAX+ 395 (Strix Halo, 128 GB unified) + RTX 3090 Ti on an eGPU link. One llama.cpp process, AMD on Vulkan, NVIDIA on CUDA, one 27B model split across both. Baseline: 9.474 tok/s. Target-only, no speculation, AMD side alone. Now, code-shaped generation: 32K context: 153.32 tok/s 200K context: 87.74 tok/s HumanEval, 164 problems, driven by a real coding agent (pi-agent), graded by executing the official tests: Local: 159/164 in 29.7 min Remote 2x RTX 3090 running vLLM TP2: 157/164 in 42.4 min Long-context retrieval suite: 15/15, 352 s local vs 551 s on the remote. Things that actually moved the needle, roughly in order of surprise: The chat template. Swapping to a terser template (Qwen-Sharp) cut wall time 44% and output tokens 51%, with no accuracy change. Bigger than several weeks of GPU work. It just stops the model narrating. KV cache format as a placement lever. Going q8_0 -> q4_0 on both K and V freed exactly 2,176 MiB, which was enough to move every full-attention layer onto the fast card at full 262K. +28% prefill, +20% generation. Qwen3.8 is 3:1 Gated DeltaNet to full attention, so only 16 of 64 layers have KV at all, and those are the only ones whose cost scales with context. Which layer sits on which GPU matters enormously. --spec-type draft-mtp,ngram-mod. The flag accumulates. n-gram on top of MTP is worth +72 to +140% on code-shaped output, about -1% on prose, and costs nothing in VRAM. Free win if you generate code. A one-line llama.cpp patch: --spec-draft-ubatch. The speculative context was inheriting the target's 512 micro-batch and reserving a 2.2 GiB compute buffer to draft 4 tokens. Setting it to 64 freed 1,039 MiB for a 1.78% prefill cost, which bought another layer of placement. MTMD_BACKEND_DEVICE=Vulkan1 to put the vision encoder on the idle iGPU. Undocumented as far as I can tell. Default puts it on the first GPU-type device, which is the already-full 3090 Ti, and it dies allocating 884 MiB. Things that did NOT work, so you don't have to try them: External draft models (0.8B, 4B): 5.06-5.77 tok/s, worse than no speculation. Sequential drafting latency across PCIe dominates. Multi-lane parallel drafting: down to 1.62 tok/s. The 27B verification pass is the bottleneck; anything that widens the verification graph loses. Q6_K: 2.3x the wall time, marginally lower accuracy (because if hit the 64K output limit wall twice). Only 15.6 of its 20.5 GiB fits on the card, so 4 attention layers get pushed to the iGPU instead of 1. Caveats, because this is r/LocalLLaMA and you'd find them anyway: 153 tok/s is code-shaped generation at 32K where n-gram does heavy lifting. Same config on prose at 200K is 35.8. Different workloads, kept separate on purpose. The remote comparison is deployed-stack vs deployed-stack, not hardware isolated: it serves an AWQ-MTP fine-tune, local serves Q4_K_M of the base. I asked for standalone functions rather than the canonical prompt+completion format, so 3 of the 5 HumanEval failures are NameErrors on helper functions that the official harness would have kept in scope. 159/164 is probably a slight undercount. Same undercount applies to every arm. q4_0 keys are a real quality change. Passed my 15/15 gate; validate on your own workload before trusting it. Full writeup with every flag, patch and failed branch: https://definedrr.medium.com/qwen3-8-27b-9-tokens-per-second-to-153-ee3781f4a3f5?sharedUserId=definedrr submitted by /u/TrifleHopeful5418 [link] [comments]

Source: r/LocalLLaMA | 2026-08-21