AMD llama.cpp: reducing MTP buffer overhead gave me 64K → 149K context for Qwen 27B

Available context length with and without the patch: Model: QWEN 27B ROCm stock patched Vulkan stock patched IQ4_XS Pure, single 16GB GPU 19.456 76.032 68,352 78,592 Q6_K_L on 16GB + 12GB 64,256 149,248 68,864 151,296 The issue is that llama.cpp overestimates the memory needed for MTP compute-buffer/scheduler allocation during auto-fit, that leaves much less ctx available to the user than what actually needed by MTP. This patch stops the fitter from throwing away context based on an inflated MTP memory estimate Patch, launch scripts used for llama-server and raw logs: https://store.piffa.net/lm/bug/ Tested against: llama.cpp version: 909, based on master commit 7bd8282 , ROCm 7.14 Especially for ROCm with double GPU (16GPU + 12GB here) the amount of ctx gain is substantial, with longer session ROCm allows almost double prefill performances vs Vulkan yet on the mainline code the price to pay in ctx reduction for the extra compute is taxing. You can build for both vulkan and ROCm backends at the same time, the idea is that Vulkan saves some more vRAM while ROCm gives better prefill performance. With a single 16GB GPU and limited ctx size you may wanna use Vulkan while when using 2 GPUs with layer splitting ROCm is worth the expense with this patch as you have much more ctx length for long sessions. submitted by /u/ea_man [link] [comments]

Related

• Qwen 3.6 27B flags/settings in llama.cpp
• Qwen 3.6 27B Q5 on 3x2080ti: 55tps with llama.cpp. Can I squeeze out more?
• BeeLlama.cpp: advanced DFlash & TurboQuant with support of reasoning and vision. Qwen 3.6 27B Q5 with 200k context on 3090, 2-3x faster than baseline (peak 135 tps!)
• KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates

Source: r/LocalLLaMA | 2026-08-09