Model Releases
New beellama fork 76% faster tg with kvarn KV quants
When using kvarn quants at low context depth tg speed is similar to llama.cpp on and equivalent qx_x quant. However, as context depth grows kvarn tanks your tg speed. This fork optimises kvarn to have
When using kvarn quants at low context depth tg speed is similar to llama.cpp on and equivalent qx_x quant. However, as context depth grows kvarn tanks your tg speed. This fork optimises kvarn to have similar or better performance ay high context depths than llama.cpp at an equivalent qx_x quant and in my testing up to 76% faster tg than beellama's implementation of kvarn. My testing capped at ctx 99328 but for higher context your gains will be even better. From the github (translated from Russian) at ctx 163840: https://preview.redd.it/qz6s5giclrmh1.png?width=621&format=png&auto=webp&s=a14564598d11c7c3c2078ed5f60422beb6c09917 But my own testing was even better. https://preview.redd.it/zdog4fwolrmh1.png?width=570&format=png&auto=webp&s=acdbbabf252251e9ada23589690939a7f8504e0d https://preview.redd.it/g6tjj73wlrmh1.png?width=571&format=png&auto=webp&s=6107e71595ad8f506053f3b9aa01f39257b732cc https://preview.redd.it/qbxxve70mrmh1.png?width=563&format=png&auto=webp&s=304b1b6bcd6790af172707dd7e6e2dee1f7bc0d7 As you can see results are more similar to q3_0 at various depths. Here are q4 quants: https://preview.redd.it/110ol7iamrmh1.png?width=573&format=png&auto=webp&s=0881d58100872568ca8ff52409017aa3d187f1b0 https://preview.redd.it/6t4s1ptbmrmh1.png?width=572&format=png&auto=webp&s=a10e9a74aef962504bb6598041072b9c97540503 https://preview.redd.it/swaxghxcmrmh1.png?width=568&format=png&auto=webp&s=8228f9a92497bc3371711dceb0fd7a72f68e4a22 And here are KV q5/q4: https://preview.redd.it/6kxx6xahmrmh1.png?width=563&format=png&auto=webp&s=c2a0a083a8ce566390b96bee0d9b12d4d40afa3d https://preview.redd.it/aydulleimrmh1.png?width=567&format=png&auto=webp&s=90a6127fc8d031e93d588863777d0c184a3d8b06 https://preview.redd.it/zapn0pfkmrmh1.png?width=564&format=png&auto=webp&s=ec0774cab1b72e30a3e871b67d075d987ead7a7b As you can see this fork brings kvarn performance to about the same as qx_x. My only question is does this optimisation break anything. I will have to compare KLD between beellama and this fork. I did try testing this but for some reason kvarn causes llama-perplexity to be very slow. Sorry I could not test higher quants or larger context sizes on my 16GB VRAM. As soon as I drop -ngl my pp tanks to under 200t/s when using kvarn on either beellama or this fork. If anyone has a solution, such as a compile flag or llama setting let me know. Running on Win 11, 5070Ti 16GB VRAM, 48GB RAM using Qwen3.8 IQ4 XS without MTP or tail tokens. submitted by /u/Fancy-Snow7 [link] [comments]
Related
- KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates
- BeeLlama.cpp v0.4.1: KVarN, KV precision tail, q2_0-q3_1 KV cache, improved support. KLD benchmarks: tail 1024 makes kvarn5 and q6_0 match q8_0, for much less VRAM
- llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch
Source: r/LocalLLaMA | 2026-08-31