Model Releases
BeeLlama.cpp v0.4.1: KVarN, KV precision tail, q2_0-q3_1 KV cache, improved support. KLD benchmarks: tail 1024 makes kvarn5 and q6_0 match q8_0, for much less VRAM
TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), a
TL;DR llama.cpp fork with more KV cache quantization features, with all claims supported by benchmarks: KVarN, KV cache precision tail, additional types of standard KV cache (q2_0-q3_1, q6_0, q6_1), and more. BeeLLama v0.4.1 is here, building up on top of v0.4.0 feature set, now with better backend and model support. KVarN. Variance-normalized KV-cache quantization (paper) with better precision per bit. Although it was already introduced a few weeks ago in v0.3.2 Preview, that was a very raw implementation, with performance issues and VRAM usage spikes. Now in v0.4.1 it's the real deal: the precision is still above what usual quants offer for the same bit width, but now with very modest sacrifices to prefill, decode, and memory. KV cache precision tail. A promising new feature in the domain of mixed-precision KV cache. It allows to specify a specific numbers of recent tokens that will be stored in BF16 or F16, with the rest of KV cache being quantized as usual. This way we can store the hottest tokens in a lossless fashion, preventing a model from misreading your task details, code, or data. Additional types of standard KV cache. q6_0 and q6_1 join the high end of the ladder, allowing to fine-tune precision vs VRAM in-between upstream's q5_0/1 and q8_0 types. q2_0, q2_1, q3_0 and q3_1 are added as a replacement for turbo3 and turbo2 for cases where KVarN doesn't work well, but you just can't fit everything into VRAM without extreme quantization. Please note that for SWA architecture (Gemma, GPT-OSS) the precision of KVarN and KVPT is the same, but VRAM and performance costs are higher due to complications between SWA ring and mixed precision KV cache. GitHub repo: https://github.com/Anbeeld/beellama.cpp KLD results for Qwen 3.6 27B Q5_K_S 64k Here are all symmetrical qX_0 pairs and kvarnX pairs where X >= 4 with tail 0/1024/2048, compared against q8_0 t0 from the same benchmarks, and sorted by ratio between median KLD and VRAM costs. Full benchmark data and analysis: KV Cache Precision Tail: Implementation and Benchmarks. Cache Tail KV MiB Size vs q8_0 Median/size vs q8_0 Median vs q8_0 P99.9 vs q8_0 kvarn4 1024 1232.00 56.6% 1.62 91.4% 102.9% kvarn4 2048 1296.00 59.6% 1.60 95.5% 95.6% kvarn4 0 1184.00 54.4% 1.50 81.8% 82.5% q4_0 1024 1248.00 57.4% 1.50 86.0% 89.0% q4_0 2048 1312.00 60.3% 1.48 89.2% 100.6% kvarn5 0 1440.00 66.2% 1.48 98.1% 107.5% kvarn5 1024 1488.00 68.4% 1.48 101.3% 106.1% kvarn5 2048 1552.00 71.3% 1.43 101.9% 105.6% q5_0 1024 1504.00 69.1% 1.40 96.9% 105.6% q5_0 2048 1568.00 72.1% 1.36 98.0% 103.7% kvarn6 0 1696.00 77.9% 1.31 102.2% 104.5% kvarn6 1024 1744.00 80.1% 1.29 103.4% 109.9% kvarn6 2048 1808.00 83.1% 1.25 103.8% 108.1% q6_0 0 1664.00 76.5% 1.24 94.7% 102.1% q6_0 1024 1760.00 80.9% 1.24 100.1% 109.2% q5_0 0 1408.00 64.7% 1.22 78.8% 95.8% q6_0 2048 1824.00 83.8% 1.20 100.6% 103.5% kvarn8 0 2208.00 101.5% 1.03 104.4% 104.9% kvarn8 1024 2256.00 103.7% 1.01 104.4% 106.2% q8_0 0 2176.00 100.0% 1.00 100.0% 100.0% q8_0 1024 2272.00 104.4% 0.97 101.3% 106.1% kvarn8 2048 2320.00 106.6% 0.97 103.6% 104.7% q8_0 2048 2336.00 107.4% 0.95 101.6% 106.8% q4_0 0 1152.00 52.9% 0.93 49.2% 60.2% submitted by /u/Anbeeld [link] [comments]
Source: r/LocalLLaMA | 2026-07-26