Model Releases
Qwen 3.8 27B UD-IQ4_XS even faster on 16GB CUDA
This is an evolution on top of Raymond's KV cache streaming fork - all credits to what enabled this goes to him. The basic idea behind what he enabled was a pool of memory in VRAM that is used differe
This is an evolution on top of Raymond's KV cache streaming fork - all credits to what enabled this goes to him. The basic idea behind what he enabled was a pool of memory in VRAM that is used differently depending on the phase (prompt processing or decoding) and when total used context is larger than what fits in VRAM it's instead streamed from host RAM in time for when the current layer needs it. It enables much higher TG tps than regular llama.cpp offloading to host RAM. I've used it to run Qwen 3.8 27B UD-IQ4_XS on my 5060Ti since release, but I've also had this idea that during the time the VRAM pool isn't fully utilized it should be possible to also do speculative decoding (MTP or DFlash2) - if it could be possible to eject the spec model and all the VRAM it uses, and then load it back when the context gets low enough again (compaction). I've got this working on my fork of Raymond's fork today. It's basically hot-swappable speculative decoding and while my focus is completely on making this work well on this specific model on 16GB, I would assume that part could also be useful for others with more VRAM. Repo here: https://github.com/troed/llama.cpp-adaptive-kv-streaming Regarding the images: Dark red = MTP ejected as soon as KV streaming starts. Light red = keeping it. The optimal setting is thus to eject it after a certain amount of pages (one page = 256 bytes) and then eject. Same for green (DFlash2) submitted by /u/tsangberg [link] [comments]
Related
- Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant
- KV cache quantization benchmarks: 413 pairs tested on Qwen 3.6 27B, Gemma 4 31B. KLD with BeeLlama.cpp v0.4.0: KVarN 6-bit beats q8_0, precision tail 1024 dominates
- llama.cpp PR reports up to 169% faster quantized-KV decode at 118K context on Intel Battlemage from one SYCL kernel switch
- BeeLlama.cpp v0.4.1: KVarN, KV precision tail, q2_0-q3_1 KV cache, improved support. KLD benchmarks: tail 1024 makes kvarn5 and q6_0 match q8_0, for much less VRAM
Source: r/LocalLLaMA | 2026-09-13