Model Releases
Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant
So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to mak
So after all my work, yeah, Raymond did it better, so I ported his work over, extended it turboX, extended it multiple other models (he had only Qwen models), and benchmarked the crap out of it to make sure it was worth it still. So really the credit goes to Raymond ( https://github.com/RaymondHuang210129/llama.cpp-adaptive-kv-streaming ) submitted by /u/giveen [link] [comments]
Related
- Over 200k context on 16GB VRAM with Qwen 3.8 27B UD-IQ3_XXS
- After pushing 1M+ tokens through Qwen 3.8 27B, here is my optimal llama.cpp config for 16GB VRAM (73k Context, Agentic Coding)
- (NInfer Fork) I wanted to have a 1M context Qwen-3.8 27B, tp2, dual 5090s
Source: r/LocalLLaMA | 2026-09-06