Model Releases
Is llama.cpp meant to be slow at long context, even when you aren't using that context?
I am trying out a few fine-tunes of Qwen 3.5 9B @ IQ4_XS @ 131K context and trying to go mostly local (free beats cheap, after all). However, it is much slower than at, say 16K context, even when I am
I am trying out a few fine-tunes of Qwen 3.5 9B @ IQ4_XS @ 131K context and trying to go mostly local (free beats cheap, after all). However, it is much slower than at, say 16K context, even when I am not actually using 131K tokens in the first place. Anyone know why this is? I am using the following command on an 8 GB laptop 4060 (I've tried to set aside ~1GB for the OS and whatnot). llama serve -hf bartowski/Ornith-1.5-9B-GGUF:IQ4_XS --fit on --cache-type-k q8_0 --cache-type-v q4_1 -c 131072 --temp 0.7 submitted by /u/Aggravating-Push-207 [link] [comments]
Related
- Block KV cache streaming: bound VRAM at long context via a shared CUDA phase arena by giveen · Pull Request #357 · TheTom/llama-cpp-turboquant
- How do you deal with long-context sessions after restarting llama.cpp?
- Anyone managed to get Qwen 3.8 27B running smoothly on vLLM? Can't get rid of endless thinking
- Qwen3.8-Flash-Next in llama.cpp vs SGLang vs FreeToken: 35s vs 258s to first token at full context. My findings on new PRs coming to engines.
- Local agentic coding Benchmark : Qwen 3.8 27B (in many weights quants / cache quants / engine / reasoning effort) vs others.
Source: r/LocalLLaMA | 2026-09-22