Model Releases
PSA: llama.cpp -cram should be increased for agentic workflows (default is 8192)
Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a poin
Just a quick PSA. llama.cpp does have prompt caching. if you are running large context lengths and have long multiturn projects, increasing -cram can provide you with massive speedups. There is a point where context lengths can get so large that 8192mb is not enough and the whole context needs to be re processed again on every turn. personally, I have found 20480 to work well with Qwen 27B 3.8 at 262K context. the main downside is this uses more ram. vram usage doesnt increase. submitted by /u/My_Unbiased_Opinion [link] [comments]
Related
- CachyLLama: llama.cpp fork with persistent SSD-backed KV caching for local agent workflows
- How do you deal with long-context sessions after restarting llama.cpp?
- Is llama.cpp meant to be slow at long context, even when you aren't using that context?
- 3 days benchmarking most llama.cpp flags on my weird 40gb vram laptop + tb4 egpu setup. Got +70% generation, +40% prefill, 60k more context, and filed a bug in llama around MTP. What I learned.
Source: r/LocalLLaMA | 2026-09-24