CachyLLama: llama.cpp fork with persistent SSD-backed KV caching for local agent workflows

If you run local agentic coding harnesses (Aider, Claude Code, etc.), prompt evaluation usually eats up most of your execution time. Every turn re-evaluates thousands of identical prefix tokens_system prompts, tool schemas, and conversation history. CachyLLama is a llama.cpp fork built specifically to solve this bottleneck on mid-tier hardware and APUs where generation speeds are fine, but prompt processing is brutal.Key FeaturesPersistent On-Disk KV Cache: Saves conversation checkpoints to SSD. State survives server restarts and power cycles, restoring from disk on cold start. Dedicated System Prompt Cache: Maintains a global, cross-conversation cache for static prefixes. Subsequent requests skip reevaluation entirely.Hybrid MoE/SSM Support: Properly tracks and restores recurrent state alongside attention cells for hybrid architectures (Qwen 3.5/3.6, Gemma 4, GLM-4.7, DeepSeek-V3).Multi-Tiering: Keeps active states in RAM, demotes idle sessions to disk, and uses kernel readahead to overlap disk I/O with compute work. Official Benchmarks (AMD Ryzen 7840U / 780M) Note: CachyLLama does not speed up token generation itself; it solely eliminates redundant prompt-processing overhead. Prompt Size ~1,243 tokens, Cold Start 9.3s, Warm (Cached) 0.41s Prompt Size ~15,700 tokens, Cold Start 143.1s, Warm (Cached) 0.99s submitted by /u/UsedMorning9886 [link] [comments]

Source: r/LocalLLaMA | 2026-07-25

Related