b11642

server: save only restorable slot states to the prompt cache (#30498) server: save only restorable slot states to the prompt cache The idle-slot save loop stores every idle slot whenever a task starts, while only completion tasks restore from the prompt cache. Rerank, embedding, decision and infill states fill the cache up to its limit without ever being read back, and slots already cached are offered again at every task start. Each slot now tracks with cache_pending whether its state is restorable and missing from the cache: a completion release or an explicit slot restore sets it, prompt_save consumes it, prompt_clear resets it. Both save paths share this rule, and the LRU path restores only for completion tasks. server: test that decision slots skip the prompt cache Decision tasks never mark their slot as cacheable, so starting new tasks must not save their idle slots to the RAM prompt cache, with or without a unified KV cache. server: restore any reusing task from the prompt cache Embedding and rerank tasks never reuse their state, so their slots are cleared on release like child slots and the cache only holds states that a later task can restore. Infill and decision tasks now restore from the cache too, which replaces the cache_pending flag. Decision models disable the prompt cache unless --cache-ram is given, since their states seldom repeat. Website: https://llama.app Attestations: https://github.com/ggml-org/llama.cpp/attestations/56156929 macOS/iOS: macOS Apple Silicon (arm64) macOS Apple Silicon (arm64, KleidiAI enabled) DISABLED macOS Intel (x64) iOS XCFramework Linux: Ubuntu x64 (CPU) Ubuntu arm64 (CPU) Ubuntu s390x (CPU) Ubuntu x64 (Vulkan) Ubuntu arm64 (Vulkan) Ubuntu x64 (CUDA 12) - CUDA 12.8 libraries Ubuntu x64 (CUDA 13) - CUDA 13.4 libraries Ubuntu arm64 (CUDA 13) - CUDA 13.4 libraries Ubuntu x64 (ROCm 10.0) Ubuntu x64 (OpenVINO) Ubuntu x64 (SYCL FP32) Ubuntu x64 (SYCL FP16) Linux arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Android: Android arm64 (CPU) Android arm64 (Snapdragon: CPU, Adreno GPU, Hexagon NPU) - setup guide Windows: Windows x64 (CPU) Windows arm64 (CPU) Windows arm64 (OpenCL Adreno) Windows x64 (CUDA 12) - CUDA 12.4 DLLs Windows x64 (CUDA 13) - CUDA 13.4 DLLs Windows arm64 (CUDA 13) - CUDA 13.4 DLLs Windows x64 (Vulkan) Windows arm64 (Vulkan) Windows x64 (OpenVINO) Windows x64 (SYCL) Windows x64 (ROCm 10.0) openEuler: DISABLED openEuler x86 (310p) openEuler x86 (910b, ACL Graph) openEuler aarch64 (310p) openEuler aarch64 (910b, ACL Graph) UI: UI

Source: llama.cpp Releases | 2026-10-15