Toward Robust and Efficient ML-Based GPU Caching for Modern Inference

arXiv:2509.20979v2 Announce Type: replace Abstract: In modern GPU inference, cache efficiency remains a major bottleneck, and heuristic policies such as extsc{LRU} can perform far worse than the offline optimum. Existing learning-based caching systems improve hit rates mainly through predictor design, but often follow learned predictions blindly, making performance unreliable when predictions are inaccurate. In contrast, emerging learning-augmented caching algorithms~ite{pmlr-v80-lykouris18a,mitzenmacher2022algorithms} provide performance guarantees by carefully integrating predictions into caching policies, achieving both consistency (near-optimality under perfect predictions) and robustness (bounded worst-case performance under prediction errors). However, deployment remains challenging. A practical algorithm should satisfy strict time and space efficiency constraints, which some theoretical work overlooks, while also incurring low deployment overhead. We propose learning-augmented LRU, a deployment-oriented learning-augmented caching algorithm that guarantees 1-consistency and O(k)-robustness, incurs low time and space overhead, and maintains strong compatibility. We further build a GPU cache, called extsc{LCR}, on top of learning-augmented LRU to benefit from its theoretical guarantees and translate them into practical performance. In experiments, extsc{LCR} reduces P99 time-to-first-token (TTFT) by up to 28.3% on LLM workloads and increases throughput by up to 24.2% on deep learning recommendation (DLRM) workloads. Even with poor predictions, performance degrades gracefully and remains close to extsc{LRU}, demonstrating robustness with practical value.

Source: arXiv cs.LG | 2026-04-27

Related