Recency/Frequency Adaptive KV Caching for Large Language Model Serving
arXiv:2606.21238v1 Announce Type: cross Abstract: Key-value (KV) caching is a powerful technique for accelerating large language model inference and generation. Inference workloads are large and diver