Safety
MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches
arXiv:2604.22881v1 Announce Type: cross Abstract: Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long
arXiv:2604.22881v1 Announce Type: cross Abstract: Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%).
Related
- ProEval: Proactive Failure Discovery and Efficient Performance Estimation for Generative AI Evaluation
- ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
- Deep Interest Mining with Cross-Modal Alignment for SemanticID Generation in Generative Recommendation
Source: arXiv cs.AI | 2026-04-28