MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression
arXiv:2607.00760v1 Announce Type: new Abstract: Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving co