Research
FlexComp: One Model for Every Ratio in Context Compression
arXiv:2609.11192v1 Announce Type: new Abstract: Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix
arXiv:2609.11192v1 Announce Type: new Abstract: Soft context compression condenses a context into a few memory tokens that a frozen LLM consumes in place of the raw text, but existing compressors fix the compression ratio at training and inference: each deployed ratio requires a separately trained model, and the chosen ratio is applied uniformly to all inputs, whose actual needs vary drastically. We propose FlexComp, a method-agnostic framework that decouples the ratio from both training and deployment: Matryoshka-style training samples the memory budget K per instance, turning one model into an any-ratio compressor, and the budget is then chosen per input by: (1) confidence-based cascade routing or (2) a lightweight learned K predictor. Across ICAE, 500xCompressor, and SAC on MRQA, a single FlexComp model matches separately trained fixed-ratio specialists with minimal degradation. Cascade routing preserves over 98% of the mildest ratio's accuracy at up to 266x average compression; the K predictor, in a single compression-decoding pass, reaches 158-236x within 0.7 F1 of the mildest ratio. At serving-scale batch sizes, the K predictor cuts context KV cache by 50% and improves decoding throughput by 47%.
Related
- SEER: Long-Context Reasoning via Selective Visual-Text Compression
- GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent
- MoE-nD: Per-Layer Mixture-of-Experts Routing for Multi-Axis KV Cache Compression
- AnchorKV: Anchor-Residual KV Cache Compression
- ResKV: Reconstructing Omitted Attention Contributions for Fixed-Budget KV Cache Compression
Source: arXiv cs.CL | 2026-09-11