SALT: Salience-Aware Lexical Trie for Long-Context Compression

arXiv:2607.17486v2 Announce Type: replace-cross Abstract: As large language models (LLMs) process increasingly longer prompts, computation and KV-cache memory costs have emerged as major bottlenecks in inference systems. Existing input-level prompt compression methods address this, but rank each sentence by a scalar relevance score, treating the document as an unstructured pool of words and sentences. Under tight budgets, this causes theme collapse, where the dominant theme(s) of a document consumes the budget, discarding less-frequent yet task-relevant themes. Preserving thematic coverage instead requires allocating the budget across recurring themes rather than scoring sentences in isolation. To this end, we propose SALT, a model-agnostic extractive framework that organizes per-sentence keywords into a trie ordered by sentence frequency (SF), a lightweight, reusable proxy for document thematic structure. This trie-based organization smooths memory allocation and prevents dominant themes from monopolizing the budget. Multi-anchor retrieval activates trie nodes labeled by query keywords at any depth, and the trie persists across dialogue turns, supporting multi-turn use without re-encoding the document. By preserving document themes, SALT reduces the prefill computation and memory cost of long-context prompts while remaining composable with KV-cache methods that target decoding-time latency and memory.

Related

• TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
• When Does Value-Aware KV Eviction Help? A Fixed-Contract Diagnostic for Non-Monotone Cache Compression
• Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM
• Moment-KV: Momentum-Based Decode-Time KV Cache Compression for Long Generation
• AdaComp: Extractive Context Compression with Adaptive Predictor for Retrieval-Augmented Large Language Models

Source: arXiv cs.AI | 2026-09-01