Learning how to Forget: Fine-tuning for Long-Context Sparse Attention

arXiv:2608.19920v1 Announce Type: new Abstract: A lot of prior work addressed key-value (KV) cache selection and compression by sparse attention to enable long-context inference for transformer language models without excessive hardware budgets. We provide a new method for fine-tuning models with sparse attention. It works for any KV cache policy, runs on a moderate hardware budget (e.g., a single Nvidia A100 GPU with 40 GB RAM), and allows the model to co-adapt with the policy, often outperforming models trained with exact attention (sequence parallelism). We also provide an efficient implementation of H2O sparse attention (the leading policy in our experiments) with dedicated scaled dot product attention kernel support. KeysAndValues (https://github.com/awslabs/keys_values), a new open source library for long-context inference and fine-tuning, provides easy-to-use and performant code for all methods discussed here.

xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extraction
Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Serving
Looped Latent Attention: Cross-Loop KV Compression for Looped Transformers
Compress the Context, Keep the Commitments: A Formal Framework for Verifiable LLM Context Compression
Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

Source: arXiv cs.CL | 2026-08-21

Related