Local Ai
SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs
arXiv:2604.07737v1 Announce Type: new Abstract: While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when
arXiv:2604.07737v1 Announce Type: new Abstract: While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose Separate Sequence (SepSeq), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention sink, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6% across diverse domains while reducing total inference token consumption by 16.4% on average.
Related
- DYCP: Dynamic Context Pruning for Long-Form Dialogue with LLMs
- A Decomposition Perspective to Long-context Reasoning for LLMs
- Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
- Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework
- Prompt reinforcing for long-term planning of large language models
Source: arXiv cs.CL | 2026-04-10