SinkTrack: Attention Sink based Context Anchoring for Large Language Models

arXiv:2604.10027v1 Announce Type: new Abstract: Large language models (LLMs) suffer from hallucination and context forgetting. Prior studies suggest that attention drift is a primary cause of these problems, where LLMs' focus shifts towards newly generated tokens and away from the initial input context. To counteract this, we make use of a related, intrinsic characteristic of LLMs: attention sink -- the tendency to consistently allocate high attention to the very first token (i.e., ) of a sequence. Concretely, we propose an advanced context anchoring method, SinkTrack, which treats as an information anchor and injects key contextual features (such as those derived from the input image or instruction) into its representation. As such, LLM remains anchored to the initial input context throughout the entire generation process. SinkTrack is training-free, plug-and-play, and introduces negligible inference overhead. Experiments demonstrate that SinkTrack mitigates hallucination and context forgetting across both textual (e.g., +21.6% on SQuAD2.0 with Llama3.1-8B-Instruct) and multi-modal (e.g., +22.8% on M3CoT with Qwen2.5-VL-7B-Instruct) tasks. Its consistent gains across different architectures and scales underscore the robustness and generalizability. We also analyze its underlying working mechanism from the perspective of information delivery. Our source code is available at https://github.com/67L1/SinkTrack.

Related

• Attention Sinks as Internal Signals for Hallucination Detection in Large Language Models
• HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Models
• TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection
• Mitigating Entangled Steering in Large Vision-Language Models for Hallucination Reduction
• Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs

Source: arXiv cs.CV | 2026-04-14