Research
Attention-Guided Layer Selection for Contrastive Decoding in Large Language Models
arXiv:2607.23067v1 Announce Type: cross Abstract: Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and
arXiv:2607.23067v1 Announce Type: cross Abstract: Contrastive decoding methods such as DoLa improve the factuality of Large Language Models (LLMs) by contrasting the output distributions of mature and premature layers. However, DoLa's dynamic layer selection relies solely on divergences in output vocabulary distributions. In this work, we propose three attention-guided strategies: Attention-JSD, Attention-Entropy-Max, and Attention-Entropy-Min, which leverage structural information carried by internal self-attention mechanisms as a signal for layer selection. Experimental results on TruthfulQA demonstrate that our strategies, particularly Attention-JSD and Attention-Entropy-Min, consistently outperform the original DoLa. We observe significant gains on multi-answer metrics (MC2 and MC3), suggesting that attention distributions can provide a more sensitive signal for resolving factual knowledge than output vocabulary distributions.
Related
- SpecBound: Adaptive Bounded Self-Speculation with Layer-wise Confidence Calibration
- Adaptive Perturbation Selection for Contrastive Audio Decoding
- LC-ICL: Label-Guided Contrastive In-Context Learning for Robust Information Extraction
- Entropy-informed Decoding: Adaptive Information-Driven Branching
Source: arXiv cs.AI | 2026-07-28