Aligning What LLMs Do and Say: Towards Self-Consistent Explanations

arXiv:2506.07523v3 Announce Type: replace Abstract: Large language models (LLMs) seem to offer an easy path to interpretability: just ask them to explain their answers. Yet the features driving an answer often differ from those emphasized in its explanation, meaning post-hoc rationales can misrepresent what actually shaped the model's output. We quantify this gap by comparing the feature-importance distributions of answers and their explanations. Prior analyses reveal such discrepancies, but large-scale study has been limited by the high computational cost of attribution methods. To address this, we introduce the Post-hoc Self-Consistency Bank (PSCB), a large-scale benchmark linking model decisions with diverse explanations and attribution vectors across datasets, methods, and model families. Using PSCB, we find that Spearman rank correlation provides a more reliable signal of alignment than cosine similarity. Building on this insight, we apply Direct Preference Optimization (DPO) to attribution-based preference data, improving alignment without degrading task accuracy, and show that standard supervised fine-tuning on the same data fails to achieve comparable gains. These improvements generalize robustly across domains, paving the way toward scalable and faithful alignment between LLM decisions and their natural language explanations.

Related

• ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
• IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
• ConsistRM: Improving Generative Reward Models via Consistency-Aware Self-Training
• Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text
• ReFACT: A Benchmark for Scientific Confabulation Detection with Positional Error Annotations
• FaithLens: Detecting and Explaining Faithfulness Hallucination

Source: arXiv cs.CL | 2026-04-14