Popular but Wrong: Understanding and Mitigating LLM Overconfidence through Knowledge Popularity

arXiv:2505.17537v2 Announce Type: replace Abstract: Large language models (LLMs) often produce incorrect answers with high confidence, yet the factors associated with such overconfidence remain insufficiently understood. We study this problem through the lens of knowledge popularity. Using entity-centric factual QA derived from Wikidata triplets, we characterize popularity through question entity popularity, answer popularity, and question-answer co-occurrence. We find two consistent patterns. First, hallucinated answers are far from random: compared with ground-truth answers, they tend to be more popular or more frequently associated with the question entity. Second, confidence is strongly tied to the popularity of generated answers: even among incorrect predictions, more popular answers or those with higher question-answer co-occurrence receive higher confidence. Together, these findings suggest that popular but wrong alternatives may contribute to overconfidence. We further show that popularity-related signals can mitigate overconfidence and improve overall confidence estimation. Across six models and three datasets, incorporating knowledge popularity reduces average confidence on incorrect answers from 0.765 to 0.254 and overall ECE from 0.356 to 0.050, while improving Alignment from 77.08% to 83.72%.

Related

• ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
• Context-Fidelity Boosting: Enhancing Faithful Generation through Watermark-Inspired Decoding
• How Large Language Models Balance Internal Knowledge with User and Document Assertions
• MedFabric and EtHER: A Data-Centric Framework for Word-Level Fabrication Generation and Detection in Medical LLMs

Source: arXiv cs.CL | 2026-09-01