Knowing Is Not Enough: Information Retrievability as a Precondition to Effective LLM Oversight

arXiv:2609.01976v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly embedded in organizational work, yet their errors often pass human review. Prior research locates such failures in users' capability to review LLM output or their engagement in doing so. We develop an alternative, retrieval-based account of human oversight and posit that error detection is more effective when oversight-relevant information is accessible to users at the moment of review. Across two randomized lab-in-the-field experiments with 640 customer-facing employees, we show that self-generated explanations improve error detection and strengthen recall of verification-relevant reasoning, while cues that reactivate such reasoning help sustain detection under repeated LLM use. Theoretically, we identify information retrievability as a distinct precondition for effective oversight and specify generative encoding and cue-supported reactivation as mechanisms that build and sustain it. Practically, lightweight onboarding self-explanations and daily retrieval cues can make human oversight more resilient as LLM use becomes routine.

Related

• Do large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effects
• Reading Is Not Using: Retrieval, Judgment, and the Design of AI Financial Research Workflows
• Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination

Source: arXiv cs.AI | 2026-09-03