Beyond 'ChatGPT Can Make Mistakes': Designing Interventions to Support Metacognitive Monitoring in AI-Assisted Work

arXiv:2609.17065v1 Announce Type: cross Abstract: AI assistance places a metacognitive demand on users, who must judge their own competence and the system's. Yet designers lack comparative evidence on which interventions to choose, where to place them, and how to tell whether they worked. We elicited 30 interventions from 11 experts and, with prior work, organized them into a design space of time (when an intervention acts), level (whose competence is judged), and source (who supplies the monitoring cue). A between-subjects experiment (N = 917; 12 planning-and-organizing problems) compared a per-task reliability card, contrasting replies, pause points, and post-problem reflection against a baseline LLM assistant. Reliability cards and contrasting replies reduced estimation error and overconfidence and increased aggregate confidence discrimination. No task-performance improvement or average within-item discrimination gain was established. We contribute a shared vocabulary, a design space, and evidence that measured monitoring and task performance are separable design targets.

Related

• The Role of Instructional Guidance in Generative AI-Assisted Learning: Empirical Evidence from Construction Engineering Education
• AI-Generated Traces for Novice Programmers: Learning Effects and Learner Differences in a Multi-Institutional Study
• Preliminary Guidelines for Using and Evaluating GenAI Tools to Support Systematic Literature Reviews
• Self-Regulated Reading with AI Support: An Eight-Week Study with Students

Source: arXiv cs.AI | 2026-09-16