Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety

arXiv:2512.06227v3 Announce Type: replace Abstract: Real-world indicators play an important role in many Natural Language Processing (NLP) applications, such as life events for mental health analysis and risky behaviours for online safety, yet labelling such information is often costly and/or difficult due to its multi-label and dynamic nature. Large Language Models (LLMs) show promising potential for automated annotation, but the multi-label setting remains challenging. In this work, we propose a Confidence-Aware Fine-Grained Debate (CFD) framework that simulates human collaborative annotation using fine-grained communication to better support automated multi-label enrichment. We introduce two expert-annotated resources: life-event and symptom annotation for a mental health well-being dataset, and a new online safety sharenting dataset. Experiments show that CFD achieves the most robust enrichment performance across tasks, and that the benefit of fine-grained confidence is influenced by its quality and variability. We further evaluate training-free strategies for incorporating enrichment indicators into downstream tasks and show that automated enrichment consistently improves performance, by up to 9.9 Macro-F1 points, with the most effective integration format depending on how the indicator relates to the downstream objective.

Related

• MHSafeEval: Role-Aware Interaction-Level Evaluation of Mental Health Safety in Large Language Models
• Whose Story Gets Told? Positionality and Bias in LLM Summaries of Life Narratives
• Cognitive Policy-Driven LLM for Diagnosis and Intervention of Cognitive Distortions in Emotional Support Conversation
• Graph2Counsel: Clinically Grounded Synthetic Counseling Dialogue Generation from Client Psychological Graphs

Source: arXiv cs.CL | 2026-08-12