Assessing and Refining ChatGPT's Performance in Identifying Targeting and Inappropriate Language: A Comparative Study

arXiv:2505.21710v2 Announce Type: replace Abstract: This study evaluates the effectiveness of ChatGPT, an advanced AI model for natural language processing, in identifying targeting and inappropriate language in online comments. With the increasing challenge of moderating vast volumes of user-generated content on social network sites, the role of AI in content moderation has gained prominence. We compared ChatGPT's performance against crowd-sourced annotations and expert evaluations to assess its accuracy, scope of detection, and consistency. Our findings show that ChatGPT performs well in detecting inappropriate content, and that almost the entire gain from iterative prompt refinement came from a single change --- simplifying the label scheme and moving to GPT-4 --- after which further adjustments to prompt structure and sampling temperature had little effect. Its performance on targeting language was weaker and less stable, with higher false positive rates than expert judgments before refinement and higher false negative rates after a change of model. This study contributes to the field by demonstrating the potential of AI models like ChatGPT to enhance automated content moderation systems while also identifying areas for further improvement. The results underscore the importance of careful prompt design and model selection, and of contextual understanding, in supporting automated moderation and mitigating harmful online behavior.

Source: arXiv cs.CL | 2026-10-15

Related