GPTBIAS: A Comprehensive Framework for Evaluating Bias in Large Language Models

arXiv:2312.06315v2 Announce Type: replace Abstract: Warning: This paper contains content that may be offensive or upsetting. There has been a significant increase in the usage of large language models (LLMs) in various applications, both in their original form and through fine-tuned adaptations. As a result, LLMs have gained popularity and are being widely adopted by a large user community. However, one of the concerns with LLMs is the potential generation of socially biased content. The existing evaluation methods have many constraints, and their results exhibit a limited degree of interpretability. In this work, we propose a bias evaluation framework named GPTBIAS that leverages the high performance of LLMs (e.g., GPT-4 ite{openai2023gpt4}) to assess bias in models. We also introduce prompts called Bias Attack Instructions, which are specifically designed for evaluating model bias. To enhance the credibility and interpretability of bias evaluation, our framework not only provides a bias score but also offers detailed information, including bias types, affected demographics, keywords, reasons behind the biases, and suggestions for improvement. We conduct extensive experiments to demonstrate the effectiveness and usability of our bias evaluation framework.

Related

• SMARTER: A Data-efficient Framework to Improve Toxicity Detection with Explanation via Self-augmenting Large Language Models
• Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
• Mitigating Coordinate Prediction Bias from Positional Encoding Failures
• Evaluating LLMs for Demographic-Targeted Social Bias Detection: A Comprehensive Benchmark Study

Source: arXiv cs.CL | 2026-09-03