From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

arXiv:2506.23101v2 Announce Type: replace-cross Abstract: Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially sensitive domains. Existing benchmarks primarily assess bias in isolated contexts and are less suited to capturing biases that emerge through interpersonal interactions. We introduce Genres, a benchmark for evaluating gender bias in MLLMs through the lens of social relationships in narrative generation. Grounded in Fiske's relational models theory, Genres comprises 1,440 multimodal narrative elicitation pairs spanning four fundamental relationship types, diverse social scenarios, and three age groups. Its dual-character generation tasks elicit socially situated interpersonal narratives and enable fine-grained assessment across multiple dimensions of gender bias. Experiments on six recent MLLMs reveal persistent yet strongly relationship- and model-dependent gender disparities, with some patterns becoming more pronounced in interpersonal settings. We further evaluate two complementary generation-stage mitigation strategies. Our findings demonstrate the value of relationship-aware evaluation for uncovering subtle, interaction-driven gender bias in MLLMs.

Related

• HSSBench: Benchmarking Humanities and Social Sciences Ability for Multimodal Large Language Models
• Prototypicality Bias Reveals Blindspots in Multimodal Evaluation Metrics
• Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes
• Polyrating: A Cost-Effective and Bias-Aware Rating System for LLM Evaluation

Source: arXiv cs.AI | 2026-10-15