Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs

arXiv:2608.18131v1 Announce Type: new Abstract: Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages, the consequences are immediate and user-facing: voice assistants and spoken dialogue systems may produce stereotype-reinforcing outputs, bypassing the standard English-focused safety alignments and propagating harmful bias to non-English speaking communities. For spoken language technologies deployed across India's linguistically diverse population, this represents a critical failure mode. To address this cross-lingual gap, we introduce INCLUDE (Indian Cultural Lens for Understanding and Detecting Embedded Biases), a multilingual evaluation benchmark designed to quantify Indian-centric socio-cultural biases. INCLUDE consists of 2,604 prompts spanning six prompt languages: English, Hindi, Bengali, Marathi, Tamil, and Hinglish (Hindi-English code-mix). We evaluate ten open- and closed-source LLMs against this benchmark, analyzing 14,988 bias scores. Our statistical results reveal two key findings. First, Bengali yielded the highest average bias score in open-source models. Second, English demonstrated a notable reversal, producing the lowest bias in open-source models but the highest bias in closed-source models.

Related

• TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages
• LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety
• SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
• IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia

Source: arXiv cs.AI | 2026-08-20