Finding Culture-Sensitive Neurons in Vision-Language Models

arXiv:2510.24942v2 Announce Type: replace-cross Abstract: Despite their impressive performance, vision-language models (VLMs) still struggle on culturally situated inputs. To understand how VLMs process culturally grounded information, we study the presence of culture-sensitive neurons, i.e., neurons whose activations show preferential sensitivity to inputs associated with particular cultural contexts. We examine whether such neurons are important for culturally diverse visual question answering and where they are located. Using the CVQA benchmark, we identify neurons of culture selectivity and perform diagnostic tests by deactivating the neurons flagged by various identification methods. Experiments on three VLMs across 25 cultural groups demonstrate the existence of neurons whose ablation disproportionately harms performance on questions about the corresponding cultures, while having limited effects on others. Moreover, we introduce a new margin-based selector Contrastive Activation Margin (ConAct) and show that it outperforms probability- and entropy-based methods in identifying neurons associated with cultural selectivity. Finally, our layer-wise analyses reveal that such neurons are not uniformly distributed: they cluster in specific decoder layers in a model-dependent way.

Related

• From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Models
• Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos
• Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
• More Than Meets the Eye: Measuring the Semiotic Gap in Vision-Language Models via Semantic Anchorage

Source: arXiv cs.CL | 2026-04-21