Where Do Multilingual Vision-Language Encoders Fail on Low-Resource Languages?

arXiv:2608.30725v1 Announce Type: new Abstract: Recent multilingual vision--language encoders cover hundreds of languages in a single model, yet on two state-of-the-art instances retrieval on low-resource languages (LRL; e.g. Swahili) trails high-resource ones (HRL; e.g. English) by 30^+,pp. We ask where in the trained encoder this gap is located. Prior modality-gap and cross-lingual subspace work suggests a linear language direction at the output crowds out alignment-relevant geometry. We falsify this: LEACE drives the linear language classifier from >99% to near chance and iterated INLP to 37--50% while LRL retrieval moves within pm 1.5,pp and all tier means within 2.2,pp, tracking random controls. The linear bias is a symptom, not the cause. Instead, the alignment-causal factor lies along the encoder's forward path: the EOS (end-of-sequence) hidden state's per-language trajectory diverges with depth. Substituting the EOS with its parallel English value three blocks before the projector lifts Swahili from 22.1% to 69.1% on one encoder (and reproduces on the other); three controls rule out pooled-position tautology and English specificity. A front-layer trunk that pulls each language's projection toward the parallel-content centroid corroborates the diagnosis at training time, recovering +9.6 / +17.1,pp on LRL XM3600 retrieval (1{,}000-image subset), with consistent gains across three further benchmarks while preserving HRL performance.

Related

• Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages
• Hierarchical Pre-Training of Vision Encoders with Large Language Model
• Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family

Source: arXiv cs.CL | 2026-09-01