Evaluating Geographic Lexical Knowledge in Large Language Models Across 21 Spanish-Speaking Countries

arXiv:2602.09346v2 Announce Type: replace Abstract: Large language models (LLMs) increasingly serve users across regional varieties, yet their knowledge of geographic lexical variation remains insufficiently evaluated. We investigate country-conditioned lexical usage judgments in Spanish using an expert-curated database covering 934 lexical items across 21 Spanish-speaking countries. Treating ten proprietary and open-weight LLMs as virtual informants, we evaluate two complementary question formats: Yes-No question format (YNQF) and multiple-choice question format (MCQF). The evaluation comprises 10,164 YNQF questions and 16,637 MCQF questions, assessed using F1 and an adjusted Jaccard coefficient, respectively. Results reveal substantial differences across models, countries, and formats. Spain receives the highest country-level score for eight models in YNQF and nine in MCQF; Chile receives the lowest MCQF score for every model. Regression analyses suggest that digital resource volume is generally more strongly associated with YNQF performance, whereas lexical distance from Spain is more strongly associated with MCQF performance. These findings provide a fine-grained diagnostic of uneven regional lexical knowledge and highlight the importance of complementary elicitation formats and evaluation metrics. The benchmark measures explicit usage judgments, motivating separate assessment of region-appropriate generation.

Related

• Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models
• DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledge
• Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures
• The Language You Ask In: Language-Conditioned Ideological Divergence in LLM Analysis of Contested Political Documents

Source: arXiv cs.CL | 2026-10-28