PARSA-Bench: A Comprehensive Persian Audio-Language Model Benchmark

arXiv:2603.14456v2 Announce Type: replace Abstract: Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching, none of which is captured by existing benchmarks. We introduce extbf{PARSA-Bench} (extbf{P}ersian extbf{A}udio extbf{R}easoning and extbf{S}peech extbf{A}ssessment Benchmark), the first dedicated benchmark for evaluating LALMs on Persian language and culture. It covers 16 tasks, ten of them new, spanning speech understanding, paralinguistic analysis, and culturally grounded audio reasoning. Across most tasks, text-only baselines outperform their audio counterparts, so audio understanding rather than language knowledge remains the main limitation, and supplying the transcript alongside the audio lifts weak models to near their text-only level. The consistent exception is Persian poetry, where prosody carries information the written form cannot: audio beats text on both poetry tasks, and metre detection shows the first signs of being learnable only at the largest model scale. The dataset is publicly available at: https://huggingface.co/datasets/MohammadJRanjbar/PARSA-Bench

Related

• VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents
• Language-Switching Triggers Take a Latent Detour Through Language Models
• AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering
• Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

Source: arXiv cs.CL | 2026-09-16