Search: “benchmark”

MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models

arXiv:2604.19809v1 Announce Type: new Abstract: We introduce MIRROR, a benchmark comprising eight experiments across four metacognitive levels that evaluates whether large language models can use self

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

DGX agent

arXiv:2604.20738v1 Announce Type: new Abstract: We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer

model-releasesarxiv-cs-cl

23 Apr 2026

The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

arXiv:2604.20225v1 Announce Type: new Abstract: Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current bench

Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

arXiv:2603.23146v2 Announce Type: replace-cross Abstract: The widespread adoption of Large Language Models (LLMs) has made the detection of AI-Generated text a pressing and complex challenge. Although

X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis

arXiv:2604.20350v1 Announce Type: new Abstract: Despite significant progress in Multi-modal Large Language Models (MLLMs), their clinical reasoning capacity for multi-modal diagnosis remains largely u

DW-Bench: Benchmarking LLMs on Data Warehouse Graph Topology Reasoning

arXiv:2604.18964v1 Announce Type: new Abstract: This paper introduces DW-Bench, a new benchmark that evaluates large language models (LLMs) on graph-topology reasoning over data warehouse schemas, exp

PC2Model: ISPRS benchmark on 3D point cloud to model registration

arXiv:2604.19596v1 Announce Type: new Abstract: Point cloud registration involves aligning one point cloud with another or with a three-dimensional (3D) model, enabling the integration of multimodal d

Safety

Personalized Benchmarking: Evaluating LLMs by Individual Preferences

arXiv:2604.18943v1 Announce Type: new Abstract: With the rise in capabilities of large language models (LLMs) and their deployment in real-world tasks, evaluating LLM alignment with human preferences

safetyarxiv-cs-ai

Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images

arXiv:2509.07966v2 Announce Type: replace-cross Abstract: Visual reasoning over structured data such as tables is a critical capability for modern vision-language models (VLMs), yet current benchmarks

Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India

arXiv:2604.19151v1 Announce Type: new Abstract: Existing Indic ASR benchmarks often use scripted, clean speech and leaderboard driven evaluation that encourages dataset specific overfitting. In additi

ArgBench: Benchmarking LLMs on Computational Argumentation Tasks

arXiv:2604.17366v1 Announce Type: new Abstract: Argumentation skills are an essential toolkit for large language models (LLMs). These skills are crucial in various use cases, including self-reflection

BengaliMoralBench: A Benchmark for Auditing Moral Reasoning in Large Language Models within Bengali Language and Culture

arXiv:2511.03180v2 Announce Type: replace Abstract: As multilingual Large Language Models (LLMs) gain traction across South Asia, their alignment with local ethical norms, particularly for Bengali, sp

Research

Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation

arXiv:2604.17020v1 Announce Type: new Abstract: Static benchmarks for harmful content detection face limitations in scalability and diversity, and may also be affected by contamination from web-scale

researcharxiv-cs-cl

Domain-oriented RAG Assessment (DoRA): Synthetic Benchmarking for RAG-based Question Answering on Defense Documents

arXiv:2604.17943v1 Announce Type: new Abstract: Open-domain RAG benchmarks over public corpora can overestimate deployment performance due to pretraining overlap and weak attribution requirements. We

IDOBE: Infectious Disease Outbreak forecasting Benchmark Ecosystem

arXiv:2604.18521v1 Announce Type: new Abstract: Epidemic forecasting has become an integral part of real-time infectious disease outbreak response. While collaborative ensembles composed of statistica

LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection

arXiv:2604.04815v2 Announce Type: replace Abstract: The rapid development of Large Language Models (LLMs) has transformed fake news detection and fact-checking tasks from simple classification to comp

LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

arXiv:2601.16504v3 Announce Type: replace Abstract: Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks

Macaron: Controlled, Human-Written Benchmark for Multilingual and Multicultural Reasoning via Template-Filling

arXiv:2602.10732v2 Announce Type: replace Abstract: Multilingual benchmarks rarely test reasoning over culturally grounded premises: translated datasets keep English-centric scenarios, while culture-f

MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

arXiv:2601.03331v2 Announce Type: replace Abstract: Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models tru

NL2SQLBench: A Modular Benchmarking Framework for LLM-Enabled NL2SQL Solutions

arXiv:2604.16493v1 Announce Type: cross Abstract: Natural Language to SQL (NL2SQL) technology empowers non-expert users to query relational databases without requiring SQL expertise. While large langu

Pulse Shape Discrimination Algorithms: Survey and Benchmark

arXiv:2508.02750v2 Announce Type: replace Abstract: This review presents a comprehensive survey and benchmark of pulse shape discrimination (PSD) algorithms for radiation detection, classifying nearly

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

arXiv:2602.14466v2 Announce Type: replace Abstract: With natural language generation becoming a popular use case for language models, the Bias Benchmark for Question-Answering (BBQ) has grown to be an

Sky2Ground: A Benchmark for Site Modeling under Varying Altitude

arXiv:2603.13740v3 Announce Type: replace Abstract: We introduce Sky2Ground, a three-view dataset designed for varying altitude camera localization, correspondence learning, and reconstruction. The da

ToxiFrench: Benchmarking and Enhancing Language Models via CoT Fine-Tuning for French Toxicity Detection

arXiv:2508.11281v3 Announce Type: replace Abstract: Detecting toxic content using language models is crucial yet challenging. While substantial progress has been made in English, toxicity detection in

TransXion: A High-Fidelity Graph Benchmark for Realistic Anti-Money Laundering

arXiv:2604.17420v1 Announce Type: new Abstract: Money laundering poses severe risks to global financial systems, driving the widespread adoption of machine learning for transaction monitoring. However

Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmark

arXiv:2510.13759v3 Announce Type: replace Abstract: Unified multimodal models aim to jointly enable visual understanding and generation, yet current benchmarks rarely examine their true integration. E

DPrivBench: Benchmarking LLMs' Reasoning for Differential Privacy

arXiv:2604.15851v1 Announce Type: cross Abstract: Differential privacy (DP) has a wide range of applications for protecting data privacy, but designing and verifying DP algorithms requires expert-leve

FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound

arXiv:2512.22278v2 Announce Type: replace Abstract: The growing demand for prenatal ultrasound imaging has intensified a global shortage of trained sonographers, creating barriers to essential fetal h

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs

arXiv:2604.16054v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have achieved impressive progress on vision language benchmarks, yet their capacity for visual cognitive and

neuralCAD-Edit: An Expert Benchmark for Multimodal-Instructed 3D CAD Model Editing

arXiv:2604.16170v1 Announce Type: new Abstract: We introduce neuralCAD-Edit, the first benchmark for editing 3D CAD models collected from expert CAD engineers. Instead of text conditioning as in prior

Towards Realistic Open-Vocabulary Remote Sensing Segmentation: Benchmark and Baseline

arXiv:2604.15652v1 Announce Type: new Abstract: Open-vocabulary remote sensing image segmentation (OVRSIS) remains underexplored due to fragmented datasets, limited training diversity, and the lack of

VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models

arXiv:2512.14554v5 Announce Type: replace-cross Abstract: The rapid advancement of large language models (LLMs) has enabled new possibilities for applying artificial intelligence within the legal doma

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

arXiv:2603.04738v2 Announce Type: replace Abstract: Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback f

17 Apr 2026

MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models

arXiv:2505.20122v2 Announce Type: replace Abstract: This paper introduces MEBench, a novel benchmark for evaluating mutual exclusivity (ME) bias, a cognitive phenomenon observed in children during wor

17 Apr 2026

DF3DV-1K: A Large-Scale Dataset and Benchmark for Distractor-Free Novel View Synthesis

arXiv:2604.13416v1 Announce Type: new Abstract: Advances in radiance fields have enabled photorealistic novel view synthesis. In several domains, large-scale real-world datasets have been developed to

16 Apr 2026

FieldWorkArena: Agentic AI Benchmark for Real Field Work Tasks

arXiv:2505.19662v3 Announce Type: replace-cross Abstract: This paper introduces FieldWorkArena, a benchmark for agentic AI targeting real-world field work. With the recent increase in demand for agent

16 Apr 2026

RiskWebWorld: A Realistic Interactive Benchmark for GUI Agents in E-commerce Risk Management

arXiv:2604.13531v1 Announce Type: cross Abstract: Graphical User Interface (GUI) agents show strong capabilities for automating web tasks, but existing interactive benchmarks primarily target benign,

16 Apr 2026

A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents

arXiv:2512.20798v4 Announce Type: replace Abstract: As autonomous AI agents are deployed in high-stakes environments, ensuring their safety has become a paramount concern. Existing safety benchmarks p

15 Apr 2026

Do VLMs Truly 'Read' Candlesticks? A Multi-Scale Benchmark for Visual Stock Price Forecasting

arXiv:2604.12659v1 Announce Type: cross Abstract: Vision-language models(VLMs) are increasingly applied to visual stock price forecasting, yet existing benchmarks inadequately evaluate their understan

15 Apr 2026

Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

arXiv:2604.12744v1 Announce Type: new Abstract: While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most lang

15 Apr 2026

ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks

arXiv:2604.10981v1 Announce Type: new Abstract: ATANT v1.0 (arXiv:2604.06710) defined continuity as a system property with 7 required properties and introduced a 10-checkpoint, LLM-free evaluation met

BlasBench: An Open Benchmark for Irish Speech Recognition

arXiv:2604.10736v1 Announce Type: new Abstract: No open Irish-specific benchmark compares end-user ASR systems under a shared Irish-aware evaluation protocol. To solve this, we release BlasBench, an o

EmbodiedGovBench: A Benchmark for Governance, Recovery, and Upgrade Safety in Embodied Agent Systems

arXiv:2604.11174v1 Announce Type: cross Abstract: Recent progress in embodied AI has produced a growing ecosystem of robot policies, foundation models, and modular runtimes. However, current evaluatio

From UAV Imagery to Agronomic Reasoning: A Multimodal LLM Benchmark for Plant Phenotyping

arXiv:2604.09907v1 Announce Type: cross Abstract: To improve crop genetics, high-throughput, effective and comprehensive phenotyping is a critical prerequisite. While such tasks were traditionally per

Safety

MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion

arXiv:2604.09587v1 Announce Type: new Abstract: Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as Androi

safetyarxiv-cs-ai

Switch-JustDance: Benchmarking Whole Body Motion Tracking Controllers Using a Commercial Console Game

arXiv:2511.17925v3 Announce Type: replace-cross Abstract: Recent advances in whole-body robot control have enabled humanoid and legged robots to perform increasingly agile and coordinated motions. How

YUV20K: A Complexity-Driven Benchmark and Trajectory-Aware Alignment Model for Video Camouflaged Object Detection

arXiv:2604.09985v1 Announce Type: new Abstract: Video Camouflaged Object Detection (VCOD) is currently constrained by the scarcity of challenging benchmarks and the limited robustness of models agains

HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?

arXiv:2604.09408v1 Announce Type: new Abstract: Frontier coding agents solve complex tasks when given complete context but collapse when specifications are incomplete or ambiguous. The bottleneck is n