Search: “benchmark”

arXiv:2607.05545v1 Announce Type: cross Abstract: LLM conformity is often used to describe cases where a model changes a correct answer toward a peer or group response. We show that most of this appar

researcharxiv-cs-ai

8 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

DGX agent

arXiv:2607.06196v1 Announce Type: new Abstract: Current AI safety evaluation and benchmarking frameworks predominantly rely on Western-centric culture-agnostic defaults that mask critical regional law

model-releasesarxiv-cs-cl

8 Jul 2026

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

arXiv:2607.06254v1 Announce Type: cross Abstract: Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and

8 Jul 2026

WebRetriever: A Large-Scale Comprehensive Benchmark for Efficient Web Agent Evaluation

arXiv:2607.06118v1 Announce Type: new Abstract: As web agents increasingly demonstrate capabilities in automated task execution, the development of robust evaluation frameworks for assessing their nav

8 Jul 2026

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

model-releasesarxiv-cs-lg

arXiv:2607.02586v1 Announce Type: new Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common fo

Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions

arXiv:2607.04072v1 Announce Type: cross Abstract: Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (S

Industrial3D: A Water-Treatment TLS Point Cloud Dataset and Cross-Paradigm Benchmark for MEP Scene Understanding

arXiv:2603.28660v2 Announce Type: replace Abstract: Automated semantic understanding of dense terrestrial laser scanning (TLS) point clouds is a prerequisite for Scan-to-BIM, digital twin maintenance,

JavaVulBench: A Java Vulnerability Benchmark with Realistic Splits, a Unified Multi-Backend Harness, and a Leakage-Aware Evaluation Mode

arXiv:2607.02825v1 Announce Type: cross Abstract: We release extsc{JavaVulBench}, a benchmark dataset and evaluation harness for Java vulnerability detection. The dataset contains sim30{,}600 Java met

MatPhaseBench: A Semantics-Guided Benchmark for Materials Phase Diagrams Understanding

arXiv:2607.02934v1 Announce Type: cross Abstract: Materials phase diagrams are a core knowledge representation in materials science, encoding temperature,composition, phase stability, and phase transf

Moonstone: A Multimodal Foundation Model and Benchmark for Lunar Remote Sensing

arXiv:2607.03644v1 Announce Type: cross Abstract: Decades of orbital missions have produced multi-modal remote sensing data for the Moon, spanning optical imagery, spectroscopy, thermal emission, rada

SoftVTBench: A Safety-Aware Visuo-Tactile Benchmark for Physically Constrained Robotic Manipulation of Deformable Objects

arXiv:2607.04234v1 Announce Type: cross Abstract: Deformable object manipulation poses challenges beyond task completion: successful execution must also maintain safe physical interaction, holding the

Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework

arXiv:2606.19073v2 Announce Type: replace Abstract: Current image editing methods excel at static attributes but fail at complex Human-Object Interactions (HOI), a critical challenge unaddressed by ex

The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

arXiv:2607.04401v1 Announce Type: new Abstract: How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation mod

TiROD: Tiny Robotics Dataset and Benchmark for Continual Object Detection

arXiv:2409.16215v4 Announce Type: replace-cross Abstract: Detecting objects with visual sensors is crucial for numerous mobile robotics applications, from autonomous navigation to inspection. However,

Safety

Towards Data-Driven Metrics for Social Robot Navigation Benchmarking

arXiv:2509.01251v3 Announce Type: replace Abstract: This paper presents a joint effort towards the development of a data-driven Social Robot Navigation metric to facilitate benchmarking and policy opt

safetyarxiv-cs-ro

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

arXiv:2607.02269v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated immense promise in Spatio-Temporal Video Grounding (STVG). However, current evaluation protocols are l

MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation

arXiv:2508.16674v2 Announce Type: replace-cross Abstract: Medical report understanding from real-world document images is essential for generating patient-facing explanations and enabling structured i

Office Comprehension Benchmark

arXiv:2607.01245v1 Announce Type: cross Abstract: We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension

Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge

arXiv:2607.01829v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly proposed for aviation business operations, from documentation and training generation to customer facing a

PreScience: A Dataset and Benchmark for Scientific Forecasting

arXiv:2602.20459v2 Announce Type: replace Abstract: Can AI systems trained on the existing scientific record forecast the advances that will follow? We introduce PreScience, a dataset and benchmark fo

Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity

arXiv:2607.01153v1 Announce Type: cross Abstract: Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model has followed an in

RoadBench: Benchmarking MLLMs on Fine-Grained Spatial Understanding and Reasoning under Urban Road Scenarios

arXiv:2511.18011v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have demonstrated powerful capabilities in general spatial understanding and reasoning. However, their fine

SocialOmni: Benchmarking Audio-Visual Social Interactivity in Omni Models

arXiv:2603.16859v2 Announce Type: replace Abstract: Omni-modal large language models (OLMs) redefine human-machine interaction by natively integrating audio, vision, and text. However, existing OLM be

YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese

model-releasesarxiv-cs-cl

arXiv:2607.00664v1 Announce Type: new Abstract: We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese

Clinically Structured Rank-Gated LoRA for Cross-Benchmark Medical Question Answering

model-releasesarxiv-cs-cl

arXiv:2606.31432v1 Announce Type: new Abstract: Medical multiple-choice question answering requires parameter-efficient adaptation across heterogeneous knowledge domains and reasoning operations. A me

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

arXiv:2606.31179v1 Announce Type: new Abstract: As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward

Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

arXiv:2606.30790v1 Announce Type: cross Abstract: Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of

JL1-CC&QA: Extending the JL1-CD Benchmark with Change Captioning and Question Answering

arXiv:2606.31745v1 Announce Type: cross Abstract: Remote sensing change detection (CD) traditionally focuses on pixel-level binary segmentation, which identifies where changes occur but neither what n

RigorBench: Benchmarking Engineering Process Discipline in Autonomous AI Coding Agents

arXiv:2606.22678v2 Announce Type: replace-cross Abstract: Agentic coding harnesses - such as Agent-Skills, Superpowers, and Agent-Rigor - are increasingly deployed to augment underlying LLMs for real-

Can LLMs Prove Robotic Path Planning Optimality? A Benchmark for Research-Level Algorithm Verification

model-releasesarxiv-cs-ro

arXiv:2603.19464v2 Announce Type: replace Abstract: Robotic path planning problems are often NP-hard, and practical solutions typically rely on approximation algorithms with provable performance guara

Can OCR-VLMs Read Devanagari? A Stress-Test Benchmark and Post-Correction Study

model-releasesarxiv-cs-cl

arXiv:2606.29213v1 Announce Type: new Abstract: OCR systems, ranging from classical engines to specialised OCR vision-language models (OCR-VLMs) and frontier multimodal LLMs, report strong results on

CAREBench: A Child-Safety Risk Benchmark for Language Models

model-releasesarxiv-cs-lg

arXiv:2606.29685v1 Announce Type: new Abstract: How can we evaluate whether frontier AI systems recognize child-safety risks before they escalate into explicit harm? Existing child safety evaluations

CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

arXiv:2606.29771v1 Announce Type: new Abstract: LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading. Ye

Cross-Modal Iteration Distillation for Robust IHD Screening: The IDNet Framework and A New Benchmark

arXiv:2606.30027v1 Announce Type: new Abstract: Color Fundus Photography (CFP) offers a low-cost and non-invasive route for ischemic heart disease (IHD) screening, but current studies are limited by s

HEARTS: Benchmarking LLM Reasoning on Health Time Series

arXiv:2603.06638v3 Announce Type: replace-cross Abstract: The rise of large language models (LLMs) has shifted time series analysis from narrow analytics to general-purpose reasoning. Yet, existing be

IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations

arXiv:2606.28556v1 Announce Type: new Abstract: Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical app

MuseBench: Benchmarking Intent-Level Audiovisual Arts Understanding in MLLMs

arXiv:2606.30026v1 Announce Type: cross Abstract: Audiovisual arts encompass diverse creative disciplines, including cinema, visual arts, stage performance, and game design, where artistic meaning ari

SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing

arXiv:2606.29887v1 Announce Type: new Abstract: In real-world applications, guardrails are often expected to identify unsafe user-model interactions according to application-specific safety policies,

Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

arXiv:2601.17642v2 Announce Type: replace Abstract: Safety alignment in Large Language Models is critical for healthcare; however, reliance on binary refusal boundaries often results in over-refusal o

29 Jun 2026

Speculative Refinement: A Hybrid Autoregressive Diffusion Decoding Strategy and Its Behavior Across Benchmarks

arXiv:2606.27474v1 Announce Type: cross Abstract: How should we evaluate generation systems that combine autoregressive (AR) and diffusion decoding? We study this question through Speculative Refineme

29 Jun 2026

Video-MME-Logical: A Controlled Diagnostic Benchmark for Video Temporal-Logical Reasoning

arXiv:2606.27828v1 Announce Type: new Abstract: Recent interest in multimodal large language models (MLLMs) raises a central question: can they reason over dynamic visual evidence rather than merely r

29 Jun 2026

DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cues

arXiv:2606.26602v1 Announce Type: new Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive fine-grained perception capabilities. However, existing ben

MKG-RAG-Bench: Benchmarking Retrieval in Multimodal Knowledge Graph-Augmented Generation

arXiv:2606.26458v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) over knowledge graphs has emerged as a promising approach for grounding large language models, yet existing benchma

scBench-Long: Verifiable Benchmarking of Long-Horizon Single-Cell Biology

arXiv:2606.26563v1 Announce Type: cross Abstract: Single-cell studies require analysts to convert raw measurements into specific biological claims through multi-step workflows and integration of metad

Towards Video Anomaly Detection from Event Streams: A Baseline and Benchmark Datasets

arXiv:2603.24991v2 Announce Type: replace Abstract: Event-based vision, characterized by low redundancy, focus on dynamic motion, and inherent privacy-preserving properties, naturally fits the demands

AMVICC: A Novel Benchmark for Cross-Modal Failure Mode Profiling for VLMs and IGMs

arXiv:2601.17037v2 Announce Type: replace Abstract: We investigate visual reasoning limitations of both multimodal large language models (MLLMs) and image generation models (IGMs) by creating a novel

25 Jun 2026

Research

BCoughBench: Benchmarking Respiratory Acoustic Foundation Models Under Body-Coupled Wearable Sensor Conditions

arXiv:2606.25116v1 Announce Type: cross Abstract: Respiratory acoustic foundation models (FMs) are benchmarked exclusively on smartphone recordings, yet clinical deployment increasingly targets body-c

researcharxiv-cs-ai

25 Jun 2026

FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming

arXiv:2606.19887v2 Announce Type: replace-cross Abstract: Existing safety benchmarks target general adversarial scenarios but miss finance-specific risks. Financial LLMs face regulatory compliance vio