Search: “benchmark”

CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers

arXiv:2605.07905v1 Announce Type: cross Abstract: Despite the rapid development of AI reviewers, evaluating such systems remains challenging: metrics favor overlap with human reviews over correctness.

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

CrossCult-KIBench: A Benchmark for Cross-Cultural Knowledge Insertion in MLLMs

DGX agent

arXiv:2605.06115v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs), trained primarily on English-centric data, frequently generate culturally inappropriate or misaligned resp

model-releasesarxiv-cs-ai

11 May 2026

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks

arXiv:2605.07024v1 Announce Type: new Abstract: Large Language Models for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks -- plausible but incorrect completions suc

DKDS: A Benchmark Dataset of Degraded Kuzushiji Documents with Seals for Detection and Binarization

arXiv:2511.09117v4 Announce Type: replace Abstract: Kuzushiji, a pre-modern Japanese cursive script, can currently be read and understood by only a few thousand trained experts in Japan. With the rapi

DRIP-R: A Benchmark for Decision-Making and Reasoning Under Real-World Policy Ambiguity in the Retail Domain

arXiv:2605.07699v1 Announce Type: cross Abstract: LLM-based agents are increasingly deployed for routine but consequential tasks in real-world domains, where their behavior is governed by inherently a

EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams

arXiv:2605.07299v1 Announce Type: cross Abstract: Existing Multimodal Large Language Models (MLLMs) remain primarily reactive, failing to continuously perceive environments or proactively assist users

FinReasoning: A Hierarchical Benchmark for Reliable Financial Research Reporting

arXiv:2603.19254v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly deployed in financial research workflows, where their role is evolving from single-model assistance fo

IntentGrasp: A Comprehensive Benchmark for Intent Understanding

arXiv:2605.06832v1 Announce Type: cross Abstract: Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assis

LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

arXiv:2605.07640v1 Announce Type: cross Abstract: Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general lan

McNdroid: A Longitudinal Multimodal Benchmark for Robust Drift Detection in Android Malware

arXiv:2605.06894v1 Announce Type: cross Abstract: Machine learning (ML) in real-world systems must contend with concept drift, adversarial actors, and a spectrum of potential features with varying cos

SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Following

arXiv:2605.06353v2 Announce Type: replace Abstract: In a conversation, a helpful assistant must reliably follow user directives, even as they refine, modify, or contradict earlier requests. Yet most i

SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

arXiv:2605.07161v1 Announce Type: new Abstract: AI agents are increasingly used to diagnose and mitigate failures in production systems, known as agentic Site Reliability Engineering (SRE). Current SR

TEA-Bench: A Systematic Benchmarking of Tool-enhanced Emotional Support Dialogue Agent

arXiv:2601.18700v2 Announce Type: replace Abstract: Emotional Support Conversation requires not only affective expression but also grounded instrumental support to provide trustworthy guidance. Howeve

TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

arXiv:2601.18744v2 Announce Type: replace Abstract: Time series are ubiquitous in real-world scenarios and crucial for applications ranging from energy management to traffic control. Consequently, the

Utility-Preserving De-Identification for Math Tutoring: Investigating Numeric Ambiguity in the MathEd-PII Benchmark Dataset

arXiv:2602.16571v2 Announce Type: replace Abstract: Large-scale sharing of dialogue data is key to advancing the science of teaching and learning, yet rigorous de-identification remains a major barrie

Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios

arXiv:2605.03242v1 Announce Type: new Abstract: Tool-using agent systems powered by large language models (LLMs) are increasingly deployed across web, app, operating-system, and transactional environm

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

arXiv:2605.04313v1 Announce Type: new Abstract: Causal reasoning in natural language requires identifying relevant variables, understanding their interactions, and reasoning about effects and interven

OpenVTON-Bench: A Large-Scale High-Resolution Benchmark for Controllable Virtual Try-On Evaluation

arXiv:2601.22725v3 Announce Type: replace Abstract: Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remain

Safety

OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking

arXiv:2605.03762v1 Announce Type: new Abstract: Large language models are moving from static text generators toward real-world decision-support systems, where forecasting is a composite capability tha

safetyarxiv-cs-ai

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

arXiv:2605.03361v2 Announce Type: new Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organizati

UAV as Urban Construction Change Monitor: A New Benchmark and Change Captioning Model

arXiv:2605.04409v1 Announce Type: new Abstract: Remote Sensing Image Change Captioning (RSICC) aims to generate spatially grounded natural language descriptions of scene evolution from bi-temporal ima

A Benchmark for Interactive World Models with a Unified Action Generation Framework

arXiv:2605.03941v1 Announce Type: new Abstract: Achieving Artificial General Intelligence (AGI) requires agents that learn and interact adaptively, with interactive world models providing scalable env

Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks

arXiv:2605.03759v1 Announce Type: new Abstract: While Large Vision-Language Models (LVLMs) offer powerful capabilities, they pose privacy risks by unintentionally memorizing sensitive personal informa

Local Ai

Benchmarking Local Language Models for Social Robots using Edge Devices

arXiv:2605.03111v1 Announce Type: cross Abstract: Social-educational robots designed for socially interactive pedagogical support, such as the Robot Study Companion (RSC), rely on responsive, privacy-

local-aiarxiv-cs-cl

DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis

arXiv:2605.02503v1 Announce Type: new Abstract: Evaluating autonomous data analysis agents requires testing their ability to perform exploratory analysis in underexplored data environments. However, m

EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving

arXiv:2509.17677v2 Announce Type: replace Abstract: Large language models (LLMs) have shown strong performance on mathematical reasoning under well-defined conditions. However, real-world engineering

From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

arXiv:2601.03731v3 Announce Type: replace-cross Abstract: As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consiste

MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

arXiv:2602.00933v2 Announce Type: replace-cross Abstract: The Model Context Protocol (MCP) is rapidly becoming the standard interface for Large Language Models (LLMs) to discover and invoke external t

MHPR: Multidimensional Human Perception and Reasoning Benchmark for Large Vision-Languate Models

arXiv:2605.03485v1 Announce Type: new Abstract: Multidimensional human understanding is essential for real-world applications such as film analysis and virtual digital humans, yet current LVLM benchma

ReLeaf: Benchmarking Leaf Segmentation across Domains and Species

arXiv:2605.03784v1 Announce Type: new Abstract: Rising global food demand and growing climate pressure increase the need for sustainable, precise agricultural practices. Automated, individualized plan

Benchmarking Single-Pose Docking, Consensus Rescoring, and Supervised ML on the LIT-PCBA Library: A Critical Evaluation of DiffDock, AutoDock-GPU, GNINA, and DiffDock-NMDN

arXiv:2605.01681v1 Announce Type: new Abstract: Virtual screening performance depends heavily on the chosen docking and scoring methods. Recent AI-based tools such as DiffDock and NMDN have reported s

Differentiable Multiphysics Co-Optimization via Implicit Neural Representations: A Transient Hamburger-Cooking Benchmark

arXiv:2605.01040v1 Announce Type: cross Abstract: The co-optimization of geometry and physical parameters remains challenging in transient multiphysics systems involving moving boundaries, nonlinear m

Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather

arXiv:2605.01126v1 Announce Type: new Abstract: Forecasting the wide variety of high-impact weather events experienced globally is a challenge for both Artificial Intelligence (AI) and Numerical Weath

GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models

arXiv:2605.01203v1 Announce Type: cross Abstract: Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly genera

LUMINA: A Grid Foundation Model for Benchmarking AC Optimal Power Flow Surrogate Learning

arXiv:2605.02133v1 Announce Type: new Abstract: AC optimal power flow (ACOPF) is foundational yet computationally expensive in power grid operations, driving learning-based surrogates for large-scale

PepSpecBench: A Unified Evaluation Benchmark for Peptide Tandem Mass Spectrometry Prediction

arXiv:2605.01945v1 Announce Type: new Abstract: Tandem mass spectrometry provides a high-throughput framework for identifying and quantifying proteins in complex biological samples. In computational p

RMGAP: Benchmarking the Generalization of Reward Models across Diverse Preferences

arXiv:2605.01831v1 Announce Type: new Abstract: Reinforcement Learning from Human Feedback has become the standard paradigm for language model alignment, where reward models directly determine alignme

Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark

arXiv:2603.27259v2 Announce Type: replace Abstract: Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable prog

SurGE: A Benchmark and Evaluation Framework for Scientific Survey Generation

arXiv:2508.15658v5 Announce Type: replace Abstract: The rapid growth of academic literature makes the manual creation of scientific surveys increasingly infeasible. While large language models show pr

Towards High Fidelity Face Swapping: A Comprehensive Survey and New Benchmark

arXiv:2605.00883v1 Announce Type: new Abstract: Face swapping has witnessed significant progress in recent years, largely driven by advances in deep generative models such as GANs and diffusion models

BanglaSocialBench: A Benchmark for Evaluating Sociopragmatic and Cultural Alignment of LLMs in Bangladeshi Social Interaction

arXiv:2603.15949v3 Announce Type: replace Abstract: Large Language Models have demonstrated strong multilingual fluency, yet fluency alone does not guarantee socially appropriate language use. In high

ControBench: An Interaction-Aware Benchmark for Controversial Discourse Analysis on Social Networks

arXiv:2605.00513v1 Announce Type: new Abstract: Understanding how people argue across ideological divides online is important for studying political polarization, misinformation, and content moderatio

Research

Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues

arXiv:2605.00119v1 Announce Type: new Abstract: There is a significant gap in evaluating cultural reasoning in LLMs using conversational datasets that capture culturally rich and dialectal contexts. M

researcharxiv-cs-cl

Foresight Arena: An On-Chain Benchmark for Evaluating AI Forecasting Agents

arXiv:2605.00420v1 Announce Type: cross Abstract: Evaluating the true forecasting ability of AI agents requires environments resistant to overfitting, free from centralized trust, and grounded in ince

AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR

arXiv:2604.27543v1 Announce Type: new Abstract: Evaluating English ASR systems for conversational AI applications remains difficult, as many publicly available corpora are either pre-segmented into sh

Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark

arXiv:2604.27582v1 Announce Type: new Abstract: Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate asse

Do Papers Tell the Whole Story? A Benchmark and Framework for Uncovering Hidden Implementation Gaps in Bioinformatics

arXiv:2603.22018v2 Announce Type: replace Abstract: Ensuring consistency between research papers and their corresponding software code implementations is a fundamental prerequisite for guaranteeing th

Research

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

arXiv:2604.27453v1 Announce Type: new Abstract: Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, exi

researcharxiv-cs-cl