Search: “benchmark”

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

arXiv:2605.26418v1 Announce Type: cross Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every wor

27 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

AstroMind: A High-Fidelity Benchmark for Spacecraft Behavior Reasoning Based on Large Language Models

DGX agent

arXiv:2605.24573v1 Announce Type: new Abstract: Understanding why a spacecraft maneuvers -- rather than simply that it did -- is an increasingly important problem for space domain awareness as Earth o

model-releasesarxiv-cs-cl

26 May 2026

Benchmarking Pathology Foundation Models for Spatial Domain Understanding

arXiv:2605.25764v1 Announce Type: cross Abstract: Pathology foundation models (PFMs) have emerged as a core approach for learning transferable representations from whole slide images (WSIs), and they

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking

arXiv:2605.26087v1 Announce Type: cross Abstract: Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of establis

GlobalDentBench: A Multinational Benchmark for Evaluating LLM Clinical Reasoning in Dentistry with Expert Calibration

arXiv:2605.24636v1 Announce Type: new Abstract: While large language models (LLMs) hold transformative potential for medicine, their reasoning robustness and safety in real-world clinical scenarios re

Safety

Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks

arXiv:2605.24217v1 Announce Type: new Abstract: As Large Language Models (LLMs) transition from research environments to production deployments, evaluating their performance against strict Service Lev

safetyarxiv-cs-ai

Omissive Bias in Religious Representation: Benchmarking LLM Answers to Everyday Ethical Decision-making

arXiv:2605.24319v1 Announce Type: new Abstract: As large language models become a default source of guidance on personal, moral, and existential questions, it matters whether they draw on the religiou

RAW: Robust Avatar Watermarking -- Benchmarking and Baseline

arXiv:2605.23994v1 Announce Type: cross Abstract: Digital avatar watermarking presents unique challenges: avatars are routinely post-processed with background replacement, reframing, and format conver

Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models

arXiv:2605.25601v1 Announce Type: cross Abstract: Teacher education requires deliberate practice with learners who exhibit identifiable strengths, weaknesses, and partial mastery. Large language model

When the Manual Lies: A Realistic Benchmark to Evaluate MCP Poisoning Attacks for LLM Agents

arXiv:2605.24069v1 Announce Type: cross Abstract: The rise of tool-using Large Language Model (LLM) agents, standardized by protocols like the Model Context Protocol (MCP), has unlocked unprecedented

CaST-Bench: Benchmarking Causal Chain-Grounded Spatio-Temporal Reasoning for Video Question Answering

arXiv:2605.23216v1 Announce Type: new Abstract: Cause-and-effect reasoning in video is a significant challenge for Vision-Language Models (VLMs), as it requires going beyond surface-level perception t

CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models

arXiv:2605.23699v1 Announce Type: new Abstract: Video prediction is increasingly viewed as a path toward generalizable world models, yet it remains unclear whether these systems learn underlying causa

DDX-TRACE: A Benchmark for Medical Diagnostic Trajectories in VLMs

arXiv:2605.23629v1 Announce Type: new Abstract: Medical diagnosis is not a single prediction from a fully specified vignette. It is a sequential workup: clinicians decide what evidence to obtain, revi

MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarks

arXiv:2601.14652v5 Announce Type: replace Abstract: While multi-agent systems (MAS) promise elevated intelligence through coordination of agents, current approaches to automatic MAS design under-deliv

PoisonForge: Task-Level Targeted Poisoning Benchmark for Instruction-Tuned LLMs

arXiv:2605.23168v1 Announce Type: cross Abstract: When practitioners fine-tune LLMs on unvetted datasets, an adversary can exploit the data supply chain through task-level poisoning: inserting a small

ProtDBench: A Unified Benchmark of Protein Binder Design and Evaluation

arXiv:2605.04118v2 Announce Type: replace-cross Abstract: Recent advances in de novo protein binder design have enabled increasing experimental validation, yet reported in silico metrics remain diffic

RoboSurg-VQA: A Multimodal Benchmark for Surgical Segmentation-Aware Visual Question Answering

arXiv:2605.23068v1 Announce Type: new Abstract: Reliable visual understanding in robot-assisted and minimally invasive surgery (RMIS/MIS) demands more than accurate masks: in clinical practice, clinic

VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

arXiv:2605.22907v1 Announce Type: new Abstract: Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal s

Cross-domain benchmarks reveal when coordinated AI agents improve scientific inference from partial evidence

arXiv:2605.22300v1 Announce Type: cross Abstract: Scientific evidence often spans instruments, databases, and disciplines, so no single source records the full phenomenon. This makes it difficult to d

23 May 2026

Beyond Benchmark Islands: Toward Representative Trustworthiness Evaluation for Agentic AI

arXiv:2603.14987v2 Announce Type: replace Abstract: Agentic AI systems increasingly act through tool-augmented, multi-step workflows whose failures (unsafe tool use, unauthorised actions, social harm)

Do Vision Models Encode Object-Level Semantic Relatedness? A Cognitive Psychology-Inspired Benchmark

arXiv:1709.03806v2 Announce Type: replace Abstract: Modern vision models have achieved strong object-recognition performance, yet it remains unclear whether their representations encode object-level s

From Recognition to Reasoning: Benchmarking and Enhancing MLLMs on Real-World Receipt Document Understanding

arXiv:2605.22413v1 Announce Type: new Abstract: Extracting structured information from visual documents (Visual Information Extraction, VIE) is a cornerstone of business automation. While recent Multi

GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety

arXiv:2605.20203v1 Announce Type: cross Abstract: As older adults increasingly use LLM-based chatbots for companionship and assistance, a safety gap is emerging. Older adults may face vulnerabilities

MM-Conv: A Multimodal Dataset and Benchmark for Context-Aware Grounding in 3D Dialogue

arXiv:2605.21796v1 Announce Type: cross Abstract: Grounding language in the physical world requires AI systems to interpret references that emerge dynamically during conversation. While current vision

RAGCap-Bench: Benchmarking Capabilities of LLMs in Agentic Retrieval Augmented Generation Systems

arXiv:2510.13910v2 Announce Type: replace Abstract: Retrieval-Augmented Generation (RAG) mitigates key limitations of Large Language Models (LLMs)-such as factual errors, outdated knowledge, and hallu

RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

arXiv:2605.21748v1 Announce Type: new Abstract: As interactive LLM-based applications are created and refined, model developers need to evaluate the quality of generated text along many possible axes.

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

arXiv:2605.20204v1 Announce Type: cross Abstract: LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstraine

TransitLM: A Large-Scale Dataset and Benchmark for Map-Free Transit Route Generation

arXiv:2605.22355v1 Announce Type: new Abstract: Public transit route planning traditionally depends on structured map infrastructure and complex routing engines, and no existing dataset supports train

VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

arXiv:2605.22570v1 Announce Type: new Abstract: Spatio-temporal reasoning is a core capability for Multimodal Large Language Models (MLLMs) operating in the real world. As such, evaluating it precisel

The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents

arXiv:2605.20544v1 Announce Type: cross Abstract: Vision-language models (VLMs) are used as high-level planners for embodied agents, translating natural language instructions and visual observations i

21 May 2026

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

arXiv:2605.20676v1 Announce Type: new Abstract: Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimoda

21 May 2026

Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth

arXiv:2605.19797v1 Announce Type: new Abstract: Monocular depth estimation has improved significantly in recent years, driven by increasingly powerful models and large-scale training data. Predicted d

EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data

arXiv:2605.19130v1 Announce Type: cross Abstract: Children acquire language grounding with remarkable robustness from limited visuo-linguistic input in ways that surpass today's best large multimodal

FineBench: Benchmarking and Enhancing Vision-Language Models for Fine-grained Human Activity Understanding

arXiv:2605.19846v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities in general video understanding, yet they often struggle with the fine-grained

Quantifying the Generalization Gap in Seizure Detection: A Large-Scale Empirical Benchmark via the SzCORE Challenge

arXiv:2505.18191v2 Announce Type: replace-cross Abstract: Reliable automatic seizure detection from long-term electroencephalography (EEG) remains an unsolved challenge, as current models often fail t

A Pilot Benchmark for NL-to-FOL Translation in Planetary Exploration

arXiv:2605.17911v1 Announce Type: new Abstract: Future planetary exploration envisions autonomous robotic agents operating under severe communication constraints, without global positioning, and with

BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting

arXiv:2605.17937v1 Announce Type: cross Abstract: Quantitative backtesting is essential for evaluating trading strategies but remains hampered by high technical barriers and limited scalability. While

Benchmarking Recurrent Event-Based Object Detection for Industrial Multi-Class Recognition on MTevent

arXiv:2603.21787v2 Announce Type: replace Abstract: Event cameras are attractive for industrial robotics because they provide high temporal resolution, high dynamic range, and reduced motion blur. How

ClawArena: Benchmarking AI Agents in Evolving Information Environments

arXiv:2604.04202v2 Announce Type: replace-cross Abstract: AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is s

CT-DegradBench: A Physics-Informed Benchmark for CT Degradation Detection and Severity Estimation

arXiv:2605.16431v1 Announce Type: new Abstract: Computed tomography (CT) images are frequently degraded by acquisition artifacts, including noise, blur, streaking, aliasing, and metal artifacts. Yet C

Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memory

arXiv:2511.20857v2 Announce Type: replace-cross Abstract: Statefulness is essential for large language model (LLM) agents to perform long-term planning and problem-solving. This makes memory a critica

From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models

arXiv:2508.01608v2 Announce Type: replace Abstract: Image geolocalization, the task of identifying the geographic location depicted in an image, is important for applications in crisis response, digit

GenTS: A Comprehensive Benchmark Library for Generative Time Series Models

arXiv:2605.17804v1 Announce Type: new Abstract: Generative models have demonstrated remarkable potential in time series analysis tasks, like synthesis, forecasting, imputation, etc. However, offering

How Good LLMs Are at Answering Bangla Medical Visual Questions? Dataset and Benchmarking

arXiv:2605.18111v1 Announce Type: new Abstract: Recent advancements in Large Language Models (LLMs) and Large Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promisi

MentalBench: A DSM-Grounded Benchmark for Evaluating Psychiatric Diagnostic Capability of Large Language Models

arXiv:2602.12871v2 Announce Type: replace Abstract: Large language models (LLMs) have attracted growing interest as supportive tools for psychiatric assessment and clinical decision support. However,

MirrorBench: A Benchmark to Evaluate Conversational User-Proxy Agents for Human-Likeness

arXiv:2601.08118v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as human simulators, both for evaluating conversational systems and for generating fine-tuning da

MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

arXiv:2605.16616v1 Announce Type: new Abstract: Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks

OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding

arXiv:2605.18577v1 Announce Type: new Abstract: Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emer