Search: “evaluation”

Are Benchmarks Reliable? Toward Structural Diagnosis via Item-Level Capability Boundaries

arXiv:2609.33196v2 Announce Type: replace Abstract: Evaluating large language models (LLMs) relies heavily on benchmark scores, yet aggregate metrics can obscure whether individual items serve their i

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

ASIWorld: Benchmarking Autonomous Agents in a Persistent Survival Sandbox

DGX agent

arXiv:2610.14807v1 Announce Type: new Abstract: Evaluating autonomous agents requires examining whether they can translate goals into coherent action across extended interactions. In persistent enviro

agentsarxiv-cs-ai

15 Oct 2026

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

model-releasesarxiv-cs-lg

arXiv:2610.20012v1 Announce Type: new Abstract: Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still

Agents

Choosing LLMs for Strategic Agent Interactions

arXiv:2610.18354v1 Announce Type: new Abstract: Game-theoretic benchmarks claim that large language model (LLM) strategic decision-making varies across game types, interaction structures, and capabili

agentsarxiv-cs-ai

ClinEdge: An Expert-Graded Benchmark of Real Patient Edge Cases Where Frontier Multimodal Models Fail

arXiv:2610.15239v1 Announce Type: new Abstract: Most benchmarks for large language models in medicine are built from licensing examinations and theoretical questions. The few that draw on real clinica

DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

arXiv:2610.19118v1 Announce Type: new Abstract: Emerging decision models such as JEV provide typed interfaces that map task instructions and observations directly to choices, judgments, and scores. As

DuplexAct: How Interactive Are Frontier Full-Duplex Spoken Language Models?

arXiv:2610.17738v1 Announce Type: new Abstract: End-to-end (E2E) full-duplex spoken language models (FD-SLMs) enable simutanous listening and speaking continously, requiring models to take appropriate

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fin

Measuring Epistemic Humility in Multimodal Large Language Models

arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant risk

On Symmetry for Characterizing Agent Heterogeneity in Multi-Agent Systems

arXiv:2610.18538v1 Announce Type: cross Abstract: Understanding agent heterogeneity can inform the design of shared and agent-specific policy representations, however existing criteria provide no unif

safetyarxiv-cs-ai

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

arXiv:2610.20548v1 Announce Type: new Abstract: Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and mult

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

arXiv:2610.16412v1 Announce Type: new Abstract: As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizi

Task-evoked eye movements as candidate digital biomarkers of depression severity in adolescents and young adults

arXiv:2610.18494v1 Announce Type: cross Abstract: Objective digital measures of depression severity could complement conventional symptom assessments and facilitate more timely clinical evaluation. He

safetyarxiv-cs-ai

UA-Bench: Benchmarking Large Audio-Language Models on Underwater Acoustic Recognition

arXiv:2610.16942v1 Announce Type: cross Abstract: Large audio-language models (LALMs) are fast becoming a general interface to sound, yet how far their broad pretraining carries into specialized domai

Uncovering Robustness Weaknesses in LLM Agents via Persona Evolution with Task-Integrity Constraints

arXiv:2610.16155v1 Announce Type: new Abstract: Interactive-agent evaluations commonly rely on standardized LLM-based user simulators that exhibit broadly cooperative behavior, potentially overlooking

VETNET: Assembling Benchmark Datasets for AI-Driven Veterinary Innovation

arXiv:2610.14149v1 Announce Type: new Abstract: VETNET represents an ecosystem of live benchmark datasets and evaluation frameworks designed to accelerate AI-driven innovation in animal science and ve

A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?

arXiv:2610.15562v1 Announce Type: new Abstract: Large-scale video generation models are increasingly viewed as a promising path toward world models, yet their ability to capture causally coherent phys

FENAS: A Few-Shot Meta-Ensembling Framework for Efficient Neural Architecture Search with Synergized Zero-Cost Proxies

arXiv:2610.15307v1 Announce Type: new Abstract: Efficient neural architecture search (NAS) requires accurate performance estimation under limited evaluation budgets. Existing zero-cost proxies (ZCPs)

safetyarxiv-cs-lg

Tutorials

GpFlash: Maximizing Speed and Minimizing Memory for Gaussian Process Training and Prediction

arXiv:2610.16861v1 Announce Type: new Abstract: This paper presents GpFlash, a Gaussian process library that is focused on fast training and posterior evaluations, minimal memory usage, and scalabilit

tutorialsarxiv-cs-lg

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

arXiv:2610.16634v1 Announce Type: cross Abstract: Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing close

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

arXiv:2610.18461v1 Announce Type: new Abstract: Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test

Local Ai

PRISM: Calibrating Black-Box Judges with Counterfactual Probes

arXiv:2610.18389v1 Announce Type: new Abstract: Large language models are increasingly used as judges to reduce reliance on costly human evaluation, yet their scores can systematically disagree with h

local-aiarxiv-cs-lg

Successful but Unsafe: An Analytical Benchmark for Revealing Collisions in Vision-and-Language Navigation

model-releasesarxiv-cs-ro

arXiv:2610.14682v1 Announce Type: new Abstract: Vision-and-language navigation (VLN) agents are typically evaluated by whether they reach a target and how efficiently they follow an instructed route,

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

arXiv:2610.17057v1 Announce Type: cross Abstract: Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This

PPG-Bench: A Unified Benchmark for PPG Foundation Models

model-releasesarxiv-cs-lg

arXiv:2610.13435v1 Announce Type: new Abstract: Photoplethysmography (PPG) is increasingly recorded by wearable and clinical devices at enormous scale, and a growing number of PPG foundation models (F

12 Oct 2026

Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2610.12696v1 Announce Type: new Abstract: Vision-language-action models (VLAs) offer a promising route toward general-purpose robotics by transferring semantic knowledge from large-scale vision-

12 Oct 2026

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

arXiv:2610.12183v1 Announce Type: cross Abstract: Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited. Recent large la

FastJEV: Understanding Redundancy for Compact JEV Inference

arXiv:2610.11379v1 Announce Type: new Abstract: JEV models make multimodal decisions by directly scoring candidates. Although the common context is encoded once, candidate evaluation can still repeat

researcharxiv-cs-cv

LiteNWM: Efficient Latent World Models for Onboard Visual Navigation in the Wild

arXiv:2610.12368v1 Announce Type: new Abstract: Direct visual navigation policies generate trajectories efficiently but do not explicitly evaluate their future consequences. Generative navigation worl

researcharxiv-cs-ro

MedBenchAgent: Towards Systematic Automation of Medical VLM Benchmark Construction

arXiv:2610.11312v1 Announce Type: new Abstract: Large-scale construction of medical vision-language model (VLM) benchmarks is increasingly feasible with richly annotated imaging datasets and large lan

Applications

Multimodal Large Language Models as Image Classifiers

arXiv:2603.06578v3 Announce Type: replace Abstract: Multimodal Large Language Model (MLLM) classification performance depends critically on evaluation protocol and ground truth quality. Studies compar

applicationsarxiv-cs-cv

RH-Detect: A Unified Benchmark for Reward Hacking Detection

model-releasesarxiv-cs-lg

arXiv:2610.10947v1 Announce Type: new Abstract: Reward hacking, where a model exploits an evaluation signal without completing the intended task, threatens the reliability of deployed language model s

Safe Actions Alone Do Not Ensure Safe Agents: Identifying Unfulfilled Obligations with Guard Models

arXiv:2610.11773v1 Announce Type: new Abstract: Guard models are increasingly used to safeguard LLM-based agents, primarily by identifying actions that agents are forbidden to perform. However, identi

Arctic Questions, Missing Answers: A Dataset and Benchmark for LLM Abstention in Arctic Science

arXiv:2610.09446v1 Announce Type: new Abstract: Large language models (LLMs) should abstain from scientific multiple-choice questions when no option is valid, but frequent abstention alone does not de

8 Oct 2026

Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media Texts

arXiv:2610.10318v1 Announce Type: new Abstract: Social media is a rich source of real-time public sentiment, but widely used sentiment analysis tools are often applied without understanding their limi

8 Oct 2026

PlaySuite: A Large-Scale Benchmark for Interactive Visual Intelligence

arXiv:2610.07127v1 Announce Type: cross Abstract: Recent advances in multimodal foundation models yield strong performance on static perception and reasoning benchmarks, yet such evaluations largely o

7 Oct 2026

Agents

Wiki-Talkie: Multilingual Benchmarking of Persona-Based Agents on Real-World Discussions

arXiv:2610.08513v1 Announce Type: cross Abstract: LLMs are increasingly deployed as autonomous agents in social environments, making it critical to study their ability to faithfully simulate human int

agentsarxiv-cs-ai

7 Oct 2026

Certification of Real Images through Calibrated Content Authentication

arXiv:2610.05870v1 Announce Type: new Abstract: Generative models can synthesize high-quality inauthentic multimedia content that is already being misused at scale. We evaluate twenty deepfake detecto

Continual Action Quality Assessment via Adaptive Manifold-Aligned Graph Regularization

arXiv:2510.06842v2 Announce Type: replace Abstract: Action Quality Assessment (AQA) quantifies human actions in videos, supporting applications in sports scoring, rehabilitation, and skill evaluation.

Repeated-Measure Leakage, Distribution Shift, and Reliability under Partial Observation in Patient World Models

arXiv:2610.04778v1 Announce Type: new Abstract: Patient world models are increasingly proposed for longitudinal prediction, intervention-aware reasoning, and clinical-trial simulation. Causal or clini

researcharxiv-cs-lg

Revealing After Overwriting: An Exponential POMDP OPE Lower Bound under History-Dependent Logging

arXiv:2610.05063v1 Announce Type: new Abstract: Multi-step revealing can make off-policy evaluation tractable under memoryless logging. With history-dependent logging, state decodability and target-re

safetyarxiv-cs-lg

Revisiting the Generalization of Neural Graph Edit Distance Models

arXiv:2610.04644v1 Announce Type: new Abstract: Neural approaches to Graph Edit Distance (GED) have achieved strong results under standard within-dataset evaluation, but much less is known about how w

researcharxiv-cs-lg

Beyond Trained Models: Compiling GNNs for a Sound Explainer Benchmark

arXiv:2610.03526v1 Announce Type: cross Abstract: Explainers for Graph Neural Networks (GNNs) are commonly evaluated by their plausibility, i.e., how well their explanations recover a predefined groun

Coverage You Can Steer: Online Conformal Calibration for RL-Driven Hardware-Aware NAS

arXiv:2610.03127v1 Announce Type: new Abstract: Hardware-aware neural architecture search (NAS) is dominated by evaluation cost: every architecture must be trained before its reward is known. Conforma

safetyarxiv-cs-lg

DEPICT: Scoring Text-to-Image Alignment by Answer Agreement

arXiv:2610.03617v1 Announce Type: new Abstract: Image-text alignment is a core problem in computer vision with applications in caption evaluation, hallucination detection, data curation, and the bench

safetyarxiv-cs-cv

Effects of interpulse-interval variation on deep-learning classification of bat vocalizations

arXiv:2610.02284v1 Announce Type: new Abstract: Temporal context may aid automated bat-species classification, but the contribution of specific features remains unclear. We investigated whether variat

researcharxiv-cs-lg

Multilingual GSM-Symbolic: What determines capability transfer across languages?

arXiv:2610.03367v1 Announce Type: new Abstract: We understand little about how capabilities acquired in one language carry over to another, or what governs this transfer: evaluations rely on incompara

researcharxiv-cs-ai

SyntaxBench: A Statistical Diagnostic Framework for Character-Level Reasoning in Large Language Models

arXiv:2610.03329v1 Announce Type: cross Abstract: Large language models are increasingly used where small syntactic errors matter, yet character-level reasoning is still evaluated mostly through isola