Search: “evaluation”

GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives

arXiv:2605.09027v1 Announce Type: new Abstract: In multi-agent systems (MAS), a single deceptive agent can nullify all gains of an agentic AI collective and evade deployed defenses. However, existing

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

GraphBench: Next-generation graph learning benchmarking

DGX agent

arXiv:2512.04475v5 Announce Type: replace-cross Abstract: Machine learning on graphs has made substantial progress across domains such as molecular property prediction and chip design. Yet benchmarkin

model-releasesarxiv-cs-ai

12 May 2026

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

arXiv:2605.09972v1 Announce Type: cross Abstract: End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving

Improving TMS EEG Signal Quality for Closed-Loop Neuro Stimulation via Source-Domain Denoising

arXiv:2605.08184v1 Announce Type: cross Abstract: This research addresses a validated TMS EEG cleaning pipeline and a corresponding benchmark dataset. It evaluates two widely used artifact removal pip

LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

arXiv:2605.10779v1 Announce Type: cross Abstract: The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content s

MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems

model-releasesarxiv-cs-ro

arXiv:2605.10904v1 Announce Type: new Abstract: Vehicle-to-Everything (V2X) communication has emerged as a promising paradigm for autonomous driving, enabling connected agents to share complementary p

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

arXiv:2605.09661v1 Announce Type: cross Abstract: Large language models (LLMs) have saturated standard medical benchmarks that test factual recall, yet their ability to perform higher-order reasoning,

MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs

arXiv:2605.08374v1 Announce Type: new Abstract: Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval

MESD: A Risk-Sensitive Metric for Explanation Fairness Across Intersectional Subgroups

arXiv:2603.13452v2 Announce Type: replace Abstract: Fairness in machine learning is predominantly evaluated through outcome-oriented metrics, such as Demographic parity, which measure whether predicti

Multi-layer attentive probing improves transfer of audio representations for bioacoustics

arXiv:2605.10494v1 Announce Type: cross Abstract: Probing heads map the representations learned from audio by a machine learning model to downstream task labels and are a key component in evaluating r

safetyarxiv-cs-ai

Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search

arXiv:2605.08762v1 Announce Type: cross Abstract: Current omni-modal benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start

Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

arXiv:2410.14702v2 Announce Type: replace Abstract: Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstra

Probing Routing-Conditional Calibration in Attention-Residual Transformers

arXiv:2605.09850v1 Announce Type: cross Abstract: Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly ac

researcharxiv-cs-ai

Reducing Annotation Burden for Femoral Cartilage Segmentation in Knee MRI via Cross-Sequence Transfer Learning

arXiv:2605.09067v1 Announce Type: new Abstract: Purpose: To develop and evaluate cross-sequence transfer learning for automatic femoral cartilage segmentation, testing bidirectional transfer between d

researcharxiv-cs-cv

RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

arXiv:2605.10899v1 Announce Type: new Abstract: Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyo

safetyarxiv-cs-cl

SalesSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators

arXiv:2605.08334v1 Announce Type: new Abstract: We present SalesSim, a framework and testbed for evaluating the ability of Multimodal Large Language Models (MLLMs) to simulate realistic, persona-drive

safetyarxiv-cs-cl

SayNext-Bench: Why Do LLMs Struggle with Next-Utterance Anticipation?

arXiv:2602.00327v2 Announce Type: replace Abstract: We explore the use of large language models (LLMs) for next-utterance anticipation in human dialogue. Despite recent advances in LLMs demonstrating

SMOG: Scalable Meta-Learning for Multi-Objective Bayesian Optimization

arXiv:2601.22131v2 Announce Type: replace Abstract: Multi-objective optimization aims to solve problems with competing objectives. Evaluating such problems is often slow or expensive, limiting the bud

researcharxiv-cs-lg

SoccerLens: Grounded Soccer Video Understanding Beyond Accuracy

arXiv:2605.09598v1 Announce Type: new Abstract: Vision-language models (VLMs) have recently shown strong potential in soccer video understanding. However, given the high complexity of soccer videos du

SPDEBench: An Extensive Benchmark for Learning Stochastic PDEs

arXiv:2505.18511v2 Announce Type: replace Abstract: Stochastic Partial Differential Equations (SPDEs) driven by random noise play a central role in modeling physical processes with rough spatio-tempor

V4FinBench: Benchmarking Tabular Foundation Models, LLMs, and Standard Methods on Corporate Bankruptcy Prediction

arXiv:2605.10896v1 Announce Type: new Abstract: Corporate bankruptcy prediction is a high-stakes financial task characterized by severe class imbalance and multi-horizon forecasting demands. Public da

VISOR: A Vision-Language Model-based Test Oracle for Testing Robot

model-releasesarxiv-cs-ro

arXiv:2605.10408v1 Announce Type: cross Abstract: Testing robots requires assessing whether they perform their intended tasks correctly, dependably, and with high quality, a challenge known as the tes

VISTA: A Benchmark for Real-Time Video Streaming under Network Impairments in Surgical Teleoperation

model-releasesarxiv-cs-ro

arXiv:2605.08886v1 Announce Type: cross Abstract: Real-time video streaming is crucial in surgical teleoperation, yet reproducible evaluation under realistic network impairments remains limited. This

WorldReasonBench: Human-Aligned Stress Testing of Video Generators as Future World-State Predictors

arXiv:2605.10434v1 Announce Type: new Abstract: Commercial video generation systems such as Seedance2.0 and Veo3.1 have rapidly improved, strengthening the view that video generators may be evolving i

Your Simulation Runs but Solves the Wrong Physics: PDE-Grounded Intent Verification for LLM-Generated Multiphysics Simulation Code

arXiv:2605.09360v1 Announce Type: cross Abstract: Execution-based evaluation of LLM-generated code implicitly treats successful execution as a proxy for correctness. In scientific simulation, this pro

A Finite-Iteration Theory for Asynchronous Categorical Distributional Temporal-Difference Learning

arXiv:2605.06866v1 Announce Type: new Abstract: Recent non-asymptotic analyses have substantially advanced the theory of distributional policy evaluation, but they largely concern synchronous full-sta

safetyarxiv-cs-lg

Bias and Uncertainty in LLM-as-a-Judge Estimation

arXiv:2605.06939v1 Announce Type: new Abstract: LLM-as-a-Judge evaluation has become a standard tool for assessing base model performance. However, characterizing performance via the naive estimator,

safetyarxiv-cs-lg

Can You Break RLVER? Probing Adversarial Robustness of RL-Trained Empathetic Agents

arXiv:2605.07138v1 Announce Type: new Abstract: Reinforcement learning from verifiable emotion rewards RLVER has produced language models with strong empathetic performance, evaluated on benchmarks th

EgoPro-Bench: Benchmarking Personalized Proactive Interaction in Egocentric Video Streams

arXiv:2605.07299v1 Announce Type: cross Abstract: Existing Multimodal Large Language Models (MLLMs) remain primarily reactive, failing to continuously perceive environments or proactively assist users

Applications

Ensemble Learning for Healthcare: A Comparative Analysis of Hybrid Voting and Ensemble Stacking in Obesity Risk Prediction

arXiv:2509.02826v2 Announce Type: replace-cross Abstract: Obesity is a critical global health issue driven by dietary, physiological, and environmental factors, and is strongly associated with chronic

applicationsarxiv-cs-ai

Exact Is Easier: Credit Assignment for Cooperative LLM Agents

arXiv:2603.06859v2 Announce Type: replace-cross Abstract: Removing an agent from a cooperative team to measure its contribution seems natural, yet in multi-agent LLM systems this evaluation distorts t

GAD in the Wild: Benchmarking Graph Anomaly Detection under Realistic Deployment Challenges

arXiv:2605.07133v1 Announce Type: cross Abstract: Graph Anomaly Detection (GAD) is a critical task in graph machine learning with vital applications in financial fraud detection and social platform go

Measuring and Mitigating the Distributional Gap Between Real and Simulated User Behaviors

arXiv:2605.07847v1 Announce Type: new Abstract: As user simulators are increasingly used for interactive training and evaluation of AI assistants, it is essential that they represent the diverse behav

researcharxiv-cs-cl

SCENE: Recognizing Social Norms and Sanctioning in Group Chats

arXiv:2605.07823v1 Announce Type: new Abstract: Online group chats are social spaces with implicit behavior patterns that, when broken, are often met with social sanctioning from the group. The abilit

Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression

arXiv:2502.01941v3 Announce Type: replace-cross Abstract: While Key-Value (KV) cache compression is essential for efficient LLM inference, current evaluations disproportionately focus on sparse retrie

TraceAV-Bench: Benchmarking Multi-Hop Trajectory Reasoning over Long Audio-Visual Videos

arXiv:2605.07593v1 Announce Type: new Abstract: Real-world audio-visual understanding requires chaining evidence that is sparse, temporally dispersed, and split across the visual and auditory streams,

Uneven Evolution of Cognition Across Generations of Generative AI Models

arXiv:2605.06815v1 Announce Type: new Abstract: The pursuit of artificial general intelligence necessitates robust methods for evaluating the cognitive capabilities of models beyond narrow task perfor

Agents

Signals V2: an LLM-free analyzer to scores live agent trajectories as OpenTelemetry spans.

Signals V2 is an LLM-free analyzer tool that evaluates live agent execution trajectories by scoring them as OpenTelemetry spans, enabling developers to monitor and trace AI agent behavior without rely

agentsr-ollama

9 May 2026

A unified Benchmark for Multi-Frame Image Restoration under Severe Refractive Warping

arXiv:2605.05079v1 Announce Type: new Abstract: Video sequence capturing through refractive dynamic media, such as a turbulent air or water surface, often suffer from severe geometric distortions and

Are LLMs Ready for Conflict Monitoring? Empirical Evidence from West Africa

arXiv:2605.04177v1 Announce Type: new Abstract: As LLMs enter conflict monitoring, understanding systematic distortions in their outputs is critical for humanitarian accountability. We evaluate four v

Applications

Automatically Finding and Validating Unexpected Side-Effects of Interventions on Language Models

arXiv:2605.05090v1 Announce Type: new Abstract: We present an automated, contrastive evaluation pipeline for auditing the behavioral impact of interventions on large language models. Given a base mode

applicationsarxiv-cs-cl

Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)

arXiv:2605.04556v1 Announce Type: cross Abstract: The Massive Sound Embedding Benchmark (MSEB) has emerged as a standard for evaluating the functional breadth of audio models. While initial baselines

BenCSSmark: Making the Social Sciences Count in LLM Research

arXiv:2605.04886v1 Announce Type: new Abstract: This position paper argues that the under-representation of social science tasks in contemporary LLM benchmarks limits advances in both LLM evaluation a

Beyond Fixed Thresholds and Domain-Specific Benchmarks for Explainable Multi-Task Classification in Autonomous Vehicles

arXiv:2605.04299v1 Announce Type: new Abstract: Scene understanding is a vital part of autonomous driving systems, which requires the use of deep learning models. Deep learning methods are intrinsical

safetyarxiv-cs-cv

Can AI Help You Get Over Your Breakup? One Session with a Belief-Reframing Chatbot Shows Sustained Distress Reduction

arXiv:2605.03261v1 Announce Type: cross Abstract: Romantic breakups are among the most common and intense sources of psychological distress. We evaluated *overit*, a single-session AI chatbot that use

researcharxiv-cs-ai

DiffCap-Bench: A Comprehensive, Challenging, Robust Benchmark for Image Difference Captioning

arXiv:2605.04503v1 Announce Type: new Abstract: Image Difference Captioning (IDC) generates natural language descriptions that precisely identify differences between two images, serving as a key bench

OracleProto: A Reproducible Framework for Benchmarking LLM Native Forecasting via Knowledge Cutoff and Temporal Masking

arXiv:2605.03762v1 Announce Type: new Abstract: Large language models are moving from static text generators toward real-world decision-support systems, where forecasting is a composite capability tha

safetyarxiv-cs-ai

QUIVER: Cost-Aware Adaptive Preference Querying in Surrogate-Assisted Evolutionary Multi-Objective Optimization

arXiv:2605.04267v1 Announce Type: new Abstract: Interactive multi-objective optimization systems face a budget allocation dilemma: one can spend resources on expensive objective evaluations or on elic

researcharxiv-cs-lg