Search: “evaluation”

Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

arXiv:2605.14040v1 Announce Type: new Abstract: We audit the multimodal-physics evaluation pipeline end-to-end and document three undetected construction practices that distort how the field measures

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

Ready from Day 1: Population-Aware Coordination for Large-Scale Constrained Multi-Agent Systems

DGX agent

arXiv:2605.13900v1 Announce Type: cross Abstract: In large-scale multi-agent systems with shared resource constraints, an upstream planner must iteratively evaluate candidate resource plans -- assessi

agentsarxiv-cs-lg

15 May 2026

Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases

arXiv:2601.03630v2 Announce Type: replace Abstract: This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judges to non-reasoning LLMs. O

safetyarxiv-cs-cl

Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings

arXiv:2605.14284v1 Announce Type: new Abstract: Comparative evaluation of multiple dynamic treatment policies is essential for healthcare and policy decisions, yet conventional longitudinal causal inf

safetyarxiv-cs-lg

The Great Pretender: A Stochasticity Problem in LLM Jailbreak

arXiv:2605.14418v1 Announce Type: cross Abstract: 'Oh-Oh, yes, I'm the great pretender. Pretending that I'm doing well. My need is such, I pretend too much...' summarizes the state in the area of jail

When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering

arXiv:2605.14115v1 Announce Type: new Abstract: Biomedical retrieval-augmented large language models (LLMs) often face evidence that is incomplete, misleading, or internally contradictory, yet evaluat

researcharxiv-cs-cl

BEAVER: An Enterprise Benchmark for Text-to-SQL

arXiv:2409.02038v3 Announce Type: replace-cross Abstract: Existing text-to-SQL benchmarks have largely been constructed from public databases with well-structured schemas and simplistic question-SQL p

BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability

arXiv:2605.13059v1 Announce Type: new Abstract: Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural

researcharxiv-cs-cv

Cross Modality Image Translation In Medical Imaging Using Generative Frameworks

arXiv:2605.13686v1 Announce Type: cross Abstract: Medical image-to-image (I2I) translation enables virtual scanning, i.e. the synthesis of a target imaging modality from a source one without additiona

researcharxiv-cs-ai

Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

model-releasesarxiv-cs-cv

arXiv:2605.13062v1 Announce Type: new Abstract: Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, e

GHGbench: A Unified Multi-Entity, Multi-Task Benchmark for Carbon Emission Prediction

arXiv:2605.13743v1 Announce Type: new Abstract: Open datasets and benchmarks for entity-level carbon-emission prediction remain fragmented across access, scale, granularity, and evaluation. We introdu

Pragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active Inference

arXiv:2602.06104v2 Announce Type: replace Abstract: Many engineering and scientific workflows rely on expensive black-box evaluations, requiring sequential decisions that must both improve task perfor

researcharxiv-cs-lg

Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis

arXiv:2605.12869v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in a wide range of applications, yet remain vulnerable to adversarial jailbreak attacks that ci

Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency

arXiv:2605.13047v1 Announce Type: cross Abstract: Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Tr

VERA-MH: Validation of Ethical and Responsible AI in Mental Health

arXiv:2605.13318v1 Announce Type: new Abstract: Chatbot usage has increased, including in fields for which they were never developed for--notably mental health support. To that end, we introduce Valid

Watermarking Should Be Treated as a Monitoring Primitive

arXiv:2605.13095v1 Announce Type: cross Abstract: Watermarking is widely proposed for provenance, attribution, and safety monitoring in generative models, yet is typically evaluated only under adversa

When Does Hierarchy Help? Benchmarking Agent Coordination in Event-Driven Industrial Scheduling

arXiv:2605.13172v1 Announce Type: cross Abstract: Recent advances in agent and multi-agent systems have shown strong performance on tool use, reasoning, and collaborative tasks. However, existing benc

CheXTemporal: A Dataset for Temporally-Grounded Reasoning in Chest Radiography

arXiv:2605.11304v1 Announce Type: new Abstract: Chest radiograph interpretation requires temporal reasoning over prior and current studies, yet most vision-language models are trained on static image-

safetyarxiv-cs-cv

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

arXiv:2605.11143v1 Announce Type: new Abstract: Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation,

Agents

Deep Reasoning in General Purpose Agents via Structured Meta-Cognition

arXiv:2605.11388v1 Announce Type: new Abstract: Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambiguity t

agentsarxiv-cs-cl

QuIDE: Mastering the Quantized Intelligence Trade-off via Active Optimization

arXiv:2605.10959v1 Announce Type: new Abstract: There is currently no unified metric for evaluating the efficiency of quantized neural networks. We propose QuIDE, built around the Intelligence Index I

Space Syntax-guided Post-training for Residential Floor Plan Generation

arXiv:2602.22507v2 Announce Type: replace-cross Abstract: Residential floor plan generation requires not only geometric fidelity but also spatial configurational logic: shared living spaces should be

safetyarxiv-cs-cv

VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference

arXiv:2605.11334v1 Announce Type: cross Abstract: LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be tr

Agents

A Reflective Storytelling Agent for Older Adults: Integrating Argumentation Schemes and Argument Mining in LLM-Based Personalised Narratives

arXiv:2605.10531v1 Announce Type: new Abstract: This work investigates whether knowledge-driven large language model (LLM)-based storytelling can support purposeful narrative interaction with a digita

agentsarxiv-cs-ai

ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

model-releasesarxiv-cs-cv

arXiv:2605.08567v1 Announce Type: new Abstract: Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely res

Beyond Static Bias: Adaptive Multi-Fidelity Bandits with Improving Proxies

arXiv:2605.08558v1 Announce Type: new Abstract: As an extension of the classical multi-armed bandit problem, multi-fidelity multi-armed bandits (MF-MAB) enable individual arms to be evaluated using di

safetyarxiv-cs-lg

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

arXiv:2605.08761v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles,

Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization

arXiv:2605.10673v1 Announce Type: new Abstract: Low-bit forward evaluation is an attractive route to memory-efficient zeroth-order (ZO) adaptation: the optimizer needs only scalar losses, and the mode

Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

arXiv:2605.08747v1 Announce Type: new Abstract: Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call te

EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments

arXiv:2506.08136v3 Announce Type: replace Abstract: We introduce EconWebArena, a benchmark for evaluating autonomous agents on complex, multimodal economic tasks in realistic web environments. The ben

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

arXiv:2605.09874v1 Announce Type: cross Abstract: Next-generation visual assistants, such as smart glasses, embodied agents, and always-on life-logging systems, must reason over an entire day or more

Applications

Elite Polarization in European Parliamentary Speeches: a Novel Measurement Approach Using Large Language Models

arXiv:2507.06658v2 Announce Type: replace-cross Abstract: Theories of democratic stability, populism, and party-system crisis often point to a form of polarization that comparative research rarely mea

applicationsarxiv-cs-ai

Fast-MIA: Efficient and Scalable Membership Inference for LLMs

arXiv:2510.23074v2 Announce Type: replace-cross Abstract: We propose Fast-MIA (https://github.com/Nikkei/fast-mia), a Python library for efficiently evaluating membership inference attacks (MIA) again

researcharxiv-cs-cl

FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models

arXiv:2605.10141v1 Announce Type: new Abstract: Recent neural theorem provers use reinforcement learning with verifiable rewards (RLVR), where proof assistants provide binary correctness signals. Whil

GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives

arXiv:2605.09027v1 Announce Type: new Abstract: In multi-agent systems (MAS), a single deceptive agent can nullify all gains of an agentic AI collective and evade deployed defenses. However, existing

GraphBench: Next-generation graph learning benchmarking

arXiv:2512.04475v5 Announce Type: replace-cross Abstract: Machine learning on graphs has made substantial progress across domains such as molecular property prediction and chip design. Yet benchmarkin

HiDrive: A Closed-Loop Benchmark for High-Level Autonomous Driving

model-releasesarxiv-cs-cv

arXiv:2605.09972v1 Announce Type: cross Abstract: End-to-end autonomous driving has witnessed rapid progress, yet existing benchmarks are increasingly saturated, with state-of-the-art models achieving

Improving TMS EEG Signal Quality for Closed-Loop Neuro Stimulation via Source-Domain Denoising

arXiv:2605.08184v1 Announce Type: cross Abstract: This research addresses a validated TMS EEG cleaning pipeline and a corresponding benchmark dataset. It evaluates two widely used artifact removal pip

LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments

arXiv:2605.10779v1 Announce Type: cross Abstract: The rapid proliferation of LLM-based autonomous agents in real operating system environments introduces a new category of safety risk beyond content s

MDrive: Benchmarking Closed-Loop Cooperative Driving for End-to-End Multi-agent Systems

model-releasesarxiv-cs-ro

arXiv:2605.10904v1 Announce Type: new Abstract: Vehicle-to-Everything (V2X) communication has emerged as a promising paradigm for autonomous driving, enabling connected agents to share complementary p

MedMeta: A Benchmark for LLMs in Synthesizing Meta-Analysis Conclusion from Medical Studies

arXiv:2605.09661v1 Announce Type: cross Abstract: Large language models (LLMs) have saturated standard medical benchmarks that test factual recall, yet their ability to perform higher-order reasoning,

MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs

arXiv:2605.08374v1 Announce Type: new Abstract: Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval

MESD: A Risk-Sensitive Metric for Explanation Fairness Across Intersectional Subgroups

arXiv:2603.13452v2 Announce Type: replace Abstract: Fairness in machine learning is predominantly evaluated through outcome-oriented metrics, such as Demographic parity, which measure whether predicti

Multi-layer attentive probing improves transfer of audio representations for bioacoustics

arXiv:2605.10494v1 Announce Type: cross Abstract: Probing heads map the representations learned from audio by a machine learning model to downstream task labels and are a key component in evaluating r

Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search

arXiv:2605.08762v1 Announce Type: cross Abstract: Current omni-modal benchmarks mainly evaluate models under settings where multiple modalities are provided simultaneously, while the ability to start

Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

arXiv:2410.14702v2 Announce Type: replace Abstract: Multi-modal Large Language Models (MLLMs) exhibit impressive problem-solving abilities in various domains, but their visual comprehension and abstra

Probing Routing-Conditional Calibration in Attention-Residual Transformers

arXiv:2605.09850v1 Announce Type: cross Abstract: Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly ac

researcharxiv-cs-ai

Reducing Annotation Burden for Femoral Cartilage Segmentation in Knee MRI via Cross-Sequence Transfer Learning

arXiv:2605.09067v1 Announce Type: new Abstract: Purpose: To develop and evaluate cross-sequence transfer learning for automatic femoral cartilage segmentation, testing bidirectional transfer between d

researcharxiv-cs-cv