Search: “evaluation”

CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

arXiv:2512.01089v2 Announce Type: replace Abstract: Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by t

18 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery

DGX agent

arXiv:2605.15412v1 Announce Type: cross Abstract: Modern quantitative trading increasingly relies on systematic models to extract predictive signals from large-scale financial data, where alpha factor

model-releasesarxiv-cs-ai

18 May 2026

HAI-Eval: Measuring Human-AI Synergy in Collaborative Coding

arXiv:2512.04111v2 Announce Type: replace-cross Abstract: LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans n

18 May 2026

Latent Video Prediction Learns Better World Models

arXiv:2605.15618v1 Announce Type: cross Abstract: Self-supervised video models are increasingly framed as world models, yet their evaluation remains largely confined to a single top-1 accuracy score o

researcharxiv-cs-ai

18 May 2026

A Benchmark for Early-stage Parkinson's Disease Detection from Speech

arXiv:2605.14066v1 Announce Type: cross Abstract: Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare

A Regret Perspective on Online Multiple Testing

arXiv:2605.13916v1 Announce Type: cross Abstract: Online Multiple Testing (OMT), a fundamental pillar of sequential statistical inference, traditionally evaluates the False Discovery Rate (FDR) and st

Agentic Design of Compositional Descriptors via Autoresearch for Materials Science Applications

arXiv:2605.14671v1 Announce Type: cross Abstract: Autoresearch offers a flexible paradigm for automating scientific tasks, in which an AI agent proposes, implements, evaluates, and refines candidate s

Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows

arXiv:2605.14322v1 Announce Type: new Abstract: Language agents are increasingly deployed in complex professional workflows, with tutoring emerging as a particularly high-stakes capability that remain

Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2

arXiv:2605.13904v1 Announce Type: cross Abstract: Brain encoder models predict cortical fMRI responses from the internal activations of pretrained vision and language networks, and are typically evalu

researcharxiv-cs-lg

Gradient Iterated Temporal-Difference Learning

arXiv:2603.07833v2 Announce Type: replace-cross Abstract: Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this par

Heuristic Pathologies and Further Variance Reduction via Uncertainty Propagation in the AIVAT Family of Techniques

arXiv:2605.14261v1 Announce Type: new Abstract: How should an agent's performance in a multiagent environment be evaluated when there is a limited sample size or a high cost of running a trial? The AI

Multi-Dimensional Model Integrity and Responsibility Assessment Index and Scoring Framework

arXiv:2605.14550v1 Announce Type: new Abstract: Artificial intelligence in high-stakes tabular domains cannot be evaluated by predictive performance alone, yet current practice still assesses explaina

safetyarxiv-cs-lg

NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces

arXiv:2605.14698v1 Announce Type: cross Abstract: Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks. They have emerged across fields, including

Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

arXiv:2605.14040v1 Announce Type: new Abstract: We audit the multimodal-physics evaluation pipeline end-to-end and document three undetected construction practices that distort how the field measures

Ready from Day 1: Population-Aware Coordination for Large-Scale Constrained Multi-Agent Systems

arXiv:2605.13900v1 Announce Type: cross Abstract: In large-scale multi-agent systems with shared resource constraints, an upstream planner must iteratively evaluate candidate resource plans -- assessi

agentsarxiv-cs-lg

Reasoning Model Is Superior LLM-Judge, Yet Suffers from Biases

arXiv:2601.03630v2 Announce Type: replace Abstract: This paper presents the first systematic comparison investigating whether Large Reasoning Models (LRMs) are superior judges to non-reasoning LLMs. O

safetyarxiv-cs-cl

Smooth Multi-Policy Causal Effect Estimation in Longitudinal Settings

arXiv:2605.14284v1 Announce Type: new Abstract: Comparative evaluation of multiple dynamic treatment policies is essential for healthcare and policy decisions, yet conventional longitudinal causal inf

safetyarxiv-cs-lg

The Great Pretender: A Stochasticity Problem in LLM Jailbreak

arXiv:2605.14418v1 Announce Type: cross Abstract: 'Oh-Oh, yes, I'm the great pretender. Pretending that I'm doing well. My need is such, I pretend too much...' summarizes the state in the area of jail

When Evidence Conflicts: Uncertainty and Order Effects in Retrieval-Augmented Biomedical Question Answering

arXiv:2605.14115v1 Announce Type: new Abstract: Biomedical retrieval-augmented large language models (LLMs) often face evidence that is incomplete, misleading, or internally contradictory, yet evaluat

researcharxiv-cs-cl

BEAVER: An Enterprise Benchmark for Text-to-SQL

arXiv:2409.02038v3 Announce Type: replace-cross Abstract: Existing text-to-SQL benchmarks have largely been constructed from public databases with well-structured schemas and simplistic question-SQL p

BrainAnytime: Anatomy-Aware Cross-Modal Pretraining for Brain Image Analysis with Arbitrary Modality Availability

arXiv:2605.13059v1 Announce Type: new Abstract: Clinical diagnostic workups typically follow a modality escalation pathway: after initial clinical evaluation, clinicians begin with routine structural

researcharxiv-cs-cv

Cross Modality Image Translation In Medical Imaging Using Generative Frameworks

arXiv:2605.13686v1 Announce Type: cross Abstract: Medical image-to-image (I2I) translation enables virtual scanning, i.e. the synthesis of a target imaging modality from a source one without additiona

researcharxiv-cs-ai

Edit-Compass & EditReward-Compass: A Unified Benchmark for Image Editing and Reward Modeling

model-releasesarxiv-cs-cv

arXiv:2605.13062v1 Announce Type: new Abstract: Recent image editing models have achieved remarkable progress in instruction following, multimodal understanding, and complex visual editing. However, e

GHGbench: A Unified Multi-Entity, Multi-Task Benchmark for Carbon Emission Prediction

arXiv:2605.13743v1 Announce Type: new Abstract: Open datasets and benchmarks for entity-level carbon-emission prediction remain fragmented across access, scale, granularity, and evaluation. We introdu

Pragmatic Curiosity: A Unified Framework for Hybrid Learning and Optimization via Active Inference

arXiv:2602.06104v2 Announce Type: replace Abstract: Many engineering and scientific workflows rely on expensive black-box evaluations, requiring sequential decisions that must both improve task perfor

researcharxiv-cs-lg

Quantifying LLM Safety Degradation Under Repeated Attacks Using Survival Analysis

arXiv:2605.12869v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed in a wide range of applications, yet remain vulnerable to adversarial jailbreak attacks that ci

Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency

arXiv:2605.13047v1 Announce Type: cross Abstract: Evaluating whether large vision-language models (VLMs) align with human perception for high-level semantic scene comprehension remains a challenge. Tr

VERA-MH: Validation of Ethical and Responsible AI in Mental Health

arXiv:2605.13318v1 Announce Type: new Abstract: Chatbot usage has increased, including in fields for which they were never developed for--notably mental health support. To that end, we introduce Valid

Watermarking Should Be Treated as a Monitoring Primitive

arXiv:2605.13095v1 Announce Type: cross Abstract: Watermarking is widely proposed for provenance, attribution, and safety monitoring in generative models, yet is typically evaluated only under adversa

When Does Hierarchy Help? Benchmarking Agent Coordination in Event-Driven Industrial Scheduling

arXiv:2605.13172v1 Announce Type: cross Abstract: Recent advances in agent and multi-agent systems have shown strong performance on tool use, reasoning, and collaborative tasks. However, existing benc

Industry

Your doctor’s AI notetaker may be making things up, Ontario audit finds

Ontario's auditor general found that AI note-taking tools intended for use by doctors provided incorrect and incomplete information or demonstrated 'hallucinations,' and were not evaluated adequately.

industryars-technica

CheXTemporal: A Dataset for Temporally-Grounded Reasoning in Chest Radiography

arXiv:2605.11304v1 Announce Type: new Abstract: Chest radiograph interpretation requires temporal reasoning over prior and current studies, yet most vision-language models are trained on static image-

safetyarxiv-cs-cv

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

arXiv:2605.11143v1 Announce Type: new Abstract: Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation,

Deep Reasoning in General Purpose Agents via Structured Meta-Cognition

arXiv:2605.11388v1 Announce Type: new Abstract: Humans intuitively solve complex problems by flexibly shifting among reasoning modes: they plan, execute, revise intermediate goals, resolve ambiguity t

agentsarxiv-cs-cl

QuIDE: Mastering the Quantized Intelligence Trade-off via Active Optimization

arXiv:2605.10959v1 Announce Type: new Abstract: There is currently no unified metric for evaluating the efficiency of quantized neural networks. We propose QuIDE, built around the Intelligence Index I

Space Syntax-guided Post-training for Residential Floor Plan Generation

arXiv:2602.22507v2 Announce Type: replace-cross Abstract: Residential floor plan generation requires not only geometric fidelity but also spatial configurational logic: shared living spaces should be

safetyarxiv-cs-cv

VERDI: Single-Call Confidence Estimation for Verification-Based LLM Judges via Decomposed Inference

arXiv:2605.11334v1 Announce Type: cross Abstract: LLM-as-Judge systems are widely deployed for automated evaluation, yet practitioners lack reliable methods to know when a judge's verdict should be tr

A Reflective Storytelling Agent for Older Adults: Integrating Argumentation Schemes and Argument Mining in LLM-Based Personalised Narratives

arXiv:2605.10531v1 Announce Type: new Abstract: This work investigates whether knowledge-driven large language model (LLM)-based storytelling can support purposeful narrative interaction with a digita

ACWM-Phys: Investigating Generalized Physical Interaction in Action-Conditioned Video World Models

model-releasesarxiv-cs-cv

arXiv:2605.08567v1 Announce Type: new Abstract: Action-conditioned world models (ACWMs) have shown strong promise for video prediction and decision-making. However, existing benchmarks are largely res

Beyond Static Bias: Adaptive Multi-Fidelity Bandits with Improving Proxies

arXiv:2605.08558v1 Announce Type: new Abstract: As an extension of the classical multi-armed bandit problem, multi-fidelity multi-armed bandits (MF-MAB) enable individual arms to be evaluated using di

safetyarxiv-cs-lg

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

arXiv:2605.08761v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles,

Compander-Aligned Query Geometry for Quantized Zeroth-Order Optimization

arXiv:2605.10673v1 Announce Type: new Abstract: Low-bit forward evaluation is an attractive route to memory-efficient zeroth-order (ZO) adaptation: the optimizer needs only scalar losses, and the mode

Done, But Not Sure: Disentangling World Completion from Self-Termination in Embodied Agents

arXiv:2605.08747v1 Announce Type: new Abstract: Standard embodied evaluations do not independently score whether an agent correctly commits to task completion at episode closure, a capacity we call te

EconWebArena: Benchmarking Autonomous Agents on Economic Tasks in Realistic Web Environments

arXiv:2506.08136v3 Announce Type: replace Abstract: We introduce EconWebArena, a benchmark for evaluating autonomous agents on complex, multimodal economic tasks in realistic web environments. The ben

EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understanding

arXiv:2605.09874v1 Announce Type: cross Abstract: Next-generation visual assistants, such as smart glasses, embodied agents, and always-on life-logging systems, must reason over an entire day or more

Applications

Elite Polarization in European Parliamentary Speeches: a Novel Measurement Approach Using Large Language Models

arXiv:2507.06658v2 Announce Type: replace-cross Abstract: Theories of democratic stability, populism, and party-system crisis often point to a form of polarization that comparative research rarely mea

applicationsarxiv-cs-ai

Fast-MIA: Efficient and Scalable Membership Inference for LLMs

arXiv:2510.23074v2 Announce Type: replace-cross Abstract: We propose Fast-MIA (https://github.com/Nikkei/fast-mia), a Python library for efficiently evaluating membership inference attacks (MIA) again

researcharxiv-cs-cl

FormalRewardBench: A Benchmark for Formal Theorem Proving Reward Models

arXiv:2605.10141v1 Announce Type: new Abstract: Recent neural theorem provers use reinforcement learning with verifiable rewards (RLVR), where proof assistants provide binary correctness signals. Whil