Search: “evaluation”

X4Val: Learning Neural Surrogates for Variance-Reduced Policy Evaluation

arXiv:2606.05159v1 Announce Type: new Abstract: Rigorous evaluation of learning-based robotic systems is an essential prerequisite for deployment. However, real-world test data is expensive to gather;

safetyarxiv-cs-ro

4 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining

DGX agent

arXiv:2508.13174v2 Announce Type: replace Abstract: Formula alpha mining, which generates predictive signals from financial data, is critical for quantitative investment. Although various algorithmic

researcharxiv-cs-ai

3 Jun 2026

Agents

SAGE: A Quantitative Evaluation of Socialized Evolution in Agent Ecosystems

arXiv:2606.03544v1 Announce Type: new Abstract: Self-improving language agents are typically evaluated in isolation: an agent attempts a task, receives feedback, and iteratively refines its own behavi

agentsarxiv-cs-ai

3 Jun 2026

CardioLens: Revealing the Clinical Reality Gap of MLLMs via Multi-Sequence Cardiac MRI Evaluations

arXiv:2606.00123v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance on public medical benchmarks, yet existing evaluations often remain weak proxie

applicationsarxiv-cs-ai

Certificate-Guided Evaluation of Reinforcement Learning Generalization

arXiv:2606.00840v1 Announce Type: new Abstract: This work presents a logic-driven framework to evaluate the performance of reinforcement learning (RL) algorithms in their ability to generalize to unse

From Evaluation to Design: Using Potential Energy Surface Smoothness Metrics to Guide Machine Learning Interatomic Potential Architectures

arXiv:2602.04861v2 Announce Type: replace-cross Abstract: Machine Learning Interatomic Potentials (MLIPs) sometimes fail to reproduce the physical smoothness of the quantum potential energy surface (P

Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?

arXiv:2510.10541v2 Announce Type: replace-cross Abstract: Current benchmarks are inadequate for evaluating progress in reinforcement learning (RL) for large language models (LLMs).Despite recent bench

Hardware

SUPREME: A Multi-GPU Framework for Reproducible Image Unlearning Method Evaluation

arXiv:2606.00380v1 Announce Type: cross Abstract: Machine unlearning removes the influence of specific training data from a trained model without retraining it from scratch. Evaluating an unlearning m

hardwarearxiv-cs-ai

A Novel Evaluation Metric for Unsupervised Learning in AIS-Based Maritime Anomaly Detection: MADQI

arXiv:2605.30388v1 Announce Type: new Abstract: This paper introduces a new systematic framework for detecting anomalies in maritime Automatic Identification System (AIS) datasets. These anomalies inc

researcharxiv-cs-lg

AdvScene: Rethinking Adversarial Patch Evaluation Through Scene Robustness

arXiv:2605.30578v1 Announce Type: cross Abstract: Adversarial patches are physical patterns attached to real objects to mislead AI vision systems. Their real-world risk is not determined by a single s

applicationsarxiv-cs-cv

Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge

arXiv:2605.30568v1 Announce Type: new Abstract: LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers o

researcharxiv-cs-cl

OpenSTBench: Beyond Semantic Evaluation for Speech Translation

arXiv:2605.30792v1 Announce Type: cross Abstract: Speech translation systems increasingly span speech-to-text translation (S2TT), speech-to-speech translation (S2ST), offline translation, and streamin

Position: Evaluation of ECG Representations Must Be Fixed

arXiv:2602.17531v2 Announce Type: replace-cross Abstract: This position paper argues that current benchmarking practice in 12-lead ECG representation learning must be fixed to ensure progress is relia

Redefining Instance Matching: A Unified Framework for Part-Aware Matching in Panoptic Segmentation Evaluation

arXiv:2605.31094v1 Announce Type: cross Abstract: The Panoptic Quality (PQ) metric is the standard for jointly evaluating instance and semantic segmentation. However, its original definition relies on

TeachObs: A Human-Validated Benchmark for Multimodal Teaching Observation and Model Evaluation

arXiv:2605.30673v1 Announce Type: new Abstract: Classroom videos contain observable teaching practices, but their pedagogical and visual signals are rarely organized in forms suitable for model evalua

Comparative Evaluation of Machine Translation Systems on Images with Text

arXiv:2605.29476v1 Announce Type: new Abstract: This work presents a comparative evaluation of machine translation systems applied to images containing textual information, a task that lies at the int

DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

arXiv:2605.30090v1 Announce Type: new Abstract: Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinema

GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models

arXiv:2605.28848v1 Announce Type: cross Abstract: Deployed language models are evaluated in a non-stationary environment: model versions, retrieval layers, safety systems, and real-world inputs all ch

GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

arXiv:2605.29532v1 Announce Type: cross Abstract: Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an a

LoRe: Adaptive Interaction-Evaluation Routing with Per-Step Interaction Budgets for Iterative Graph Solvers

arXiv:2605.29005v1 Announce Type: cross Abstract: Diffusion-based neural solvers for combinatorial optimization repeatedly re-evaluate dense edge/factor interactions, making inference expensive in wal

ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation

model-releasesarxiv-cs-cv

arXiv:2605.29579v1 Announce Type: new Abstract: While multimodal large language models (MLLMs) have achieved rapid progress in vision-language understanding, they remain prone to multimodal hallucinat

Realistic honeypot evaluations for scheming propensity

model-releasesarxiv-cs-lg

arXiv:2605.29729v1 Announce Type: new Abstract: We introduce scheming honeypot evaluations, a framework for testing whether models will pursue instrumental goals if given the opportunity. Our scheming

CANDOR: Counterfactual ANnotated DOubly Robust Off-Policy Evaluation

arXiv:2412.08052v2 Announce Type: replace Abstract: Off-policy evaluation (OPE) is critical for applying contextual bandit algorithms to high-stakes decision-making settings such as healthcare, where

safetyarxiv-cs-lg

28 May 2026

Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information

arXiv:2601.03089v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) are increasingly evaluated with input attribution methods, yet comparing such explanations remains challenging. E

Tutorials

MatFormBench: A Benchmarking Evaluation Framework for Target-Driven Materials Formulation

arXiv:2605.26741v1 Announce Type: cross Abstract: Inverse design of materials has significantly advanced target-driven formulation optimization, yet existing materials machine learning benchmarks rema

tutorialsarxiv-cs-ai

Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models

arXiv:2603.16654v2 Announce Type: replace-cross Abstract: Evaluating the reasoning abilities of large language models (LLMs) solely from final answers can obscure failures in intermediate steps, espec

Prompt Injection Detection is Regime-Dependent: A Deployment-Aware Evaluation with Interpretable Structural Signals

arXiv:2605.26999v1 Announce Type: new Abstract: Prompt injection poses a critical threat to the safe deployment of large language models, yet existing detection approaches are typically evaluated unde

Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems

model-releasesarxiv-cs-lg

arXiv:2605.22891v1 Announce Type: new Abstract: Evaluation in scientific reconstruction is dominated by pointwise metrics - RMSE, MAE, per-event resolution - under the implicit assumption that lower e

25 May 2026

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

arXiv:2605.21728v1 Announce Type: cross Abstract: Image captioning evaluation remains a significant challenge, as vision-language models evolve toward more challenging capabilities such as generating

22 May 2026

Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping

model-releasesarxiv-cs-cv

arXiv:2605.22578v1 Announce Type: new Abstract: Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (S

22 May 2026

SciCustom: A Framework for Custom Evaluation of Scientific Capabilities in Large Language Models

arXiv:2605.19357v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied to scientific research, yet existing evaluations often fail to reflect the fine-grained capabiliti

20 May 2026

The Annotation Scarcity Paradox in Low-Resource NLP Evaluation: A Decade of Acceleration and Emerging Constraints

arXiv:2605.19066v1 Announce Type: new Abstract: Over the past decade, low-resource natural language processing (NLP) has experienced explosive growth, propelled by cross-lingual transfer, massively mu

20 May 2026

AIM: Adversarial Information Masking for Faithfulness Evaluation of Saliency Maps

arXiv:2605.16905v1 Announce Type: cross Abstract: Post-hoc saliency methods are widely used to interpret deep neural networks, but their faithfulness is difficult to evaluate reliably. Existing evalua

safetyarxiv-cs-cv

From Isolated Scoring to Collaborative Ranking: A Comparison-Native Framework for LLM-Based Paper Evaluation

arXiv:2603.17588v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are currently applied to scientific paper evaluation by assigning an absolute score to each paper independently.

researcharxiv-cs-cl

LLM-Safety Evaluations Lack Robustness

arXiv:2503.02574v2 Announce Type: replace-cross Abstract: In this paper, we argue that current safety alignment research efforts for large language models are hindered by many intertwined sources of n

safetyarxiv-cs-ai

When Outcome Looks Right But Discipline Fails: Trace-Based Evaluation Under Hidden Competitor State

arXiv:2605.18580v1 Announce Type: new Abstract: Outcome-only evaluation can certify economically unsafe agents: a policy can hit a business KPI while violating deployable behavioral discipline. In hot

Defining Cultural Capabilities for AI Evaluation: A Taxonomy Grounded in Intercultural Communication Theory

arXiv:2605.15990v1 Announce Type: new Abstract: Tremendous efforts have been put into evaluating the inclusivity and effectiveness of AI systems across cultures. However, the cultural capabilities con

applicationsarxiv-cs-cl

18 May 2026

Comparative Evaluation of Machine Learning Approaches for Minority-Class Financial Distress Prediction Under Class Imbalance Constraints

arXiv:2605.14067v1 Announce Type: new Abstract: Financial distress prediction remains a significant challenge in enterprise risk analysis due to the highly imbalanced nature of real-world financial da

applicationsarxiv-cs-lg

Comparing Developer and LLM Biases in Code Evaluation

arXiv:2603.24586v2 Announce Type: replace-cross Abstract: As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial

safetyarxiv-cs-cl

JointAVBench: A Benchmark for Joint Audio-Visual Reasoning Evaluation

model-releasesarxiv-cs-cv

arXiv:2512.12772v2 Announce Type: replace-cross Abstract: Understanding videos inherently requires reasoning over both visual and auditory information. To properly evaluate Omni-Large Language Models

Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

arXiv:2605.14404v1 Announce Type: new Abstract: While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII).

researcharxiv-cs-cl

MemEye: A Visual-Centric Evaluation Framework for Multimodal Agent Memory

arXiv:2605.15128v1 Announce Type: cross Abstract: Long-term agent memory is increasingly multimodal, yet existing evaluations rarely test whether agents preserve the visual evidence needed for later r

NodeSynth: Socially Aligned Synthetic Data for AI Evaluation

arXiv:2605.14381v1 Announce Type: cross Abstract: Recent advancements in generative AI facilitate large-scale synthetic data generation for model evaluation. However, without targeted approaches, thes

Quantitative Video World Model Evaluation for Geometric-Consistency

arXiv:2605.15185v1 Announce Type: cross Abstract: Generative video models are increasingly studied as implicit world models, yet evaluating whether they produce physically plausible 3D structure and m

safetyarxiv-cs-ai

Agents

AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation

arXiv:2605.12925v1 Announce Type: cross Abstract: Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view trea

agentsarxiv-cs-ai

14 May 2026

Local Ai

AIvaluateXR: An Evaluation Framework for on-Device AI in XR with Benchmarking Results

arXiv:2502.15761v3 Announce Type: replace-cross Abstract: The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interacti

local-aiarxiv-cs-ai

14 May 2026

Strikingness-Aware Evaluation for Temporal Knowledge Graph Reasoning

arXiv:2605.13153v1 Announce Type: new Abstract: Temporal Knowledge Graph Reasoning (TKGR) aims at inferring missing (especially future) events from historical data. Current evaluation in TKGR uniforml

14 May 2026

DiffScore: Text Evaluation Beyond Autoregressive Likelihood

arXiv:2605.11601v1 Announce Type: new Abstract: Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early t