Search: “evaluation”

From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluation

arXiv:2609.19413v1 Announce Type: cross Abstract: Robot manipulation policies are improving quickly, and real-robot evaluation remains the standard evidence for that progress. It still relies on a hum

18 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

DGX agent

arXiv:2609.20758v1 Announce Type: cross Abstract: Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in

model-releasesarxiv-cs-ai

18 Sept 2026

Local Ai

Selective Cotton Boll Localization for Robotic Harvesting: Evaluation of Deep Learning Vision Models Under Field Conditions

arXiv:2609.19592v1 Announce Type: new Abstract: This study developed and evaluated a deep-learning-based perception framework for selective robotic cotton picking. The dataset contained 1,008 annotate

local-aiarxiv-cs-cv

18 Sept 2026

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

arXiv:2609.18052v1 Announce Type: cross Abstract: This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specifica

I code or AI code: A comparative evaluation of AI-rated scores in classroom observations

arXiv:2609.18274v1 Announce Type: cross Abstract: Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet t

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

arXiv:2609.19093v1 Announce Type: cross Abstract: Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might neve

researcharxiv-cs-ai

RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos

arXiv:2609.17843v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain

arXiv:2609.15242v1 Announce Type: new Abstract: Environmental, Social, and Governance (ESG) reporting is critical for corporate accountability, with Large Language Models (LLMs) and Retrieval-Augmente

Empirical Evaluation of Task-Based Permission Scoping Architecture for AI Agents

arXiv:2609.15422v1 Announce Type: new Abstract: AI agents are provisioned the same as employee-owned hosts in many enterprise settings with a static credential set fixed at deployment which includes a

One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs

arXiv:2609.14860v1 Announce Type: cross Abstract: Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benc

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

model-releasesarxiv-cs-cl

arXiv:2504.07825v2 Announce Type: replace Abstract: Commonsense reasoning is a key language model capability, as it is purportedly a prerequisite for many basic tasks, unlike specific factual knowledg

A Comparative Evaluation of Pre-trained Convolutional Neural Networks for Melanoma Detection

arXiv:2609.11550v1 Announce Type: new Abstract: Early diagnosis of melanoma is critical for improving patient survival rates. However, accurately distinguishing melanoma from other skin lesions remain

researcharxiv-cs-cv

11 Sept 2026

Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speech

arXiv:2609.11545v1 Announce Type: new Abstract: Low-resource multilingual text-to-speech (TTS) systems have expanded language coverage, but their robustness under complex text inputs remains insuffici

researcharxiv-cs-cl

11 Sept 2026

Sequence-Informed Geometric Evaluation of RNA 3D Structures

arXiv:2609.10644v1 Announce Type: cross Abstract: Computational RNA structure pipelines generate many candidate conformations for the same sequence. Reliable evaluation therefore requires more than re

safetyarxiv-cs-lg

11 Sept 2026

A Systematic Evaluation of Molecule Generation Models for De Novo Drug Design: From Benchmarks to Practical Insights

arXiv:2609.10099v1 Announce Type: new Abstract: Molecule generation has emerged as a powerful computational tool for de novo drug design, enabling the exploration of chemical space beyond the limits o

Agents

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

arXiv:2609.07712v1 Announce Type: new Abstract: Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible.

agentsarxiv-cs-ai

Local Ai

DiSCo: A Distribution-First Steering and Cultural Prior Evaluation Framework for Measuring Cultural Preference Bias in LLMs

arXiv:2609.10253v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in globally used assistants, yet their default choices in culturally grounded everyday situations

local-aiarxiv-cs-cl

Efficient Leakage-Free Neural Architecture Search under Leave-One-Subject-Out Evaluation

arXiv:2609.09433v1 Announce Type: new Abstract: Leave-One-Subject-Out (LOSO) evaluation estimates generalisation performance for subject-based classification but makes Neural Architecture Search (NAS)

researcharxiv-cs-lg

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

arXiv:2603.22714v2 Announce Type: replace-cross Abstract: We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems.

Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach

arXiv:2609.07290v1 Announce Type: cross Abstract: The rapid digital transformation of manufacturing, often referred to as Industry 4.0, relies on seamless interoperability between physical and softwar

Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions

arXiv:2601.09721v2 Announce Type: replace-cross Abstract: Consumer artificial intelligence chatbots are now accessed by hundreds of millions of users seeking health information, yet systematic evaluat

What Fixed-Rollout pass@k Evaluations Can Identify

arXiv:2609.09245v1 Announce Type: cross Abstract: Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/ra

researcharxiv-cs-lg

A Quantitative Evaluation Framework for Temporal Explainability in Echocardiographic Video Segmentation

arXiv:2609.08043v1 Announce Type: new Abstract: Deep learning has achieved state-of-the-art performance in echocardiographic video segmentation, with an increasing number of models incorporating tempo

researcharxiv-cs-cv

9 Sept 2026

Evaluation of Contextual Understanding in Large Language Models

arXiv:2609.09004v1 Announce Type: cross Abstract: Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understandin

9 Sept 2026

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

arXiv:2609.03806v1 Announce Type: new Abstract: Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progres

4 Sept 2026

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis

arXiv:2609.03203v1 Announce Type: cross Abstract: Expressive speech systems make a decision before any waveform is rendered: how an utterance is delivered. In dialogue agents, narration, and role-cond

researcharxiv-cs-cl

4 Sept 2026

Applications

Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators

arXiv:2609.02304v1 Announce Type: new Abstract: A fundamental quantity in machine learning is the optimal performance achievable by any model on a given task. Estimating this quantity allows us to dis

applicationsarxiv-cs-lg

Do Better Imagined Rollouts Mean Better Robot Control? A Controlled Study of World-Model Evaluation Under Feedback

arXiv:2609.02811v1 Announce Type: new Abstract: Predictive models are increasingly used in robotics for state estimation, planning, control, and policy evaluation, yet they are often judged by open-lo

safetyarxiv-cs-ro

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

model-releasesarxiv-cs-ro

arXiv:2609.02830v1 Announce Type: new Abstract: LiDAR-based semantic segmentation is a core perception module for autonomous vehicles and mobile robots. Despite the strong performance of recent state-

Training seeds and model-selection stability in recommender-system evaluation

arXiv:2609.02499v1 Announce Type: cross Abstract: Recommender-system experiments often rely on a single random training seed, assuming that run-to-run stochasticity has limited impact on evaluation co

Disclosure-Gated User Simulation for Companion-Agent Evaluation

arXiv:2609.00982v1 Announce Type: cross Abstract: Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is exc

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

arXiv:2609.00062v1 Announce Type: cross Abstract: Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluatio

When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting

arXiv:2609.01126v1 Announce Type: new Abstract: Online adaptation can help edge time-series forecasting under distribution drift, but its measured benefit is sensitive to evaluation choices. We study

When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation

arXiv:2609.01519v1 Announce Type: new Abstract: Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits,

Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation

arXiv:2608.30373v1 Announce Type: new Abstract: Multi-Agent Debate (MAD) has been widely adopted to improve LLM-based evaluation by prompting multiple agents to negotiate and reach a consensus. Howeve

safetyarxiv-cs-cl

FORESIGHT-9: Prospective and Process-Aware Evaluation of Adaptive Trading Agents

arXiv:2608.29372v1 Announce Type: new Abstract: Retrospective backtests provide a limited test of adaptive trading agents: they cannot rule out historical contamination, expose sensitivity to a single

Applications

MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI

arXiv:2608.30940v1 Announce Type: cross Abstract: Generative music systems are increasingly presented as tools that democratize music creation, yet their practical suitability for musicians remains un

applicationsarxiv-cs-ai

Open-Set Cattle Muzzle Identification: A Leakage-Controlled Benchmark and Evaluation Protocol

model-releasesarxiv-cs-cv

arXiv:2608.28663v1 Announce Type: new Abstract: Reliable individual cattle identification supports disease surveillance, vaccination records, breeding management, and livestock insurance. Although the

Toward Cultural Alignment: Human-Centered Evaluation of Multimodal AI Stories Across Five African Communities

arXiv:2608.29209v1 Announce Type: new Abstract: In this paper, we examine how well AI-generated multimodal stories align with the lived practices, relationships, language, values, and visual expectati

safetyarxiv-cs-cl

WorldCupArena: Fine-Grained Evaluation of Language Models and Deep-Research Agents on Football Forecasting

arXiv:2607.18084v2 Announce Type: replace Abstract: Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear predictio

Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs

model-releasesarxiv-cs-cl

arXiv:2606.21359v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used to communicate and explain scientific concepts, yet their tendency to hallucinate poses significa

31 Aug 2026

Energy-Aware Performance Evaluation of Nonlinear Mechatronic Systems Under Matched-Tracking Conditions

arXiv:2608.23578v1 Announce Type: cross Abstract: Trajectory-tracking metrics such as root-mean-square error (RMSE), overshoot, and settling time are widely used to evaluate control performance in mec

researcharxiv-cs-ro

26 Aug 2026

Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmap

arXiv:2508.18646v3 Announce Type: replace Abstract: Despite their rapid advancement, large language models (LLMs) suffer from a critical disconnect between benchmark scores and real-world utility. Cur

Clinically Grounded Privacy Evaluation of Medical LMs

arXiv:2606.09590v2 Announce Type: replace Abstract: Medical language models (LMs) can memorize and reproduce protected health information, but privacy evaluations often focus on recovery of training t

researcharxiv-cs-cl

Deep Clustering Evaluation: How to Validate Internal Clustering Validation Measures

arXiv:2403.14830v2 Announce Type: replace-cross Abstract: Deep clustering partitions complex high-dimensional data using deep neural networks for clustering. It involves projecting data into lower-dim

SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation

model-releasesarxiv-cs-cl

arXiv:2608.22390v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challengin

Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge

arXiv:2608.21021v1 Announce Type: cross Abstract: Real-world fault analysis in 5G and emerging 6G networks demands domain expertise to analyze free-text diagnostics, including root-cause explanations

24 Aug 2026

No PUN Intended: Plausible Unknown Names for Person-Centred LLM Evaluation

arXiv:2608.21206v1 Announce Type: cross Abstract: Person names are widely used as prompt variables in LLM evaluations of factuality, privacy leakage, bias and abstention, but when a name's evidential