Search: “evaluation”

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and u

24 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Validation and Simulation Catch Different Errors: Four Levels of Evaluation for LLM-Generated Circuits

DGX agent

arXiv:2609.26830v1 Announce Type: cross Abstract: Simulation success is not equivalent to structural correctness for LLM-generated circuits. We define and measure four evaluation levels -- schema vali

model-releasesarxiv-cs-ai

24 Sept 2026

Beyond Endpoint Performance: Process-Level Evaluation of Self-Evolving Agents

arXiv:2609.24663v1 Announce Type: new Abstract: Self-evolving agents convert interaction feedback into persistent artifacts, such as memories or skills, which in turn guide subsequent decisions. As th

CivBench: Progress-Based Evaluation for LLMs' Strategic Decision-Making in Civilization V

arXiv:2604.07733v2 Announce Type: replace Abstract: Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide

Context-Aware Pre-Deployment Evaluation of AI Systems: A Regulatory Framework for Nigerian Fintech

arXiv:2609.24016v1 Announce Type: new Abstract: Commercial large language models are increasingly deployed across African fintech infrastructure for fraud detection and customer communication, yet no

Exposing Blind Spots in Deep Imbalanced Regression Evaluation

arXiv:2609.25152v1 Announce Type: new Abstract: Deep Imbalanced Regression (DIR) addresses a common failure mode of regression models: target distributions are highly non-uniform, causing models to pe

Applications

Faithful Faithfulness Evaluations: Challenges & Pitfalls Learned from a Breast MRI Case Study

arXiv:2609.25978v1 Announce Type: cross Abstract: Saliency maps are widely used to explain deep learning predictions in medical imaging, yet visually plausible explanations do not necessarily reflect

applicationsarxiv-cs-lg

IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Models

model-releasesarxiv-cs-ro

arXiv:2609.25562v1 Announce Type: new Abstract: Open robot policies increasingly follow two paradigms: vision-language-action models (VLAs) directly map observations and instructions to actions, where

Quantifying Protocol-Induced Uncertainty in Comparative Predictive-Model Evaluation: Evidence from Large-Scale Daily PM10 Forecasting

arXiv:2609.26288v1 Announce Type: new Abstract: Comparative studies of predictive models often end by ranking candidate models, yet these rankings depend on evaluation protocols whose influence is rar

A Course Intelligence Platform for Higher Education: Lessons from AI-Assisted Course Evaluation

arXiv:2411.02455v3 Announce Type: replace Abstract: The rapid adoption of generative AI has created new opportunities for teaching, learning, and quality assurance. Existing applications, however, rem

safetyarxiv-cs-cl

Beyond the Stitching Assumption: A Unified Framework for Multimodal Synthetic Data Evaluation via Semantic Quantization

arXiv:2609.22149v1 Announce Type: new Abstract: Multimodal synthetic datasets combine structured attributes with free text, but are often evaluated separately. Such metrics can remain high after tabul

researcharxiv-cs-cl

DeepInstructor: An Agentic AI Instructor for Experience-Driven Idea Evaluation

arXiv:2609.22104v1 Announce Type: new Abstract: As automated scientific discovery advances, Large Language Models (LLMs) can now generate research ideas at an unprecedented scale, shifting the bottlen

safetyarxiv-cs-cl

Rank Portability Does Not Imply Feasibility Portability: Target-Specific Evaluation of Joint Hardware Constraints

arXiv:2609.22122v1 Announce Type: new Abstract: Cross-device hardware evaluation often assumes that if architecture rankings transfer across devices, a proxy device can support target-side model selec

Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

arXiv:2603.13099v3 Announce Type: replace Abstract: We introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability, and Logic), a diagnostic benchmark with 6,372 instances that evaluates multim

21 Sept 2026

From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluation

arXiv:2609.19413v1 Announce Type: cross Abstract: Robot manipulation policies are improving quickly, and real-robot evaluation remains the standard evidence for that progress. It still relies on a hum

18 Sept 2026

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

arXiv:2609.20758v1 Announce Type: cross Abstract: Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in

18 Sept 2026

Local Ai

Selective Cotton Boll Localization for Robotic Harvesting: Evaluation of Deep Learning Vision Models Under Field Conditions

arXiv:2609.19592v1 Announce Type: new Abstract: This study developed and evaluated a deep-learning-based perception framework for selective robotic cotton picking. The dataset contained 1,008 annotate

local-aiarxiv-cs-cv

18 Sept 2026

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

arXiv:2609.18052v1 Announce Type: cross Abstract: This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specifica

I code or AI code: A comparative evaluation of AI-rated scores in classroom observations

arXiv:2609.18274v1 Announce Type: cross Abstract: Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet t

Reporting Practice Matters: The Impact of Reference Choice on Chest X-ray Report Evaluation

arXiv:2609.19093v1 Announce Type: cross Abstract: Radiologists follow heterogeneous reporting practices. Two radiologists examining the same image and identifying the same clinical findings might neve

researcharxiv-cs-ai

RoboVAD: A Large Cross-Domain Evaluation Benchmark for Anomaly Detection in Robotic Arm Manipulation Videos

arXiv:2609.17843v1 Announce Type: cross Abstract: Video anomaly detection (VAD) is an actively studied task, having wide applications in typical scenarios such as public surveillance and road traffic

Empirical Evaluation of Open-Source Large Language Models for Retrieval-Augmented Generation in ESG Domain

arXiv:2609.15242v1 Announce Type: new Abstract: Environmental, Social, and Governance (ESG) reporting is critical for corporate accountability, with Large Language Models (LLMs) and Retrieval-Augmente

Empirical Evaluation of Task-Based Permission Scoping Architecture for AI Agents

arXiv:2609.15422v1 Announce Type: new Abstract: AI agents are provisioned the same as employee-owned hosts in many enterprise settings with a static credential set fixed at deployment which includes a

One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs

arXiv:2609.14860v1 Announce Type: cross Abstract: Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benc

Who Benchmarks the Benchmarks? Towards Comprehensive Evaluation of Commonsense Reasoning Benchmarks

model-releasesarxiv-cs-cl

arXiv:2504.07825v2 Announce Type: replace Abstract: Commonsense reasoning is a key language model capability, as it is purportedly a prerequisite for many basic tasks, unlike specific factual knowledg

A Comparative Evaluation of Pre-trained Convolutional Neural Networks for Melanoma Detection

arXiv:2609.11550v1 Announce Type: new Abstract: Early diagnosis of melanoma is critical for improving patient survival rates. However, accurately distinguishing melanoma from other skin lesions remain

researcharxiv-cs-cv

11 Sept 2026

Complex-Text Robustness Evaluation and Failure Diagnosis for Low-Resource Multilingual Text-to-Speech

arXiv:2609.11545v1 Announce Type: new Abstract: Low-resource multilingual text-to-speech (TTS) systems have expanded language coverage, but their robustness under complex text inputs remains insuffici

researcharxiv-cs-cl

11 Sept 2026

Sequence-Informed Geometric Evaluation of RNA 3D Structures

arXiv:2609.10644v1 Announce Type: cross Abstract: Computational RNA structure pipelines generate many candidate conformations for the same sequence. Reliable evaluation therefore requires more than re

safetyarxiv-cs-lg

11 Sept 2026

A Systematic Evaluation of Molecule Generation Models for De Novo Drug Design: From Benchmarks to Practical Insights

arXiv:2609.10099v1 Announce Type: new Abstract: Molecule generation has emerged as a powerful computational tool for de novo drug design, enabling the exploration of chemical space beyond the limits o

Agents

APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents

arXiv:2609.07712v1 Announce Type: new Abstract: Mobile GUI agents can execute tasks from natural-language instructions, but their evaluation remains difficult to make both realistic and reproducible.

agentsarxiv-cs-ai

Local Ai

DiSCo: A Distribution-First Steering and Cultural Prior Evaluation Framework for Measuring Cultural Preference Bias in LLMs

arXiv:2609.10253v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in globally used assistants, yet their default choices in culturally grounded everyday situations

local-aiarxiv-cs-cl

Efficient Leakage-Free Neural Architecture Search under Leave-One-Subject-Out Evaluation

arXiv:2609.09433v1 Announce Type: new Abstract: Leave-One-Subject-Out (LOSO) evaluation estimates generalisation performance for subject-based classification but makes Neural Architecture Search (NAS)

PopResume: Causal Fairness Evaluation of LLM/VLM Resume Screeners with Population-Representative Dataset

arXiv:2603.22714v2 Announce Type: replace-cross Abstract: We present PopResume, a population-representative resume dataset for causal fairness auditing of LLM- and VLM-based resume screening systems.

Quality Metrics for LLM-Generated Asset Administration Shells: A Perturbation-Based Evaluation Approach

arXiv:2609.07290v1 Announce Type: cross Abstract: The rapid digital transformation of manufacturing, often referred to as Industry 4.0, relies on seamless interoperability between physical and softwar

Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions

arXiv:2601.09721v2 Announce Type: replace-cross Abstract: Consumer artificial intelligence chatbots are now accessed by hundreds of millions of users seeking health information, yet systematic evaluat

What Fixed-Rollout pass@k Evaluations Can Identify

arXiv:2609.09245v1 Announce Type: cross Abstract: Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/ra

A Quantitative Evaluation Framework for Temporal Explainability in Echocardiographic Video Segmentation

arXiv:2609.08043v1 Announce Type: new Abstract: Deep learning has achieved state-of-the-art performance in echocardiographic video segmentation, with an increasing number of models incorporating tempo

researcharxiv-cs-cv

9 Sept 2026

Evaluation of Contextual Understanding in Large Language Models

arXiv:2609.09004v1 Announce Type: cross Abstract: Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understandin

9 Sept 2026

SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation

arXiv:2609.03806v1 Announce Type: new Abstract: Scalable Vector Graphics (SVG) generation is attracting increasing attention as generative models improve in expressiveness and controllability. Progres

4 Sept 2026

VoxReason: Listener-Free Evaluation of Source-Grounded Speech Planning Before Synthesis

arXiv:2609.03203v1 Announce Type: cross Abstract: Expressive speech systems make a decision before any waveform is rendered: how an utterance is delivered. In dialogue agents, narration, and role-cond

researcharxiv-cs-cl

4 Sept 2026

Applications

Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators

arXiv:2609.02304v1 Announce Type: new Abstract: A fundamental quantity in machine learning is the optimal performance achievable by any model on a given task. Estimating this quantity allows us to dis

applicationsarxiv-cs-lg

Do Better Imagined Rollouts Mean Better Robot Control? A Controlled Study of World-Model Evaluation Under Feedback

arXiv:2609.02811v1 Announce Type: new Abstract: Predictive models are increasingly used in robotics for state estimation, planning, control, and policy evaluation, yet they are often judged by open-lo

safetyarxiv-cs-ro

Toward Robust LiDAR Semantic Segmentation for Real-World Deployment: Evaluation under Coarse Labels, Adverse Conditions, and Domain Shifts

model-releasesarxiv-cs-ro

arXiv:2609.02830v1 Announce Type: new Abstract: LiDAR-based semantic segmentation is a core perception module for autonomous vehicles and mobile robots. Despite the strong performance of recent state-

Training seeds and model-selection stability in recommender-system evaluation

arXiv:2609.02499v1 Announce Type: cross Abstract: Recommender-system experiments often rely on a single random training seed, assuming that run-to-run stochasticity has limited impact on evaluation co

Disclosure-Gated User Simulation for Companion-Agent Evaluation

arXiv:2609.00982v1 Announce Type: cross Abstract: Using a large language model to play the user is now standard in scalable evaluation. It has a repeatedly diagnosed failure: the simulated user is exc

RePro: Proof-Verified Benchmark Rewriting for Reliable Evaluation of LLM Mathematical Problem Solving

arXiv:2609.00062v1 Announce Type: cross Abstract: Data contamination undermines the reliable evaluation of large language models (LLMs) on mathematical problem solving. While rewriting-based evaluatio

When Does Online Adaptation Pay on the Edge? A Leakage-Free Evaluation of Warmup, Learning-Rate Selection, and Resource Trade-offs for Time-Series Forecasting

arXiv:2609.01126v1 Announce Type: new Abstract: Online adaptation can help edge time-series forecasting under distribution drift, but its measured benefit is sensitive to evaluation choices. We study

When Guardrails Look Effective: Construct Validity Failures in LLM Agent Commerce Evaluation

arXiv:2609.01519v1 Announce Type: new Abstract: Interactive simulations increasingly evaluate policies in markets populated by language-model agents. Their outputs can look economic---prices, profits,