Search: “evaluation”

Graph-Structured Rubrics: Compiling Rubrics into Typed Evaluation Graphs for LLM Judges

arXiv:2608.12097v1 Announce Type: new Abstract: Rubric-based evaluators commonly treat rubrics as prompt context or flat criteria: they specify what to judge but leave criterion composition implicit,

researcharxiv-cs-ai

13 Aug 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiability

DGX agent

arXiv:2608.11238v1 Announce Type: new Abstract: Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation fr

model-releasesarxiv-cs-ai

13 Aug 2026

ASR-Roundtrip Evaluation Can Mask Context- and Convention-Dependent Reading Errors in Chinese News TTS

arXiv:2608.10606v1 Announce Type: new Abstract: ASR-roundtrip evaluation is widely used as a scalable proxy for text-to-speech (TTS) intelligibility, but it can produce false negatives for reading err

12 Aug 2026

Hardware

Neuroevolution Arena: Nested Ecological Evaluation of Update-and-Inheritance Regimes across Neural Architectures

arXiv:2608.10323v1 Announce Type: new Abstract: Competitive artificial-life systems can rank trained controllers differently under training and ecological evaluation. We present Neuroevolution Arena,

hardwarearxiv-cs-ai

12 Aug 2026

XPolicyLab: A Unified Standard and Open Ecosystem for Robot Policy Evaluation and Deployment

model-releasesarxiv-cs-ro

arXiv:2608.09892v1 Announce Type: new Abstract: Robot policy evaluation and deployment remain fragmented by model-specific software dependencies, data representations, and runtime interfaces, so that

11 Aug 2026

A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

arXiv:2606.15059v2 Announce Type: replace Abstract: Simultaneous speech-to-speech translation (SimulS2ST) enables real-time cross-lingual communication, but existing evaluation has focused largely on

safetyarxiv-cs-cl

10 Aug 2026

What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

arXiv:2608.06202v1 Announce Type: cross Abstract: Large language model (LLM) benchmark evaluations are routinely used to support claims about model safety, reliability, and deployment readiness. Yet m

7 Aug 2026

Towards End-to-End Multilingual Metaphor Processing: Integrating Detection, Translation, and Evaluation

arXiv:2608.04260v1 Announce Type: new Abstract: Metaphorical language remains a major challenge for multilingual natural language processing because successful interpretation and translation require r

6 Aug 2026

CAPEval: A Decoupled Caption Evaluation across Understanding and Generation

arXiv:2608.02589v1 Announce Type: new Abstract: Captions serve as a primary supervision signal for both multimodal understanding and text-to-image generation. However, previous evaluations treat the c

4 Aug 2026

Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics

arXiv:2608.01423v1 Announce Type: cross Abstract: Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing

safetyarxiv-cs-lg

4 Aug 2026

SVGEval: A Vision-Grounded Framework for Perceptual-Quality Benchmarking and Evaluation in Text-to-SVG Generation

arXiv:2608.01977v1 Announce Type: new Abstract: Multimodal large models are increasingly used to generate scalable vector graphics (SVG), but reliable evaluation remains underexplored. Existing protoc

4 Aug 2026

Beyond a Single Judge: Simulating Social Persona Panels for Generative UI Evaluation

arXiv:2607.28439v1 Announce Type: new Abstract: Generative UI (GenUI) lets large language models synthesize a complete, renderable interface directly from a natural-language instruction, but evaluatin

31 Jul 2026

Beyond Borrowed Histories: Person-Aligned User Simulation for Interactive Role-Playing Evaluation

arXiv:2607.27816v1 Announce Type: new Abstract: Role-playing agents (RPAs) have become one of the most important consumer applications of large language models. Users engage in multi-turn conversation

31 Jul 2026

Evaluation Protocols and Cross-Subject Generalization in EEG Emotion Recognition

arXiv:2607.27655v1 Announce Type: new Abstract: Reported accuracy in electroencephalography (EEG) emotion recognition depends on the complete evaluation procedure, not only the classifier. We separate

safetyarxiv-cs-lg

31 Jul 2026

AI's Capability in Assisting Scientific Research in Physics, Astrophysics, and Cosmology II: Project Planning and Proposal Evaluation

arXiv:2607.25881v1 Announce Type: new Abstract: We investigate how well large language models (LLMs) can assist scientific project planning and proposal evaluation. One-page project plans were indepen

CARE-MH: Towards Unified, Reproducible, and Comparable Evaluation of Mental Health LLMs

arXiv:2607.24754v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to provide mental health support, requiring reliable evaluation of safety, empathy, and therapeutic

Tutorials

On the Design and Evaluation of Human-centered Explainable AI Systems: A Systematic Review and Taxonomy

arXiv:2510.12201v2 Announce Type: replace Abstract: As AI becomes more common in everyday living, there is an increasing demand for intelligent systems that are both performant and understandable. Exp

tutorialsarxiv-cs-ai

When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops

arXiv:2607.25152v1 Announce Type: new Abstract: Long-running autonomous agents plan, act, and judge their own completion without human intervention. When an agent grades its own work, self-evaluation

ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation

arXiv:2607.23058v1 Announce Type: cross Abstract: Multilingual reasoning evaluation overwhelmingly relies on translating English benchmarks, a practice that introduces linguistic artifacts and fails t

researcharxiv-cs-ai

28 Jul 2026

LLM-Based Visual Explanation Evaluation Framework for Assessing the Explainability of Facial Skin Disease Classification Models

arXiv:2606.16794v2 Announce Type: replace Abstract: This study proposes a domain-specific LLM-based Visual Explanation Evaluation Framework for assessing visual attention explanations in facial skin d

27 Jul 2026

Synthetic and Derived Training Images for Campus Waste Detection: A Multi-Seed Evaluation with YOLOv8n

arXiv:2607.19535v2 Announce Type: new Abstract: Incorrect disposal can contaminate campus recycling streams, and a bin-mounted camera could provide feedback as an item is discarded. We evaluated wheth

researcharxiv-cs-cv

23 Jul 2026

Continuously Evolving Deepfake Detection: An Architecture and Public-Benchmark Evaluation of a Dynamic Detection System

arXiv:2607.13234v1 Announce Type: cross Abstract: Deepfake detectors that achieve near-perfect scores on academic benchmarks collapse on real-world content: recent in-the-wild evaluations report AUC d

16 Jul 2026

Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

arXiv:2607.14004v1 Announce Type: new Abstract: Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is repor

16 Jul 2026

VGIF-Score: Interpretable and Diagnostic Evaluation of Spatio-Temporal Instruction Following in Video Generation

arXiv:2607.13527v1 Announce Type: new Abstract: Recent video generation models (VGMs) have made substantial progress in visual fidelity, yet their ability to follow long, compositional instructions re

16 Jul 2026

Best-of-N TTS Evaluation is Confounded by ASR Family Alignment

arXiv:2607.08256v1 Announce Type: cross Abstract: Best-of-N (BoN) inference improves content consistency in zero-shot text-to-speech by selecting from N candidates with an automatic speech recognition

10 Jul 2026

Certified Interventional Fidelity: Anytime-Valid, Adaptive Evaluation of Causal Claims in Mechanistic Interpretability

arXiv:2607.08349v1 Announce Type: new Abstract: Mechanistic interpretability often evaluates explanations by intervening on a model: swapping hidden states, patching activations, ablating components,

researcharxiv-cs-lg

10 Jul 2026

Separating signal from noise in coding evaluations

This OpenAI article discusses methods for distinguishing meaningful performance indicators from random variation when evaluating AI coding capabilities. It likely covers evaluation methodologies, stat

model-releasesopenai

8 Jul 2026

When Users Are Happy but Agents Are Wrong: Multi-Dimensional Evaluation of Tool-Augmented Dialogue

arXiv:2510.19186v3 Announce Type: replace Abstract: Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and to

7 Jul 2026

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

arXiv:2607.00139v1 Announce Type: new Abstract: The cost of human expert evaluation is a principal bottleneck to deploying language models in specialized, high-stakes domains. This is particularly acu

Local Ai

Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training

arXiv:2607.00368v1 Announce Type: new Abstract: Large language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, ta

local-aiarxiv-cs-cl

From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape

arXiv:2606.08625v2 Announce Type: replace Abstract: As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve ac

safetyarxiv-cs-cl

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

arXiv:2607.01060v1 Announce Type: new Abstract: Video world models are emerging as a scalable alternative for evaluating generalist robot policies, bypassing the physical constraints and engineering b

safetyarxiv-cs-ro

Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives

model-releasesarxiv-cs-lg

arXiv:2606.16511v2 Announce Type: replace Abstract: Recent work motivates moving large language model (LLM) evaluation from mean-based to tail-aware metrics, including conditional value-at-risk and ta

Reference-Based Prosody and Rhythm Evaluation for Spoken Dialogue Systems

arXiv:2606.31055v1 Announce Type: new Abstract: Speech-to-speech (S2S) AI agents are advancing rapidly, yet evaluation lacks interpretable speech-native measures for conversational prosody and rhythm.

1 Jul 2026

Agents

SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

arXiv:2504.07385v3 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluati

agentsarxiv-cs-ai

1 Jul 2026

Data and Evaluation Closed-Loop for Model Capability Enhancement

arXiv:2606.28471v1 Announce Type: new Abstract: Model capability is the central variable in LLM pre-training, yet is never observed directly: data shapes it prospectively, while evaluation reveals it

30 Jun 2026

A Robust Model-Based Approach for Continuous-Time Policy Evaluation with Unknown Levy Process Dynamics

arXiv:2504.01482v3 Announce Type: replace-cross Abstract: This paper develops a model-based framework for continuous-time policy evaluation (CTPE) in reinforcement learning, incorporating both Brownia

safetyarxiv-cs-lg

DiffusionBench: On Holistic Evaluation of Diffusion Transformers

arXiv:2606.24888v1 Announce Type: new Abstract: Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While met

IPO Finance Agent: Evaluation of LLM Financial Analysts beyond Finance Agent v2, with Automated Rubric Generation -- the Case of the SpaceX (SPCX) IPO

arXiv:2606.23032v2 Announce Type: replace Abstract: Finance Agent v2 (by Vals AI) has emerged as the reference benchmark for evaluating both Anthropic Claude and OpenAI ChatGPT frontier language model

On the Stability of Prompt Ranking in Large Language Model Evaluation

arXiv:2606.24381v1 Announce Type: cross Abstract: Prompt-based interaction has become a dominant paradigm for using large language models (LLMs), where multiple candidate prompts are evaluated and the

Agent Skill Evaluation and Evolution: Frameworks and Benchmarks

arXiv:2606.11435v1 Announce Type: new Abstract: The growth of agent skills has transformed how agentic systems are built, evaluated, and deployed. As skill libraries continue to scale, rigorous evalua

Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluation

arXiv:2601.07506v2 Announce Type: replace Abstract: While large language models (LLMs) are increasingly used as automatic judges for question answering (QA) and other reference-conditioned evaluation

Language Shapes Mental Health Evaluations in Large Language Models

arXiv:2603.06910v2 Announce Type: replace Abstract: Multilingual large language models (LLMs) are increasingly used in socially sensitive mental health contexts, including support chatbots, screening,

Applications

LaQual: An Automated Framework for LLM App Quality Evaluation

arXiv:2508.18636v2 Announce Type: replace-cross Abstract: Representing a new paradigm in software distribution, LLM app stores are rapidly emerging, offering users diverse choices for content generati

applicationsarxiv-cs-ai

PoQ-Judge: A Multi-Architecture Evaluation Framework for Cost-Aware Proof-of-Quality in Decentralized LLM Inference

arXiv:2606.11196v1 Announce Type: cross Abstract: Decentralized LLM inference networks need lightweight, reference-free quality evaluation for Proof of Quality (PoQ). We present PoQ-Judge, a framework

researcharxiv-cs-ai

Risk Under Pressure: Compute-Aware Evaluation of Adversarial Robustness in Language Models

arXiv:2606.11409v1 Announce Type: cross Abstract: Adversarial robustness evaluations of large language models (LLMs) typically report attack success rate (ASR) under fixed query budgets, implicitly tr

A Practical Recipe Towards Improving Sim-and-Real Correlation for VLA Evaluation

arXiv:2606.10366v1 Announce Type: cross Abstract: Simulation has become an essential tool for evaluating and improving vision-language-action (VLA) policies, offering scalable, reproducible, and contr

10 Jun 2026

DeRA-MOS: Optimizing Text-to-Music Evaluation via Decoupled Listwise Ranking and Modality Alignment

arXiv:2606.10010v1 Announce Type: cross Abstract: Evaluating text-to-music (TTM) systems remains expensive because music impression (MI) and text alignment (TA) scores rely on human mean opinion score