Search: “evaluation”

'This Wasn't Made for Me': Recentering User Experience and Emotional Impact in the Evaluation of ASR Bias

arXiv:2604.21148v1 Announce Type: new Abstract: Studies on bias in Automatic Speech Recognition (ASR) tend to focus on reporting error rates for speakers of underrepresented dialects, yet less researc

24 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation

DGX agent

arXiv:2506.21095v4 Announce Type: replace-cross Abstract: Federated Learning (FL) enables collaborative training while preserving privacy, yet it introduces a critical challenge: the 'illusion of fair

model-releasesarxiv-cs-ai

23 Apr 2026

HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

arXiv:2604.19071v1 Announce Type: new Abstract: Evaluating the writing capabilities of large language models (LLMs) remains a significant challenge due to the multidimensional nature of writing skills

22 Apr 2026

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

arXiv:2604.19047v1 Announce Type: cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate o

22 Apr 2026

A Comparative Evaluation of Geometric Accuracy in NeRF and Gaussian Splatting

model-releasesarxiv-cs-cv

arXiv:2604.18205v1 Announce Type: new Abstract: Recent advances in neural rendering have introduced numerous 3D scene representations. Although standard computer vision metrics evaluate the visual qua

Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

model-releasesarxiv-cs-cv

arXiv:2510.07143v3 Announce Type: replace Abstract: Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectivene

Distributional Off-Policy Evaluation with Deep Quantile Process Regression

arXiv:2604.18143v1 Announce Type: cross Abstract: This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of t

Multilingual Training and Evaluation Resources for Vision-Language Models

arXiv:2604.18347v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on Englis

UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

arXiv:2602.09130v3 Announce Type: replace Abstract: Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning an

VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis

arXiv:2509.16538v3 Announce Type: replace-cross Abstract: We propose VC-Inspector, a lightweight, open-source large multimodal model (LMM) for reference-free evaluation of video captions, with a focus

VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

arXiv:2604.17248v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing sp

Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation

arXiv:2507.18756v2 Announce Type: replace Abstract: Multi-Armed Bandit (MAB) algorithms are widely used in recommender systems that require continuous, incremental learning. A core aspect of MABs is t

20 Apr 2026

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

arXiv:2603.04738v2 Announce Type: replace Abstract: Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback f

17 Apr 2026

MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios

arXiv:2604.14158v1 Announce Type: new Abstract: Current evaluations of long-term memory in LLMs are fundamentally static. By fixating on simple retrieval and short-context inference, they neglect the

17 Apr 2026

Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

arXiv:2604.13804v1 Announce Type: new Abstract: The rapid evolution of multimodal large models has revolutionized the simulation of diverse characters in speech dialogue systems, enabling a novel inte

16 Apr 2026

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

arXiv:2604.13776v1 Announce Type: cross Abstract: Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infrastructure

16 Apr 2026

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

arXiv:2604.12191v1 Announce Type: new Abstract: Current evaluations of large language models aggregate performance across diverse tasks into single scores. This obscures fine-grained ability variation

15 Apr 2026

Agents

Defining and Evaluation Method for External Human-Machine Interfaces

arXiv:2604.12293v1 Announce Type: new Abstract: As the number of fatalities involving Autonomous Vehicles increase, the need for a universal method of communicating between vehicles and other agents o

agentsarxiv-cs-ro

15 Apr 2026

GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees

arXiv:2604.12757v1 Announce Type: cross Abstract: Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expe

15 Apr 2026

Decomposing and Reducing Hidden Measurement Error in LLM Evaluation Pipelines

arXiv:2604.11581v1 Announce Type: new Abstract: LLM evaluations drive which models get deployed, which safety standards get adopted, and which research conclusions get published. Yet these scores carr

Detection Is Cheap, Routing Is Learned: Why Refusal-Based Alignment Evaluation Fails

arXiv:2603.18280v2 Announce Type: replace-cross Abstract: Current alignment evaluation mostly measures whether models encode dangerous concepts and whether they refuse harmful requests. Both miss the

safetyarxiv-cs-ai

GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models

arXiv:2604.10385v1 Announce Type: new Abstract: Generating complex multi-actor scenario videos remains difficult even for state-of-the-art neural generators, while evaluating them is hard due to the l

safetyarxiv-cs-cv

SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

arXiv:2505.17012v3 Announce Type: replace-cross Abstract: Existing evaluations of multimodal large language models (MLLMs) on spatial intelligence are typically fragmented and limited in scope. In thi

YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

arXiv:2604.10968v1 Announce Type: new Abstract: Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academi

Tutorials

Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym

arXiv:2604.09338v1 Announce Type: new Abstract: Spatial reasoning is central to navigation and robotics, yet measuring model capabilities on these tasks remains difficult. Existing benchmarks evaluate

tutorialsarxiv-cs-ai

13 Apr 2026

Neural networks for Text-to-Speech evaluation

arXiv:2604.08562v1 Announce Type: cross Abstract: Ensuring that Text-to-Speech (TTS) systems deliver human-perceived quality at scale is a central challenge for modern speech technologies. Human subje

13 Apr 2026

On Semiotic-Grounded Interpretive Evaluation of Generative Art

arXiv:2604.08641v1 Announce Type: cross Abstract: Interpretation is essential to deciphering the language of art: audiences communicate with artists by recovering meaning from visual artifacts. Howeve

13 Apr 2026

AtomEval: Atomic Evaluation of Adversarial Claims in Fact Verification

arXiv:2604.07967v1 Announce Type: new Abstract: Adversarial claim rewriting is widely used to test fact-checking systems, but standard metrics fail to capture truth-conditional consistency and often l

10 Apr 2026

CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

arXiv:2604.07583v1 Announce Type: new Abstract: Real-world categorization is severely hampered by class imbalance because traditional ensembles favor majority classes, which lowers minority performanc

10 Apr 2026

VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

model-releasesarxiv-cs-cv

arXiv:2604.07634v1 Announce Type: new Abstract: Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core

10 Apr 2026

Quantum Versus Classical Machine Learning for Neuropsychiatric Disorders: A Feature-Matched Multi-Cohort Evaluation

arXiv:2610.21344v1 Announce Type: new Abstract: Resting-state functional connectivity is widely used in machine learning-based classification of neuropsychiatric disorders. Quantum machine learning mo

16 Oct 2026

Agents

Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems

arXiv:2610.15529v2 Announce Type: new Abstract: Agent-to-agent protocols allow LLM agents to discover peers, exchange results, and coordinate tool use. A locally successful protocol attack, however, m

agentsarxiv-cs-ai

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial

MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

arXiv:2610.19827v1 Announce Type: cross Abstract: LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their

Memory Echo in Single-Agent Long-Term Memory: Lifecycle Evaluation and Provenance-Guided Mitigation

arXiv:2610.17763v1 Announce Type: new Abstract: Long-term memory supports agents across sessions. Repeated retrieval and write-back can let one historical claim occupy multiple retrieval positions and

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

model-releasesarxiv-cs-lg

arXiv:2506.00482v2 Announce Type: replace Abstract: LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills

C2Flight: A Closed-Loop Co-Simulation Framework for Early-Stage Evaluation of Heterogeneous UAV Computing Systems

arXiv:2610.14715v1 Announce Type: new Abstract: Early-stage design of heterogeneous UAV computing systems requires evaluating how algorithm, hardware, communication, and system topology affect closed-

researcharxiv-cs-ro

Frozen Site-Probe Accuracy Is Not a Standalone Harmonization Score: An Audit of Evaluation Practice for Multi-Site MRI

arXiv:2610.14140v1 Announce Type: cross Abstract: Image-level MRI harmonization is often evaluated with a frozen site probe, a site classifier trained on raw scans and applied unchanged to harmonized

safetyarxiv-cs-cv

IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

model-releasesarxiv-cs-lg

arXiv:2610.16652v1 Announce Type: new Abstract: We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to estab

Local Ai

Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation

arXiv:2610.10974v1 Announce Type: cross Abstract: Off-policy evaluation (OPE) estimates the value of a target policy from logged data, but limited behavior-policy coverage can force high-variance rewe

local-aiarxiv-cs-ai

9 Oct 2026

Applications

LLM Persuasion Is in the Eye of the Evaluation

arXiv:2610.10232v1 Announce Type: new Abstract: Large language models (LLMs) have already been shown to match or exceed human experts in persuasion. While their persuasive capabilities hold promise fo

applicationsarxiv-cs-cl

8 Oct 2026

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when u

8 Oct 2026

Applications

Work While They Sleep: Exploiting Evaluation Latency for Fully Bayesian Optimization

arXiv:2610.08969v1 Announce Type: new Abstract: Black-box optimization problems are ubiquitous across science and engineering, often dealing with expensive objective functions. This objective latency

applicationsarxiv-cs-lg

8 Oct 2026

CHARTER: Auditing Reference Substitution in Hierarchical Compact-Evidence Evaluation for Computational Pathology

arXiv:2610.07843v1 Announce Type: new Abstract: In digital pathology, compact evidence is often used to explain or audit predictions made by whole-slide image multiple instance learning models. In hie

researcharxiv-cs-cv

Conversation Is a Two-Body Problem: Dyadic Evaluation of Full-Duplex Dialogue Models

arXiv:2610.08125v1 Announce Type: cross Abstract: Full-duplex spoken dialogue models listen and speak at the same time, enabling voice agents to have natural, low-latency interactions that turn-based

Evaluation of Active Feature Acquisition Policies with Tabular Foundation Models

arXiv:2610.07406v1 Announce Type: new Abstract: Active feature acquisition learns policies that sequentially acquire features to maximize information about a target variable. We study how to learn and

Joint upper-bound coverage and route-choice utility: an empirical evaluation on two urban proxy tasks

arXiv:2610.06995v1 Announce Type: new Abstract: Whether more accurate traffic forecasts or higher uncertainty coverage improve route decisions is unclear. We evaluate this question with a frozen proto

researcharxiv-cs-ai

Protocol-Sensitive Evaluation of Log Anomaly Detection: Component Costs and Target-Access Sensitivity on HDFS and BGL