Search: “evaluation”

V-MAGE: A Game Evaluation Framework for Assessing Vision-Centric Capabilities in Multimodal Large Language Models

arXiv:2504.06148v3 Announce Type: replace Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in visual-text processing. However, existi

researcharxiv-cs-cv

27 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Industry

HealthBench Professional, our new evaluation of clinician chat tasks, is now available on HuggingFace for easy access. Each example was writ…

DGX agent

HealthBench Professional, our new evaluation of clinician chat tasks, is now available on HuggingFace for easy access. Each example was written, reviewed, and adjudicated by three or more physicians.

industryclem-delangue--x

26 Apr 2026

Tutorials

Design, Modelling and Experimental Evaluation of a Tendon-driven Wrist Abduction-Adduction Mechanism for an upper limb exoskeleton

arXiv:2604.20893v1 Announce Type: new Abstract: Wrist exoskeletons play a vital role in rehabilitation and assistive applications, yet conventional actuation mechanisms such as electric motors or pneu

tutorialsarxiv-cs-ro

Applications

LAF-Based Evaluation and UTTL-Based Learning Strategies with MIATTs

arXiv:2604.20944v1 Announce Type: cross Abstract: In many real-world machine learning (ML) applications, the true target cannot be precisely defined due to ambiguity or subjectivity information. To ad

applicationsarxiv-cs-ai

Temporal Taskification in Streaming Continual Learning: A Source of Evaluation Instability

model-releasesarxiv-cs-lg

arXiv:2604.21930v1 Announce Type: new Abstract: Streaming Continual Learning (CL) typically converts a continuous stream into a sequence of discrete tasks through temporal partitioning. We argue that

'This Wasn't Made for Me': Recentering User Experience and Emotional Impact in the Evaluation of ASR Bias

arXiv:2604.21148v1 Announce Type: new Abstract: Studies on bias in Automatic Speech Recognition (ASR) tend to focus on reporting error rates for speakers of underrepresented dialects, yet less researc

FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation

arXiv:2506.21095v4 Announce Type: replace-cross Abstract: Federated Learning (FL) enables collaborative training while preserving privacy, yet it introduces a critical challenge: the 'illusion of fair

23 Apr 2026

HoWToBench: Holistic Evaluation for LLM's Capability in Human-level Writing using Tree of Writing

arXiv:2604.19071v1 Announce Type: new Abstract: Evaluating the writing capabilities of large language models (LLMs) remains a significant challenge due to the multidimensional nature of writing skills

22 Apr 2026

RARE: Redundancy-Aware Retrieval Evaluation Framework for High-Similarity Corpora

arXiv:2604.19047v1 Announce Type: cross Abstract: Existing QA benchmarks typically assume distinct documents with minimal overlap, yet real-world retrieval-augmented generation (RAG) systems operate o

22 Apr 2026

A Comparative Evaluation of Geometric Accuracy in NeRF and Gaussian Splatting

model-releasesarxiv-cs-cv

arXiv:2604.18205v1 Announce Type: new Abstract: Recent advances in neural rendering have introduced numerous 3D scene representations. Although standard computer vision metrics evaluate the visual qua

Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

model-releasesarxiv-cs-cv

arXiv:2510.07143v3 Announce Type: replace Abstract: Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectivene

Distributional Off-Policy Evaluation with Deep Quantile Process Regression

arXiv:2604.18143v1 Announce Type: cross Abstract: This paper investigates the off-policy evaluation (OPE) problem from a distributional perspective. Rather than focusing solely on the expectation of t

Multilingual Training and Evaluation Resources for Vision-Language Models

arXiv:2604.18347v1 Announce Type: new Abstract: Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on Englis

UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation

arXiv:2602.09130v3 Announce Type: replace Abstract: Model compression is increasingly essential for deploying large language models (LLMs), yet existing comparative studies largely focus on pruning an

VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis

arXiv:2509.16538v3 Announce Type: replace-cross Abstract: We propose VC-Inspector, a lightweight, open-source large multimodal model (LMM) for reference-free evaluation of video captions, with a focus

VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech

arXiv:2604.17248v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) are increasingly integrated into daily applications, yet their generative biases remain underexplored. Existing sp

Exploitation Over Exploration: Unmasking the Bias in Linear Bandit Recommender Offline Evaluation

arXiv:2507.18756v2 Announce Type: replace Abstract: Multi-Armed Bandit (MAB) algorithms are widely used in recommender systems that require continuous, incremental learning. A core aspect of MABs is t

20 Apr 2026

IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation

arXiv:2603.04738v2 Announce Type: replace Abstract: Instruction-following is a foundational capability of large language models (LLMs), with its improvement hinging on scalable and accurate feedback f

17 Apr 2026

MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarios

arXiv:2604.14158v1 Announce Type: new Abstract: Current evaluations of long-term memory in LLMs are fundamentally static. By fixating on simple retrieval and short-context inference, they neglect the

17 Apr 2026

Character Beyond Speech: Leveraging Role-Playing Evaluation in Audio Large Language Models via Reinforcement Learning

arXiv:2604.13804v1 Announce Type: new Abstract: The rapid evolution of multimodal large models has revolutionized the simulation of diverse characters in speech dialogue systems, enabling a novel inte

16 Apr 2026

Who Gets Flagged? The Pluralistic Evaluation Gap in AI Content Watermarking

arXiv:2604.13776v1 Announce Type: cross Abstract: Watermarking is becoming the default mechanism for AI content authentication, with governance policies and frameworks referencing it as infrastructure

16 Apr 2026

Beyond Scores: Diagnostic LLM Evaluation via Fine-Grained Abilities

arXiv:2604.12191v1 Announce Type: new Abstract: Current evaluations of large language models aggregate performance across diverse tasks into single scores. This obscures fine-grained ability variation

15 Apr 2026

Agents

Defining and Evaluation Method for External Human-Machine Interfaces

arXiv:2604.12293v1 Announce Type: new Abstract: As the number of fatalities involving Autonomous Vehicles increase, the need for a universal method of communicating between vehicles and other agents o

agentsarxiv-cs-ro

15 Apr 2026

GF-Score: Certified Class-Conditional Robustness Evaluation with Fairness Guarantees

arXiv:2604.12757v1 Announce Type: cross Abstract: Adversarial robustness is essential for deploying neural networks in safety-critical applications, yet standard evaluation methods either require expe

15 Apr 2026

Decomposing and Reducing Hidden Measurement Error in LLM Evaluation Pipelines

arXiv:2604.11581v1 Announce Type: new Abstract: LLM evaluations drive which models get deployed, which safety standards get adopted, and which research conclusions get published. Yet these scores carr

Detection Is Cheap, Routing Is Learned: Why Refusal-Based Alignment Evaluation Fails

arXiv:2603.18280v2 Announce Type: replace-cross Abstract: Current alignment evaluation mostly measures whether models encode dangerous concepts and whether they refuse harmful requests. Both miss the

safetyarxiv-cs-ai

GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models

arXiv:2604.10385v1 Announce Type: new Abstract: Generating complex multi-actor scenario videos remains difficult even for state-of-the-art neural generators, while evaluating them is hard due to the l

safetyarxiv-cs-cv

SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

arXiv:2505.17012v3 Announce Type: replace-cross Abstract: Existing evaluations of multimodal large language models (MLLMs) on spatial intelligence are typically fragmented and limited in scope. In thi

YIELD: A Large-Scale Dataset and Evaluation Framework for Information Elicitation Agents

arXiv:2604.10968v1 Announce Type: new Abstract: Most conversational agents (CAs) are designed to satisfy user needs through user-driven interactions. However, many real-world settings, such as academi

Tutorials

Mind the Gap Between Spatial Reasoning and Acting! Step-by-Step Evaluation of Agents With Spatial-Gym

arXiv:2604.09338v1 Announce Type: new Abstract: Spatial reasoning is central to navigation and robotics, yet measuring model capabilities on these tasks remains difficult. Existing benchmarks evaluate

tutorialsarxiv-cs-ai

Neural networks for Text-to-Speech evaluation

arXiv:2604.08562v1 Announce Type: cross Abstract: Ensuring that Text-to-Speech (TTS) systems deliver human-perceived quality at scale is a central challenge for modern speech technologies. Human subje

On Semiotic-Grounded Interpretive Evaluation of Generative Art

arXiv:2604.08641v1 Announce Type: cross Abstract: Interpretation is essential to deciphering the language of art: audiences communicate with artists by recovering meaning from visual artifacts. Howeve

Very interesting evaluation from the UK’s AI Security Institute of the not yet publicly available Claude Mythos Preview. On the happy side, …

model-releasesgary-marcus--x

Very interesting evaluation from the UK’s AI Security Institute of the not yet publicly available Claude Mythos Preview. On the happy side, in its current form, Myth is nowhere near as scary as Tom Fr

We conducted cyber evaluations of Claude Mythos Preview and found that it is the first model to complete an AISI cyber range end-to-end. 🧵

model-releasesboris-cherny--x

Boris Cherny and colleagues conducted cybersecurity evaluations of Claude Mythos Preview, finding it to be the first AI model to successfully complete an AISI (AI Safety Institute) cyber range end-to-

AtomEval: Atomic Evaluation of Adversarial Claims in Fact Verification

arXiv:2604.07967v1 Announce Type: new Abstract: Adversarial claim rewriting is widely used to test fact-checking systems, but standard metrics fail to capture truth-conditional consistency and often l

10 Apr 2026

CAMO: A Class-Aware Minority-Optimized Ensemble for Robust Language Model Evaluation on Imbalanced Data

arXiv:2604.07583v1 Announce Type: new Abstract: Real-world categorization is severely hampered by class imbalance because traditional ensembles favor majority classes, which lowers minority performanc

10 Apr 2026

VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

model-releasesarxiv-cs-cv

arXiv:2604.07634v1 Announce Type: new Abstract: Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core

10 Apr 2026

Quantum Versus Classical Machine Learning for Neuropsychiatric Disorders: A Feature-Matched Multi-Cohort Evaluation

arXiv:2610.21344v1 Announce Type: new Abstract: Resting-state functional connectivity is widely used in machine learning-based classification of neuropsychiatric disorders. Quantum machine learning mo

16 Oct 2026

Agents

Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems

arXiv:2610.15529v2 Announce Type: new Abstract: Agent-to-agent protocols allow LLM agents to discover peers, exchange results, and coordinate tool use. A locally successful protocol attack, however, m

agentsarxiv-cs-ai

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial

MAPLE: A Multi-Axis Parametric Knowledge Extraction Lens for LLM Evaluation

arXiv:2610.19827v1 Announce Type: cross Abstract: LLMs absorb vast knowledge from training data, yet their inherently probabilistic nature offers no guarantee that this knowledge will surface in their

Memory Echo in Single-Agent Long-Term Memory: Lifecycle Evaluation and Provenance-Guided Mitigation

arXiv:2610.17763v1 Announce Type: new Abstract: Long-term memory supports agents across sessions. Repeated retrieval and write-back can let one historical claim occupy multiple retrieval positions and

Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub

model-releasesarxiv-cs-lg

arXiv:2506.00482v2 Announce Type: replace Abstract: LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills

C2Flight: A Closed-Loop Co-Simulation Framework for Early-Stage Evaluation of Heterogeneous UAV Computing Systems

arXiv:2610.14715v1 Announce Type: new Abstract: Early-stage design of heterogeneous UAV computing systems requires evaluating how algorithm, hardware, communication, and system topology affect closed-

researcharxiv-cs-ro

Frozen Site-Probe Accuracy Is Not a Standalone Harmonization Score: An Audit of Evaluation Practice for Multi-Site MRI

arXiv:2610.14140v1 Announce Type: cross Abstract: Image-level MRI harmonization is often evaluated with a frozen site probe, a site classifier trained on raw scans and applied unchanged to harmonized

safetyarxiv-cs-cv

IND-Eval: Aligning Evaluation and Simulation for Industrial Time Series Foundation Models

model-releasesarxiv-cs-lg

arXiv:2610.16652v1 Announce Type: new Abstract: We introduce IND-Eval, a public benchmark that organizes industrial time series across domains into source-specific forecasting tasks. Relative to estab

Industry

Anthropic is cutting off its internal evaluations from the internet

After a recent spate of high-profile incidents in which AI agents escaped containment, Anthropic is cutting off internet access for all internal evaluations. In a report Friday, the company detailed '

industrythe-verge-ai

10 Oct 2026

Local Ai

Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation