Search: “evaluation”

Operational Reframing and Approval-Framed Delegation in Multi-Agent LLM Safety

arXiv:2607.07097v1 Announce Type: new Abstract: Safety evaluations of multi-agent LLM systems often compare a direct prompt with a planner-executor pipeline and report the difference as a single 'pipe

9 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

DGX agent

arXiv:2607.07663v1 Announce Type: new Abstract: AI systems increasingly participate in their own improvement: revising their outputs, adapting their own harnesses during deployment, training on data t

safetyarxiv-cs-ai

9 Jul 2026

Association Restoration Test: Revealing Restorable Shortcuts after Unlearning

arXiv:2607.05726v1 Announce Type: new Abstract: Association unlearning aims to disable learned label-attribute shortcuts while preserving task performance. Existing evaluations mainly measure output-l

Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

arXiv:2607.06482v1 Announce Type: cross Abstract: Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings. They typically focus on fac

Designing Maintainable Hybrid Generative Systems: A Quantum-Inspired Approach to Automated Music Harmony Generation

arXiv:2607.06296v1 Announce Type: cross Abstract: This paper presents the design and evaluation of a maintainable hybrid generative architecture for automated music harmony generation from melody. The

ExplAIner: A Declarative Query Language for Explaining Classification Models

arXiv:2607.06407v1 Announce Type: new Abstract: The XAI community has studied a wide range of queries and scores for explaining predictions of ML models. From a data management perspective, this proli

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

arXiv:2607.06254v1 Announce Type: cross Abstract: Deepfake image detection is currently served by three fundamentally different paradigms: commercial APIs, zero-shot vision-language models (LLMs), and

20 questions for the Agentic Enterprise (and how Agent Platform can help)

model-releasesgoogle-cloud-ai

If you’re an IT leader, you might be getting a lot of questions about how to build and deploy agents. The pressure to move fast is intense, but the engineering reality is incredibly complex. Where do

A Retrieval-Augmented Framework for Detecting and Resolving Pragmatic Ambiguities in Natural Language Requirements

arXiv:2607.04436v1 Announce Type: cross Abstract: Natural language requirements (NLRs) are essential for bridging communication gaps among diverse stakeholders in software development. However, the in

Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders

arXiv:2607.04071v1 Announce Type: cross Abstract: Portuguese remains underrepresented in text embedding evaluation, despite being one of the most widely spoken languages in the world. As a result, emb

EVA-Client: A Unified Data Collection, Inference, and Deployment Framework for Embodied Policies on Real Robots

arXiv:2607.02646v1 Announce Type: cross Abstract: We present EVA-Client, an open-source framework for deployment, data collection, and evaluation of trained manipulation policies on real robots. Sitti

safetyarxiv-cs-cv

FormalRx: Rectify and eXamine Semantic Failures in Autoformalization

model-releasesarxiv-cs-cl

arXiv:2607.04655v1 Announce Type: new Abstract: The veracious semantic alignment in autoformalization is significant for formal mathematical reasoning. However, existing evaluations provide only opaqu

Humanoid Everyday: A Comprehensive Robotic Dataset for Open-World Humanoid Manipulation

arXiv:2510.08807v3 Announce Type: replace-cross Abstract: From loco-motion to dextrous manipulation, humanoid robots have made remarkable strides in demonstrating complex full-body capabilities. Howev

safetyarxiv-cs-lg

Is Your Benchmark Still Useful? Dynamic Benchmarking for Code Language Models

model-releasesarxiv-cs-cl

arXiv:2503.06643v2 Announce Type: replace-cross Abstract: In this paper, we tackle a critical challenge in model evaluation: how to keep code benchmarks useful when models might have already seen them

Learning with the Nash-Sutcliffe loss

arXiv:2603.00968v2 Announce Type: replace-cross Abstract: The Nash-Sutcliffe efficiency (ext{NSE}) is a widely used, positively oriented relative measure for evaluating forecasts across multiple time

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

arXiv:2607.02879v1 Announce Type: new Abstract: Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case

The Role of Prompt Language and Translation-Theory-Driven Prompts in Large Language Models: A Case Study on Spanish-Chinese Journalistic Translation

arXiv:2607.03160v1 Announce Type: cross Abstract: This study examines how prompt language and translation theory-driven prompt design influence the quality of Spanish-Chinese journalistic translations

Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models

arXiv:2607.04199v1 Announce Type: new Abstract: The growing number of medical vision foundation models highlights the need for effective model selection. However, mainstream selection methods rely on

Training-Free Model Selection and Domain-Aware Score Calibration for First-Shot Anomalous Sound Detection

arXiv:2607.04526v1 Announce Type: cross Abstract: First-shot anomalous sound detection in DCASE Challenge Task 2 must flag anomalies of unseen machine types with a single threshold, without knowing wh

Uncertainty-Aware Last-Layer Adaptation of RETFound for Referable Diabetic Retinopathy Screening Under Dataset Shift

arXiv:2607.02569v1 Announce Type: new Abstract: This paper presents a safety-centered empirical evaluation of uncertainty-aware last-layer adaptation for referable diabetic retinopathy screening using

safetyarxiv-cs-cv

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

arXiv:2607.03386v1 Announce Type: new Abstract: Agentic AI systems are increasingly used to edit, refine, and repair decision policies, but evaluating these edits is difficult when per-state expert ac

A Novel Machine Learning Approach for Central Nervous System Tumor Classification from DNA Methylation

arXiv:2607.01307v1 Announce Type: new Abstract: NA methylation profiling has become a powerful approach for central nervous system (CNS) tumor classification, yet important challenges remain regarding

A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks

arXiv:2607.02175v1 Announce Type: new Abstract: Multiple-choice medical benchmarks are increasingly saturated, and recent rubric-based evaluations such as HealthBench have shown that open-ended clinic

AnyGroundBench: A Specialized-Domain Benchmark for Video Grounding in Vision-Language Models

arXiv:2607.02269v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have demonstrated immense promise in Spatio-Temporal Video Grounding (STVG). However, current evaluation protocols are l

Local Ai

Auto-FL-Research: Agentic Search for Federated Learning Algorithms

arXiv:2607.01366v1 Announce Type: new Abstract: Federated learning (FL) research often depends on many small but consequential algorithmic choices: optimizer variants, server aggregation rules, local

local-aiarxiv-cs-ai

Beyond the Performance Illusion: Structure-Aware Stratified Partitioning and Curriculum Distributionally Robust Optimization for Spatially Correlated Domains

arXiv:2607.02055v1 Announce Type: cross Abstract: Performance evaluation in AI systems commonly assumes that random dataset splits produce independent and identically distributed (i.i.d.) subsets. We

Tutorials

Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages

arXiv:2607.02235v1 Announce Type: cross Abstract: LLM-as-a-Judge has become the dominant evaluation paradigm for many natural language generation tasks, due to shortcomings of conventional metrics and

tutorialsarxiv-cs-ai

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

arXiv:2607.01813v1 Announce Type: cross Abstract: Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to

AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

arXiv:2607.00726v1 Announce Type: new Abstract: Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for

Cross-Domain Generalization Failure in Lightweight Intrusion Detection Models for IIoT Networks

arXiv:2607.00553v1 Announce Type: cross Abstract: Lightweight machine learning models are increasingly proposed for intrusion detection in Industrial Internet of Things (IIoT) networks due to their su

Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

arXiv:2606.18367v2 Announce Type: replace Abstract: Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating

Generative Refinement for Low-Budget Black-Box Optimization

arXiv:2607.00691v1 Announce Type: new Abstract: Black-box optimization is a fundamental science and engineering tool that makes it possible to optimize objectives without gradient information. Unfortu

OpenReward: Learning to Reward Long-form Agentic Tasks via Reinforcement Learning

arXiv:2510.24636v3 Announce Type: replace Abstract: Reward models (RMs) have become essential for aligning large language models (LLMs), serving as scalable proxies for human evaluation in both traini

safetyarxiv-cs-cl

Auditing Generalization in AI-Generated Video Detection: A Six-Control Protocol and the VidAudit Toolkit

arXiv:2606.31004v1 Announce Type: new Abstract: AI-generated video detection benchmarks such as GenVidBench and AIGVDBench are the de facto leaderboards, yet most evaluation protocols leave uncontroll

1 Jul 2026

Overview of the TalentCLEF 2026: Skill and Job Title Intelligence for Human Capital Management

arXiv:2606.31692v1 Announce Type: new Abstract: This paper presents an overview of the second edition of the TalentCLEF challenge, organized as a Lab at the Conference and Labs of the Evaluation Forum

researcharxiv-cs-cl

1 Jul 2026

Reference-Free Image Quality Assessment for Virtual Try-On via Human Feedback

arXiv:2603.13057v2 Announce Type: replace Abstract: As virtual try-on (VTON) systems become increasingly important in fashion e-commerce, there is a growing need for reliable reference-free evaluation

1 Jul 2026

Applications

A Good Talk Does not Look Like a Summary, It Teaches You! Measuring Takeaways from Paper-to-Video Talks

arXiv:2606.28531v1 Announce Type: cross Abstract: Automatically generated videos from scientific papers are increasingly used for education and research dissemination. However, existing evaluation met

applicationsarxiv-cs-cl

A Systems-Level Analysis of Sensitivity, Robustness, and Stability in Retrieval-Augmented Generation

arXiv:2606.28337v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems are often evaluated using final answer accuracy, even though their failures can originate from preprocess

Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?

model-releasesarxiv-cs-cl

arXiv:2606.29920v1 Announce Type: new Abstract: Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliab

Critical Interval MSE: Toward Reliable Offline Validation for Robot Manipulation Policies

arXiv:2606.29898v1 Announce Type: cross Abstract: Real-world evaluation is the gold standard for robot policies because it tests them against the physical conditions and deployment challenges they are

safetyarxiv-cs-ai

Entity Binding Failures in Tool-Augmented Agents

arXiv:2606.30531v1 Announce Type: new Abstract: Tool-augmented language-model agents are often evaluated by whether they select the correct tool, produce valid API arguments, and complete the requeste

safetyarxiv-cs-ai

fev-bench: A Realistic Benchmark for Time Series Forecasting

model-releasesarxiv-cs-lg

arXiv:2509.26468v3 Announce Type: replace Abstract: Benchmark quality is critical for meaningful evaluation and sustained progress in time series forecasting, particularly with the rise of pretrained

Agents

SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

arXiv:2606.30573v1 Announce Type: new Abstract: We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing front

agentsarxiv-cs-lg

Cross-view Multimodal Vision-Based Assessment Framework for Traditional Chinese Medicine Rehabilitation Training

arXiv:2606.28104v1 Announce Type: new Abstract: Vision-based assessment can provide convenient and cost-effective evaluation in Traditional Chinese Medicine (TCM) rehabilitation training, where action

29 Jun 2026

Denoising ICF Images with Multiplicative Uniform Noise: A Self-Supervised Study Based on the Log-Domain Noisier2Inverse Framework

arXiv:2606.27635v1 Announce Type: new Abstract: This paper documents the implementation and evaluation of a self-supervised denoising framework on Inertial Confinement Fusion (ICF) images corrupted by

29 Jun 2026

Consistent Distributed Ranking of Generative Models via Kernel Distances

arXiv:2310.11714v5 Announce Type: replace Abstract: Ranking generative models based on the fidelity and diversity of their outputs is required to identify the best generator in a group of candidate ge

26 Jun 2026

CORTEX: A Structured Reasoning Benchmark for Trustworthy 3D Chest CT MLLMs

arXiv:2606.27264v1 Announce Type: new Abstract: Reasoning in multimodal large language models (MLLMs) has shown strong promise in medical imaging. However, this reasoning is usually free-form text jud

26 Jun 2026

Exact and Deterministic Patch Descriptor Retrieval via Hierarchical Normalization

arXiv:2606.27280v1 Announce Type: new Abstract: We present a patch descriptor retrieval method that returns the exact nearest neighbour -- provably identical to exhaustive full-vector search -- while