Search: “evaluation”

Verifiable by Construction: Claim-Level Evaluation of Verbatim Citation in Clinical Question Answering

arXiv:2609.15964v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted for clinical question answering (QA). Current systems can attach citations to their answers, but t

15 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

The Convention Gap: Towards Measuring Implicit Communication in Cooperative AI Evaluation

DGX agent

arXiv:2609.11489v1 Announce Type: new Abstract: Cooperative AI agents are evaluated against other AIs, yet human cooperation relies on implicit conventions---shared protocols for reading meaning beyon

agentsarxiv-cs-ai

12 Sept 2026

Context operations to architecture modelling output from large language models and evaluation criteria for their use in systems engineering design

arXiv:2609.10132v1 Announce Type: cross Abstract: The development of generative artificial intelligence resources enables opportunities of speeding up systems and engineering design work. This contrib

safetyarxiv-cs-ai

DirectSwap: Paired, Mask-Free Video Head Swapping with Full-Reference Evaluation

arXiv:2512.09417v2 Announce Type: replace Abstract: Head swapping replaces an entire head while preserving pose, expression, body motion, and scene. Progress is limited by the lack of cross-identity p

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

arXiv:2609.11242v1 Announce Type: new Abstract: Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--

Grounded Evaluation and Repair for NL-to-PDDL Problem Generation

arXiv:2609.09898v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standa

Accountable and uncertainty-aware evaluation of sensor-based AI under distribution shift: devices, subjects, and nearly three years underground

arXiv:2609.09257v1 Announce Type: new Abstract: Sensor-based AI systems are rarely operated under the conditions under which they were trained: devices, personnel and recording epochs change, and each

researcharxiv-cs-lg

10 Sept 2026

AAMBERS-UAV: Acquisition-Aware Multimodal Backbone Evaluation and Ranking for UAV Weedy Rice Segmentation

arXiv:2609.05762v1 Announce Type: new Abstract: UAV image collections contain spatially and temporally related frames, yet semantic-segmentation benchmarks commonly split them at image level. Such spl

Beyond the Verdict: Evidence-Aligned Evaluation of Visual Prompt-Injection Guardrails

arXiv:2609.05535v1 Announce Type: new Abstract: Verdict-only evaluation does not reveal whether a vision-language model (VLM) used the visual evidence that should support its decision. We study this p

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

arXiv:2603.00039v2 Announce Type: replace Abstract: LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they im

DataFlex-RL: An Evaluation Platform for RLVR Data Policies

arXiv:2609.06107v1 Announce Type: new Abstract: Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which dom

Applications

EquiGQNet: Fast Grasp Quality Evaluation via Shared Equivariant Point Cloud Encoding

arXiv:2609.07145v1 Announce Type: new Abstract: Planning six-degree-of-freedom (6-DoF) grasps for unseen objects in cluttered tabletop scenes from a single-view depth image requires accurate and effic

applicationsarxiv-cs-ro

One Perturbation Is Not Enough: Identifiability and Blind Baselines for Behavioral AI Evaluation

arXiv:2609.06190v1 Announce Type: new Abstract: Behavioral evaluations perturb an input and read the induced change in the output in order to certify that a system uses that input. We show that the nu

safetyarxiv-cs-cv

Search-to-World: Evaluation of 3D World Delivery from User Request through Web Search

arXiv:2609.07605v1 Announce Type: new Abstract: Agentic systems can interpret user requests, search the live web, and use external tools, but their ability to transform retrieved web content into a us

The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation

arXiv:2609.08901v1 Announce Type: new Abstract: Approximate machine unlearning aims to remove the influence of specific training data from a trained model without retraining from scratch. We identify

BER-PEF: Unified Human Mobility Predictability Evaluation via Bayes Error Rate Estimation

arXiv:2609.04292v1 Announce Type: new Abstract: Human mobility predictability concerns the best prediction performance attainable from a given target and input information, but its ground truth is not

researcharxiv-cs-lg

7 Sept 2026

Agents

RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents

arXiv:2609.04898v1 Announce Type: cross Abstract: Repository-scale refactoring requires coding agents to propagate a single change across many interdependent files without altering program behavior, y

agentsarxiv-cs-ai

7 Sept 2026

IDSPACE: A Novel Document Generator for Reliable Evaluation of Digital Identity Verification Systems [Extended Technical Report]

arXiv:2609.03052v1 Announce Type: new Abstract: As services move online, trust institutions such as banks, lenders, and governments must verify the identity of remote users. Fraud detection tools are

4 Sept 2026

LightEMMA: A Longitudinal Evaluation of Vision-Language Models for Autonomous Driving

arXiv:2505.00284v3 Announce Type: replace-cross Abstract: Rapid advances in vision-language models (VLMs) have generated growing interest in their application to autonomous driving. A prevailing assum

4 Sept 2026

Agents

ClaimReceipt: Verifying Evidence Sufficiency and Coverage in Agent Evaluations

arXiv:2609.01992v1 Announce Type: new Abstract: Agent evaluations face two distinct evidentiary questions: whether a reported claim is recomputable from retained evidence (sufficiency), and whether th

agentsarxiv-cs-ai

GDB-Reward: From Evaluation Metrics to Training Rewards for Graphic Design

arXiv:2609.02813v1 Announce Type: new Abstract: Text-to-image models excel at natural image synthesis but struggle with graphic design, where success depends on satisfying precise constraints on typog

researcharxiv-cs-cv

Improving Health Literacy through Lay Summarization of Radiological Reports: An Evaluation of BioNER and Retrieval-Augmented Generation

arXiv:2609.02396v1 Announce Type: new Abstract: Radiology reports are written primarily for clinicians, and their specialized terminology often makes them difficult for patients to interpret. As a res

When Agents Implement Systems: A Case Study in Defects, Detection, and Evaluation Rigor

arXiv:2609.01985v1 Announce Type: new Abstract: As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requiremen

RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching

arXiv:2609.01470v1 Announce Type: new Abstract: As AI systems are increasingly used to draft radiology reports, reliably evaluating their clinical quality remains a critical challenge. Large language

safetyarxiv-cs-cv

2 Sept 2026

Local Ai

Towards AI-Assisted Clinical Trial Matching: Practical Considerations, Multicenter Evaluation, and Real-World Deployment

arXiv:2609.01202v1 Announce Type: cross Abstract: Clinical trials are essential for advancing cancer care and drug development, but many fail because of insufficient patient enrollment. While there is

local-aiarxiv-cs-ai

2 Sept 2026

A Large-scale Evaluation of Text-guided Models for Facial Editing

arXiv:2608.28802v1 Announce Type: cross Abstract: Facial appearance editing powers popular applications like FaceApp and Photoshop. Generative Adversarial Networks (GANs) and 3D Morphable Models (3DMM

researcharxiv-cs-ai

A Visual Question Answering Model to Automate Nondestructive Evaluation Image Analysis

arXiv:2608.29408v1 Announce Type: new Abstract: This study introduces a Visual Question Answering model designed specifically for nondestructive evaluation applications. VQA models allow inspectors to

researcharxiv-cs-cv

Agents

Agri-Sim: Agricultural Simulation Platform for Embodied Intelligence Evaluation in Greenhouse Robotics

arXiv:2608.29100v1 Announce Type: new Abstract: Agricultural-robot development requires simulation environments that can jointly support realistic scene construction, virtual sensing, autonomous navig

agentsarxiv-cs-ro

Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators

arXiv:2608.30110v1 Announce Type: cross Abstract: Nowcasting headline macroeconomic indicators, i.e., estimating an indicator's value for the current reference period before its official release, is c

Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry

arXiv:2608.30442v1 Announce Type: new Abstract: Recent offline reinforcement learning (RL) studies report policies that outperform physician decisions on clinical outcomes. We conduct a systematic, pa

RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction

arXiv:2608.28718v1 Announce Type: cross Abstract: Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) be

Simulation-Based Evaluation of Energy-Constrained Quantum-Classical Competition

arXiv:2308.08025v2 Announce Type: replace-cross Abstract: This paper develops a simulation-based framework for evaluating the energy implications of quantum and classical computing firms competing in

researcharxiv-cs-ai

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale

arXiv:2608.30614v1 Announce Type: new Abstract: Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at ma

researcharxiv-cs-cl

SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction

arXiv:2608.27461v1 Announce Type: new Abstract: Relational reasoning requires the process of perceptual understanding, comparing, and integrating the underlying relationships between concepts. This ab

31 Aug 2026

Leakage-Free Evaluation and Distribution-Robust Spatio-Temporal Graph Learning for Inductive Kriging

arXiv:2509.23631v2 Announce Type: replace Abstract: Inductive kriging estimates values at unobserved locations from sparse sensor data, enabling continuous field reconstruction when dense deployment i

researcharxiv-cs-lg

28 Aug 2026

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

arXiv:2505.20781v2 Announce Type: replace-cross Abstract: Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial i

28 Aug 2026

TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution

arXiv:2608.27182v1 Announce Type: new Abstract: LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however,

28 Aug 2026

Design and Evaluation of a Compliant Quasi Direct Drive End-effector for Safe Robotic Ultrasound Imaging

arXiv:2410.03086v2 Announce Type: replace Abstract: Robot-assisted ultrasound scanning promises to advance autonomous and accessible medical imaging. However, ensuring patient safety and compliant hum

safetyarxiv-cs-ro

26 Aug 2026

LLM-Guided Contextual Action Evaluation for Operational Decisions in Industrial Processes

arXiv:2608.24156v1 Announce Type: cross Abstract: Industrial actor--critic methods usually represent continuous actions as anonymous numerical coordinates. They must therefore learn from limited inter

safetyarxiv-cs-ai

26 Aug 2026

SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions

model-releasesarxiv-cs-ro

arXiv:2608.24094v1 Announce Type: new Abstract: Emergency vehicles (EMVs) can reorganize surrounding traffic as civilian vehicles brake, change lanes, or form rescue corridors in response to their pas

26 Aug 2026

Counterfactual Evaluation of Temporal Observation Protocols

arXiv:2608.22221v1 Announce Type: new Abstract: We study counterfactual protocol evaluation: whether data collected under a realised observation protocol determine the predictive value of alternatives

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

arXiv:2409.16183v2 Announce Type: replace Abstract: Radiology is a vital and complex component of modern clinical workflow and covers many tasks. Recently, vision-language (VL) foundation models in me

Applications

Explainable Adaptive Zero Trust Framework for AWS with Adversarial Robustness Evaluation

arXiv:2608.21477v1 Announce Type: cross Abstract: Cloud environments built on Amazon Web Services face a structural security vulnerability: once a credential passes authentication, the resulting sessi

applicationsarxiv-cs-lg

How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

arXiv:2608.23448v1 Announce Type: new Abstract: This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradig

Applications

Large-Scale Evaluation of Advanced Imputation Methods for Missing Values in Smart Meter Data

arXiv:2608.21638v1 Announce Type: new Abstract: Accurate and reliable collection of electricity consumption data through Advanced Metering Infrastructure (AMI) is of great importance for the operation

applicationsarxiv-cs-lg

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

arXiv:2608.23058v1 Announce Type: new Abstract: Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tool

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

arXiv:2608.20554v1 Announce Type: cross Abstract: The critical failure modes in deployed large language models (LLMs) are cross-dimensional: a model can score 99.3 in safety alignment while refusing o

24 Aug 2026

Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

arXiv:2608.20960v1 Announce Type: new Abstract: Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scien