Search: “evaluation”

A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

arXiv:2607.21274v1 Announce Type: cross Abstract: We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. W

24 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Evaluation and Prognostic Validation of Deep Regression Models for WSI-Based Gene-Expression Prediction

DGX agent

arXiv:2410.00945v2 Announce Type: replace-cross Abstract: Gene-expression profiling is widely used in research and central to many areas of precision oncology, but remains costly and not universally a

researcharxiv-cs-cv

24 Jul 2026

Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation

arXiv:2607.20478v1 Announce Type: cross Abstract: Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy con

24 Jul 2026

A Multi-Dimensional Evaluation of Explainability in Media Bias Detection

arXiv:2607.19954v1 Announce Type: new Abstract: Detecting media bias automatically is difficult because biased framing is often subtle, yet in domains such as news analysis, accurate predictions alone

safetyarxiv-cs-cl

Attention Without Grounding: Causal Evaluation of Visual Explanations in Medical VLMs

arXiv:2607.18577v1 Announce Type: new Abstract: Attention and saliency heatmaps are widely used to explain medical Vision-Language Model (VLM) outputs on chest X-rays, yet whether they truly highlight

Crashing Waves vs. Rising Tides: Findings on AI Automation from Thousands of Worker Evaluations of Labor Market Tasks

arXiv:2604.01363v2 Announce Type: replace Abstract: We propose that AI automation is a continuum between: (i) crashing waves where AI capabilities surge abruptly over small sets of tasks, and (ii) ris

FORCE-Bench: A Benchmark, Dataset, and Evaluation Harness for Agentic AI in Enterprise Finance

arXiv:2607.19409v1 Announce Type: new Abstract: Recent advances in large language models have accelerated deployment of agentic systems in operational finance. Existing benchmarks emphasize measuring

Agents

Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System

arXiv:2607.13370v1 Announce Type: cross Abstract: This paper is an extension of a paper presented at the ICAART 2026 conference, which introduced LEA (Learning Engagement Assistant), an adaptive AI tu

agentsarxiv-cs-ai

16 Jul 2026

Prospective clinical indication, post-hoc report leakage, and fusion design in multi-image chest radiograph classification: a patient-clustered evaluation

arXiv:2607.13800v1 Announce Type: cross Abstract: Chest radiograph datasets often combine multiple images with Clinical Indication, Findings, and Impression, although these inputs are produced at diff

16 Jul 2026

Agents

Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results

arXiv:2607.11183v2 Announce Type: replace Abstract: Large language models increasingly operate as tool-using agents, where small format, argument, or function-call errors can invalidate otherwise plau

agentsarxiv-cs-cl

15 Jul 2026

Calibrated Selective Prediction Using Deep Ensembles for ROI-Based Thyroid Nodule Ultrasound Classification Under Dataset Shift: A Retrospective Evaluation

arXiv:2607.12075v1 Announce Type: cross Abstract: Background: Deep learning models can classify thyroid nodules on ultrasound, but reliable clinical decision support also requires calibrated probabili

safetyarxiv-cs-ai

15 Jul 2026

Local Ai

Form, Not Content? A Preregistered, Placebo-Controlled Evaluation of Learned Error-Conditioned Self-Repair Through Prompts and Weights in Frozen Small Code Models

arXiv:2607.12962v1 Announce Type: cross Abstract: Frozen small code LLMs are deployed locally, yet the information guiding a retry after a failed attempt is still measured without placebo controls in

local-aiarxiv-cs-ai

15 Jul 2026

Metrics or Mirage? An Audit of Evaluation Inconsistencies in Colonoscopy Polyp Segmentation Benchmarks

arXiv:2607.08203v1 Announce Type: new Abstract: Progress in colonoscopy polyp segmentation is routinely reported through leaderboard comparisons on a small set of public benchmarks. We argue that this

Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms

arXiv:2607.07769v1 Announce Type: cross Abstract: Starting from the utilization of deep neural networks to approximate the state-action value function that led to winning one of the most challenging g

Systematic Evaluation of Learning Rate Scheduling Strategies Across Heterogeneous Architectures

arXiv:2607.08511v1 Announce Type: cross Abstract: Choosing a learning rate scheduling strategy is critical to neural network training, but manual selection is costly and rarely exhaustive. While class

VEGAS: Human-Aligned Video Caption Evaluation via Gaze

arXiv:2607.08489v1 Announce Type: cross Abstract: Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention. We propose V

AgentLens: Production-Assessed Trajectory Reviews for Coding Agent Evaluation

arXiv:2607.06624v1 Announce Type: new Abstract: We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the ta

9 Jul 2026

Discovering Geometric Biases in 3D Face Reconstruction: A Curvature-Aware Spectral Framework for Fairness Evaluation

arXiv:2607.07486v1 Announce Type: new Abstract: 3D Morphable Models (3DMMs) remain the standard parametric shape priors for many state-of-the-art 3D face reconstruction algorithms. However, as these m

safetyarxiv-cs-cv

9 Jul 2026

Overview of the NLPCC 2026 Shared Task 1: Difficulty-Aware Multilingual and Multimodal Medical Instructional Video Understanding Evaluation

arXiv:2607.06618v1 Announce Type: cross Abstract: Following the CMIVQA, MMI-VQA, and M4IVQA challenges in NLPCC 2023--2025, we introduce the Difficulty-Aware Medical Instructional Video Question Answe

9 Jul 2026

A VLM-Enhanced Framework for Comprehensive Traffic Sign Condition Assessment Integrating Daytime Visual Performance and Nighttime Retroreflectivity Evaluation

arXiv:2607.06478v1 Announce Type: new Abstract: Traffic signs are crucial components of road safety, serving as visual tools under all lighting conditions. The Manual on Uniform Traffic Control Device

8 Jul 2026

On the feasibility of dependency parsing of non-human sequences without a gold standard. Is evaluation possible in other species?

arXiv:2607.06542v1 Announce Type: new Abstract: Dependency parsing consists of finding a tree representation for a sequence. Unsupervised dependency parsing aims to develop parsing methods without a g

researcharxiv-cs-cl

8 Jul 2026

PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation

arXiv:2607.06440v1 Announce Type: new Abstract: Recent text-to-image models such as DALLE-3 excel at following diverse prompts yet remain blind to individual aesthetic preferences. We study personaliz

8 Jul 2026

Graph Classification via Network Usable Information: From Representation Evaluation to Structure Selection

model-releasesarxiv-cs-lg

arXiv:2607.03587v1 Announce Type: new Abstract: We propose NetinfoGC, a framework for graph classification that extends the Network Usable Information (NUI) paradigm to graph-level learning. Unlike co

Tutorials

RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models

arXiv:2510.19698v3 Announce Type: replace Abstract: Large Language Models (LLMs) can propose rules in natural language, sidestepping the need for a predefined predicate space in traditional rule learn

tutorialsarxiv-cs-ai

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models

arXiv:2607.05365v1 Announce Type: cross Abstract: Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech. However, standard speech and text benchmarks d

Towards Standardized Light Field Quality Assessment: Hybrid Subjective Benchmarking and Objective Metric Evaluation

arXiv:2607.03494v1 Announce Type: new Abstract: Benchmarking immersive media coding solutions, especially in the standardization context, requires reliable and reproducible subjective quality assessme

TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluation

arXiv:2607.03447v1 Announce Type: cross Abstract: Knowledge graphs (KGs) that underpin Graph-based Retrieval-Augmented Generation (Graph-RAG) are increasingly built automatically by LLM-driven extract

Prompt Framing Distorts Count-Based Evaluation of LLM Error Detection: Evidence from Numeric Anchoring

arXiv:2607.01240v1 Announce Type: cross Abstract: Count-based F1 is widely used as a proxy for LLM error-detection quality, but this paper shows that it can rise dramatically without a corresponding i

3 Jul 2026

Psychological Steering in LLMs: An Evaluation of Effectiveness and Trustworthiness

arXiv:2510.04484v2 Announce Type: replace-cross Abstract: The ability to control LLMs' emulated emotional states and personality traits is an essential step in enabling rich, human-centered interactio

3 Jul 2026

Sheet Music Benchmark: Standardized Optical Music Recognition Evaluation

arXiv:2506.10488v3 Announce Type: replace Abstract: In this work, we introduce the Sheet Music Benchmark (SMB), a dataset of six hundred and eighty-five pages specifically designed to benchmark Optica

2 Jul 2026

Vibe Coding Ate My Homework: An evaluation of AI approaches to greenfield software engineering and programming

arXiv:2606.18293v2 Announce Type: replace-cross Abstract: Thanks to rapid developments in generative AI, we are in the midst of a paradigm shift that may change how we interact with computers forever.

2 Jul 2026

A Realistic Protocol for Evaluation of Weakly Supervised Object Localization

arXiv:2404.10034v3 Announce Type: replace Abstract: Weakly Supervised Object Localization (WSOL) allows training deep learning models for classification and localization (LOC) using only global class-

Cross-lingual Relation Extraction with Large Language Models: Zero-Shot, Few-Shot, and Fine-Tuned Evaluation on Romanian

arXiv:2606.31718v1 Announce Type: cross Abstract: Relation extraction (RE) for low-resource languages is typically constrained by the lack of annotated corpora. We investigate the feasibility of cross

Patient-Level Elbow Abnormality Detection: Leakage-Aware Evaluation of Learned Preprocessing, Calibration, and Triage-Oriented Operating Points

arXiv:2606.31348v1 Announce Type: new Abstract: In this study, we examine learned preprocessing pipelines in the context of triage-oriented orthopedic abnormality detection task using elbow radiograph

Probing Stylistic Appropriation using Large Language Models: An Evaluation Framework for Copyright Infringement under EU Law

arXiv:2606.31250v1 Announce Type: cross Abstract: Large language models (LLM) trained on web-scale corpora generate output that may infringe copyright, yet existing technical safeguards focus narrowly

Wind and State Estimation on SE(3): Comparative Evaluation of EKF and UKF with Continuous and Discrete Quadrotor Models

arXiv:2606.30804v1 Announce Type: new Abstract: Use of quadrotor UAVs for wind velocity estimation is gaining popularity in recent studies, leveraging their maneuverability, compact size and low cost.

researcharxiv-cs-ro

Clinical Reasoning Graphs: Structured Evaluation of LLM Diagnostic Reasoning Reveals Competence Without Consistency

arXiv:2606.29876v1 Announce Type: cross Abstract: Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable

Empirical Evaluation of Multi-Modal Touch Detection in Over-the-Shoulder Video Surveillance

arXiv:2606.29504v1 Announce Type: new Abstract: Video Intelligence Surveillance (VIDINT) on over-the-shoulder footage is a proposed vector for monitoring human-computer interaction patterns without di

Overcoming Dependent Censoring in the Evaluation of Survival Models

arXiv:2502.19460v4 Announce Type: replace-cross Abstract: Dependent censoring occurs when the event time and censoring time are not conditionally independent given the observed covariates. This compli

SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics

arXiv:2606.29894v1 Announce Type: cross Abstract: As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theore

Local Ai

ProfileFoundry: A Synthetic Person-Object Substrate for Privacy, Memory, and Tool-Use Evaluation in LLM Agent

arXiv:2606.26403v1 Announce Type: new Abstract: Foundation-model research increasingly needs data about people: user state, personal histories, relationships, contact-like fields, documents, and longi

local-aiarxiv-cs-cl

26 Jun 2026

Evaluation Protocols and Validation for Cameras in Indoor Healthcare Monitoring

arXiv:2606.25284v1 Announce Type: new Abstract: Camera-based monitoring systems are increasingly adopted in healthcare settings for the continuous assessment of patient movement and activities. Howeve

safetyarxiv-cs-cv

25 Jun 2026

Gastroendoscopy View Synthesis: A New Real Dataset and Evaluation

arXiv:2606.25427v1 Announce Type: new Abstract: Novel view synthesis (NVS) is an active research topic in computer vision, owing to the success of neural radiance field (NeRF) and 3D Gaussian splattin

25 Jun 2026

Applications

A Fair Evaluation of Graph Foundation Models for Node Property Prediction

arXiv:2606.24509v1 Announce Type: cross Abstract: Due to the wide use of graph-structured data in different fields of industry and science, the development of Graph Foundation Models (GFMs) has recent

applicationsarxiv-cs-ai

Do LLM Attribution Metrics Transfer? Auditing Retrieval-Augmented Generation Evaluation Across Datasets and Constructs

arXiv:2606.23915v1 Announce Type: new Abstract: Practice often treats automatic metrics for attribution in LLM retrieval-augmented generation as interchangeable. We audit eight automatic scorers -- le

researcharxiv-cs-cl

One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline

model-releasesarxiv-cs-lg

arXiv:2606.23767v1 Announce Type: new Abstract: Headline accuracies on the Tuebingen cause-effect pairs are routinely compared across papers even though each is measured under its authors' own protoco

Trimming the Long-Tail of Visual World Modeling Evaluation

arXiv:2606.24256v1 Announce Type: new Abstract: Physical interactions follow a long-tailed distribution: a set of common and regular interactions dominates human experience and visual data, while a br

Beyond Accuracy: Measuring Bias Acknowledgment in Chain-of-Thought Reasoning for Responsible AI Evaluation