Search: “evaluation”

Instance-Level Costs for Nuanced Classifier Evaluation

arXiv:2605.03135v1 Announce Type: new Abstract: Standard classification treats all errors equally, but in content moderation, medical screening, and safety-critical applications, mistakes on clear-cut

safetyarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding

DGX agent

arXiv:2605.03514v1 Announce Type: new Abstract: The remarkable success of large language models (LLMs) has motivated researchers to adapt them as universal predictors for various graph tasks. As a wid

researcharxiv-cs-cl

6 May 2026

Hardware

Robust Visual SLAM for UAV Navigation in GPS-Denied and Degraded Environments: A Multi-Paradigm Evaluation and Deployment Study

arXiv:2605.03678v1 Announce Type: new Abstract: Reliable localization in GPS-denied, visually degraded environments is critical for autonomous UAV opera- tions. This paper presents a systematic compar

hardwarearxiv-cs-ro

TMD-Bench: A Multi-Level Evaluation Paradigm for Music-Dance Co-Generation

arXiv:2605.01809v1 Announce Type: cross Abstract: Unified audio-visual generation is rapidly gaining industrial and creative relevance, enabling applications in virtual production and interactive medi

VisInject: Disruption != Injection -- A Dual-Dimension Evaluation of Universal Adversarial Attacks on Vision-Language Models

arXiv:2605.01449v1 Announce Type: cross Abstract: Universal adversarial attacks on aligned multimodal large language models are increasingly reported with attack success rates in the 60-80% range, sug

Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models

arXiv:2605.01451v1 Announce Type: new Abstract: Large language models (LLMs) are rapidly being integrated into high-stakes public safety systems, including emergency call triage and dispatch decision

5 May 2026

An Extended Evaluation Split for DeepSpaceYoloDataset

arXiv:2604.27593v1 Announce Type: cross Abstract: Recent technological advances in astronomy, particularly the growing popularity of smart telescopes for the general public, make it possible to develo

Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

arXiv:2604.27405v1 Announce Type: cross Abstract: We adapted the Reliable Change Index (RCI; Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro i

Towards Neuro-symbolic Causal Rule Synthesis, Verification, and Evaluation Grounded in Legal and Safety Principles

arXiv:2604.28087v1 Announce Type: cross Abstract: Rule-based systems remain central in safety-critical domains but often struggle with scalability, brittleness, and goal misspecification. These limita

safetyarxiv-cs-ai

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

arXiv:2604.28093v1 Announce Type: new Abstract: Terminal-agent benchmarks have become a primary signal for measuring the coding and system-administration capabilities of large language models. As the

Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI

arXiv:2604.26382v1 Announce Type: cross Abstract: Most enterprise document AI today is a pipeline. Parse, index, retrieve, generate. Each of those stages has been studied to death on its own -- what's

Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluation

model-releasesarxiv-cs-cv

arXiv:2506.21444v4 Announce Type: replace Abstract: Atypical mitosis marks a deviation in the cell division process that has been shown be an independent prognostic marker for tumor malignancy. Howeve

Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects

arXiv:2604.25924v1 Announce Type: cross Abstract: Large Language Models have been increasingly employed in the creation of Virtual Assistants due to their ability to generate human-like text and handl

researcharxiv-cs-ai

The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation

arXiv:2604.26347v1 Announce Type: cross Abstract: Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring em

VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models

arXiv:2505.22897v2 Announce Type: replace Abstract: While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less atten

Applications

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment

arXiv:2604.25685v1 Announce Type: cross Abstract: Foundation segmentation models such as the Segment Anything Model (SAM) have demonstrated strong generalization across natural images; however, their

applicationsarxiv-cs-cv

29 Apr 2026

A systematic evaluation of vision-language models for observational astronomical reasoning tasks

arXiv:2604.24589v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astro

ECoLAD: Deployment-Oriented Evaluation for Automotive Time-Series Anomaly Detection

arXiv:2603.10926v1 Announce Type: cross Abstract: Time-series anomaly detectors are commonly compared on workstation-class hardware under unconstrained execution. In-vehicle monitoring, however, requi

researcharxiv-cs-ai

Evaluation of Pose Estimation Systems for Sign Language Translation

arXiv:2604.24609v1 Announce Type: new Abstract: Many sign language translation (SLT) systems operate on pose sequences instead of raw video to reduce input dimensionality, improve portability, and par

researcharxiv-cs-cl

Machine Learning for Network Attacks Classification and Statistical Evaluation of Adversarial Learning Methodologies for Synthetic Data Generation

arXiv:2603.17717v3 Announce Type: replace-cross Abstract: Supervised detection of network attacks has always been a critical part of network intrusion detection systems (NIDS). Nowadays, in a pivotal

researcharxiv-cs-ai

Agents

Evaluation of image simulation open source solutions for simulation of synthetic images in lunar environment

arXiv:2604.22296v1 Announce Type: new Abstract: Synthetic image generation is one of the crucial input for planetary missions. It enables researchers and engineers to visualize planned planetary missi

agentsarxiv-cs-cv

27 Apr 2026

MolClaw: An Autonomous Agent with Hierarchical Skills for Drug Molecule Evaluation, Screening, and Optimization

arXiv:2604.21937v1 Announce Type: new Abstract: Computational drug discovery, particularly the complex workflows of drug molecule screening and optimization, requires orchestrating dozens of specializ

27 Apr 2026

Toward Automated Robustness Evaluation of Mathematical Reasoning

arXiv:2506.05038v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning-intensive tasks. However, these models exhibit unexpecte

27 Apr 2026

Local Ai

Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

arXiv:2603.27189v2 Announce Type: replace-cross Abstract: Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeabilit

local-aiarxiv-cs-lg

24 Apr 2026

Applications

Differentially Private De-identification of Dutch Clinical Notes: A Comparative Evaluation

arXiv:2604.21421v1 Announce Type: cross Abstract: Protecting patient privacy in clinical narratives is essential for enabling secondary use of healthcare data under regulations such as GDPR and HIPAA.

applicationsarxiv-cs-ai

24 Apr 2026

Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

model-releasesarxiv-cs-lg

arXiv:2106.01254v3 Announce Type: replace Abstract: In many classification tasks, there is no definitive ground truth, only human judgments that may disagree. We address two challenges that arise in s

24 Apr 2026

Development and Preliminary Evaluation of a Domain-Specific Large Language Model for Tuberculosis Care in South Africa

arXiv:2604.19776v1 Announce Type: new Abstract: Tuberculosis (TB) is one of the world's deadliest infectious diseases, and in South Africa, it contributes a significant burden to the country's health

23 Apr 2026

TactileEval: A Step Towards Automated Fine-Grained Evaluation and Editing of Tactile Graphics

arXiv:2604.19829v1 Announce Type: new Abstract: Tactile graphics require careful expert validation before reaching blind and visually impaired (BVI) learners, yet existing datasets provide only coarse

23 Apr 2026

Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

arXiv:2604.19502v1 Announce Type: new Abstract: The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks t

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

arXiv:2604.19533v1 Announce Type: cross Abstract: We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of th

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture The Flag Challenges

arXiv:2604.19354v1 Announce Type: new Abstract: Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings r

Evaluation of Winning Solutions of 2025 Low Power Computer Vision Challenge

arXiv:2604.19054v1 Announce Type: new Abstract: The IEEE Low-Power Computer Vision Challenge (LPCVC) aims to promote the development of efficient vision models for edge devices, balancing accuracy wit

Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models

model-releasesarxiv-cs-cv

arXiv:2601.22737v2 Announce Type: replace Abstract: The robust safety of Vision-Language Large Models (VLLMs) against joint multilingual and multimodal threats remains severely underexplored. Current

A Real-Time Bike-Pedestrian Safety System with Wide-Angle Perception and Evaluation Testbed for Urban Intersections

arXiv:2604.17046v1 Announce Type: new Abstract: Collisions between cyclists and pedestrians at urban intersections remain a persistent source of injuries, yet few systems attempt real-time warnings to

safetyarxiv-cs-cv

A Text-To-Text Alignment Algorithm for Better Evaluation of Modern Speech Recognition Systems

arXiv:2509.24478v2 Announce Type: replace Abstract: Modern neural networks have greatly improved performance across speech recognition benchmarks. However, gains are often driven by frequent words wit

Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation

arXiv:2604.17718v1 Announce Type: new Abstract: Many benchmarks show that large language models can answer direct questions about culture. We study a different question: do they also change how they s

FLARE: Task-agnostic embedding model evaluation through a normalization process

arXiv:2604.17344v1 Announce Type: cross Abstract: When task-specific labels are not available, it becomes difficult to select an embedding model for a specific target corpus. Existing labelless measur

When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation

arXiv:2601.04609v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, desc

researcharxiv-cs-cl

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

arXiv:2604.14306v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when fac

17 Apr 2026

A Proactive EMR Assistant for Doctor-Patient Dialogue: Streaming ASR, Belief Stabilization, and Preliminary Controlled Evaluation

arXiv:2604.13059v1 Announce Type: new Abstract: Most dialogue-based electronic medical record (EMR) systems still behave as passive pipelines: transcribe speech, extract information, and generate the

16 Apr 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

arXiv:2604.12968v1 Announce Type: cross Abstract: Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-ord

15 Apr 2026

Agents

Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation

arXiv:2604.09549v1 Announce Type: cross Abstract: Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, the

agentsarxiv-cs-ai

14 Apr 2026

MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

arXiv:2604.09552v1 Announce Type: cross Abstract: Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retri

14 Apr 2026

Towards Realistic 3D Emission Materials: Dataset, Baseline, and Evaluation for Emission Texture Generation

arXiv:2604.11006v1 Announce Type: new Abstract: 3D texture generation is receiving increasing attention, as it enables the creation of realistic and aesthetic texture materials for untextured 3D meshe

14 Apr 2026

On the Representational Limits of Quantum-Inspired 1024-D Document Embeddings: An Experimental Evaluation Framework

arXiv:2604.09430v1 Announce Type: cross Abstract: Text embeddings are central to modern information retrieval and Retrieval-Augmented Generation (RAG). While dense models derived from Large Language M

safetyarxiv-cs-ai

13 Apr 2026

Structural Evaluation Metrics for SVG Generation via Leave-One-Out Analysis

arXiv:2604.08809v1 Announce Type: new Abstract: Scalable Vector Graphics (SVG) represent visual content as structured, editable code. Each element (path, shape, or text node) can be individually inspe

researcharxiv-cs-lg

13 Apr 2026

Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

arXiv:2604.06210v2 Announce Type: cross Abstract: As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement. However, existing benchmarks fac

safetyarxiv-cs-ai

10 Apr 2026

Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation

arXiv:2602.11635v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have achieved strong performance on perception-oriented tasks, yet their ability to perform mathematical sp