Search: “evaluation”

AI agents require specialized testing and debugging approaches that differ from traditional software due to their non-deterministic behavior and complex decision-making processes. This entry likely co

agentsarize-ai

5 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Auditing demographic bias in AI-based emergency police dispatch: a cross-lingual evaluation of eleven large language models

DGX agent

arXiv:2605.01451v1 Announce Type: new Abstract: Large language models (LLMs) are rapidly being integrated into high-stakes public safety systems, including emergency call triage and dispatch decision

safetyarxiv-cs-cl

5 May 2026

An Extended Evaluation Split for DeepSpaceYoloDataset

arXiv:2604.27593v1 Announce Type: cross Abstract: Recent technological advances in astronomy, particularly the growing popularity of smart telescopes for the general public, make it possible to develo

Beyond the Mean: Within-Model Reliable Change Detection for LLM Evaluation

arXiv:2604.27405v1 Announce Type: cross Abstract: We adapted the Reliable Change Index (RCI; Jacobson and Truax, 1991) from clinical psychology to item-level LLM version comparison on 2,000 MMLU-Pro i

Towards Neuro-symbolic Causal Rule Synthesis, Verification, and Evaluation Grounded in Legal and Safety Principles

arXiv:2604.28087v1 Announce Type: cross Abstract: Rule-based systems remain central in safety-critical domains but often struggle with scalability, brittleness, and goal misspecification. These limita

safetyarxiv-cs-ai

What Makes a Good Terminal-Agent Benchmark Task: A Guideline for Adversarial, Difficult, and Legible Evaluation Design

arXiv:2604.28093v1 Announce Type: new Abstract: Terminal-agent benchmarks have become a primary signal for measuring the coding and system-administration capabilities of large language models. As the

Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI

arXiv:2604.26382v1 Announce Type: cross Abstract: Most enterprise document AI today is a pipeline. Parse, index, retrieve, generate. Each of those stages has been studied to death on its own -- what's

Benchmarking Deep Learning and Vision Foundation Models for Atypical vs. Normal Mitosis Classification with Cross-Dataset Evaluation

model-releasesarxiv-cs-cv

arXiv:2506.21444v4 Announce Type: replace Abstract: Atypical mitosis marks a deviation in the cell division process that has been shown be an independent prognostic marker for tumor malignancy. Howeve

Generative AI-Based Virtual Assistant using Retrieval-Augmented Generation: An evaluation study for bachelor projects

arXiv:2604.25924v1 Announce Type: cross Abstract: Large Language Models have been increasingly employed in the creation of Virtual Assistants due to their ability to generate human-like text and handl

researcharxiv-cs-ai

The False Resonance: A Critical Examination of Emotion Embedding Similarity for Speech Generation Evaluation

arXiv:2604.26347v1 Announce Type: cross Abstract: Objective metrics for emotional expressiveness are vital for speech generation, particularly in expressive synthesis and voice conversion requiring em

safetyarxiv-cs-cl

VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models

arXiv:2505.22897v2 Announce Type: replace Abstract: While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less atten

Applications

Robustness Evaluation of a Foundation Segmentation Model Under Simulated Domain Shifts in Abdominal CT: Implications for Health Digital Twin Deployment

arXiv:2604.25685v1 Announce Type: cross Abstract: Foundation segmentation models such as the Segment Anything Model (SAM) have demonstrated strong generalization across natural images; however, their

applicationsarxiv-cs-cv

29 Apr 2026

A systematic evaluation of vision-language models for observational astronomical reasoning tasks

arXiv:2604.24589v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astro

ECoLAD: Deployment-Oriented Evaluation for Automotive Time-Series Anomaly Detection

arXiv:2603.10926v1 Announce Type: cross Abstract: Time-series anomaly detectors are commonly compared on workstation-class hardware under unconstrained execution. In-vehicle monitoring, however, requi

researcharxiv-cs-ai

Evaluation of Pose Estimation Systems for Sign Language Translation

arXiv:2604.24609v1 Announce Type: new Abstract: Many sign language translation (SLT) systems operate on pose sequences instead of raw video to reduce input dimensionality, improve portability, and par

researcharxiv-cs-cl

Machine Learning for Network Attacks Classification and Statistical Evaluation of Adversarial Learning Methodologies for Synthetic Data Generation

arXiv:2603.17717v3 Announce Type: replace-cross Abstract: Supervised detection of network attacks has always been a critical part of network intrusion detection systems (NIDS). Nowadays, in a pivotal

researcharxiv-cs-ai

Agents

Evaluation of image simulation open source solutions for simulation of synthetic images in lunar environment

arXiv:2604.22296v1 Announce Type: new Abstract: Synthetic image generation is one of the crucial input for planetary missions. It enables researchers and engineers to visualize planned planetary missi

agentsarxiv-cs-cv

MolClaw: An Autonomous Agent with Hierarchical Skills for Drug Molecule Evaluation, Screening, and Optimization

arXiv:2604.21937v1 Announce Type: new Abstract: Computational drug discovery, particularly the complex workflows of drug molecule screening and optimization, requires orchestrating dozens of specializ

Toward Automated Robustness Evaluation of Mathematical Reasoning

arXiv:2506.05038v2 Announce Type: replace Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in various reasoning-intensive tasks. However, these models exhibit unexpecte

TuneForge: an MCP server that lets your coding agent (Claude, Cursor, etc.) handle dataset generation, LoRA fine-tuning, RL, and evaluation directly in chat

TuneForge is an MCP (Model Context Protocol) server that enables coding agents like Claude and Cursor to perform machine learning operations directly within chat interfaces, including dataset generati

model-releasesr-ollama

Local Ai

Conformal Prediction Assessment: A Framework for Conditional Coverage Evaluation and Selection

arXiv:2603.27189v2 Announce Type: replace-cross Abstract: Conformal prediction provides rigorous distribution-free finite-sample guarantees for marginal coverage under the assumption of exchangeabilit

local-aiarxiv-cs-lg

24 Apr 2026

Applications

Differentially Private De-identification of Dutch Clinical Notes: A Comparative Evaluation

arXiv:2604.21421v1 Announce Type: cross Abstract: Protecting patient privacy in clinical narratives is essential for enabling secondary use of healthcare data under regulations such as GDPR and HIPAA.

applicationsarxiv-cs-ai

24 Apr 2026

Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence

model-releasesarxiv-cs-lg

arXiv:2106.01254v3 Announce Type: replace Abstract: In many classification tasks, there is no definitive ground truth, only human judgments that may disagree. We address two challenges that arise in s

24 Apr 2026

Development and Preliminary Evaluation of a Domain-Specific Large Language Model for Tuberculosis Care in South Africa

arXiv:2604.19776v1 Announce Type: new Abstract: Tuberculosis (TB) is one of the world's deadliest infectious diseases, and in South Africa, it contributes a significant burden to the country's health

23 Apr 2026

TactileEval: A Step Towards Automated Fine-Grained Evaluation and Editing of Tactile Graphics

arXiv:2604.19829v1 Announce Type: new Abstract: Tactile graphics require careful expert validation before reaching blind and visually impaired (BVI) learners, yet existing datasets provide only coarse

23 Apr 2026

Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

arXiv:2604.19502v1 Announce Type: new Abstract: The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks t

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

arXiv:2604.19533v1 Announce Type: cross Abstract: We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of th

Do Agents Dream of Root Shells? Partial-Credit Evaluation of LLM Agents in Capture The Flag Challenges

arXiv:2604.19354v1 Announce Type: new Abstract: Large Language Model (LLM) agents are increasingly proposed for autonomous cybersecurity tasks, but their capabilities in realistic offensive settings r

Evaluation of Winning Solutions of 2025 Low Power Computer Vision Challenge

arXiv:2604.19054v1 Announce Type: new Abstract: The IEEE Low-Power Computer Vision Challenge (LPCVC) aims to promote the development of efficient vision models for edge devices, balancing accuracy wit

How to add an evaluation harness to your Gemini CLI coding agent

Coding agents can update prompts, wire in tools, and change application logic across your codebase in a single run. The hard part isn’t getting the agent to make changes, but... The post How to add an

model-releasesarize-ai

Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models

model-releasesarxiv-cs-cv

arXiv:2601.22737v2 Announce Type: replace Abstract: The robust safety of Vision-Language Large Models (VLLMs) against joint multilingual and multimodal threats remains severely underexplored. Current

A Real-Time Bike-Pedestrian Safety System with Wide-Angle Perception and Evaluation Testbed for Urban Intersections

arXiv:2604.17046v1 Announce Type: new Abstract: Collisions between cyclists and pedestrians at urban intersections remain a persistent source of injuries, yet few systems attempt real-time warnings to

safetyarxiv-cs-cv

A Text-To-Text Alignment Algorithm for Better Evaluation of Modern Speech Recognition Systems

arXiv:2509.24478v2 Announce Type: replace Abstract: Modern neural networks have greatly improved performance across speech recognition benchmarks. However, gains are often driven by frequent words wit

safetyarxiv-cs-cl

Do LLMs Use Cultural Knowledge Without Being Told? A Multilingual Evaluation of Implicit Pragmatic Adaptation

arXiv:2604.17718v1 Announce Type: new Abstract: Many benchmarks show that large language models can answer direct questions about culture. We study a different question: do they also change how they s

safetyarxiv-cs-cl

FLARE: Task-agnostic embedding model evaluation through a normalization process

arXiv:2604.17344v1 Announce Type: cross Abstract: When task-specific labels are not available, it becomes difficult to select an embedding model for a specific target corpus. Existing labelless measur

When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluation

arXiv:2601.04609v2 Announce Type: replace Abstract: Vision-language models (VLMs) are increasingly used to make visual content accessible via text-based descriptions. In current systems, however, desc

researcharxiv-cs-cl

EuropeMedQA Study Protocol: A Multilingual, Multimodal Medical Examination Dataset for Language Model Evaluation

arXiv:2604.14306v1 Announce Type: new Abstract: While Large Language Models (LLMs) have demonstrated high proficiency on English-centric medical examinations, their performance often declines when fac

17 Apr 2026

A Proactive EMR Assistant for Doctor-Patient Dialogue: Streaming ASR, Belief Stabilization, and Preliminary Controlled Evaluation

arXiv:2604.13059v1 Announce Type: new Abstract: Most dialogue-based electronic medical record (EMR) systems still behave as passive pipelines: transcribe speech, extract information, and generate the

16 Apr 2026

Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

arXiv:2604.12968v1 Announce Type: cross Abstract: Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-ord

15 Apr 2026

Agents

Beyond Offline A/B Testing: Context-Aware Agent Simulation for Recommender System Evaluation

arXiv:2604.09549v1 Announce Type: cross Abstract: Recommender systems are central to online services, enabling users to navigate through massive amounts of content across various domains. However, the

agentsarxiv-cs-ai

14 Apr 2026

MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval

arXiv:2604.09552v1 Announce Type: cross Abstract: Engineering rulebooks and technical standards contain multimodal information like dense text, tables, and illustrations that are challenging for retri

14 Apr 2026

Towards Realistic 3D Emission Materials: Dataset, Baseline, and Evaluation for Emission Texture Generation

arXiv:2604.11006v1 Announce Type: new Abstract: 3D texture generation is receiving increasing attention, as it enables the creation of realistic and aesthetic texture materials for untextured 3D meshe

14 Apr 2026

On the Representational Limits of Quantum-Inspired 1024-D Document Embeddings: An Experimental Evaluation Framework

arXiv:2604.09430v1 Announce Type: cross Abstract: Text embeddings are central to modern information retrieval and Retrieval-Augmented Generation (RAG). While dense models derived from Large Language M

safetyarxiv-cs-ai

13 Apr 2026

Structural Evaluation Metrics for SVG Generation via Leave-One-Out Analysis

arXiv:2604.08809v1 Announce Type: new Abstract: Scalable Vector Graphics (SVG) represent visual content as structured, editable code. Each element (path, shape, or text node) can be individually inspe

researcharxiv-cs-lg

13 Apr 2026

Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook

arXiv:2604.06210v2 Announce Type: cross Abstract: As LLMs are globally deployed, aligning their cultural value orientations is critical for safety and user engagement. However, existing benchmarks fac

safetyarxiv-cs-ai

10 Apr 2026

Do MLLMs Really Understand Space? A Mathematical Reasoning Evaluation

arXiv:2602.11635v2 Announce Type: replace Abstract: Multimodal large language models (MLLMs) have achieved strong performance on perception-oriented tasks, yet their ability to perform mathematical sp

10 Apr 2026

Applications

Mitigating Domain Drift in Multi Species Segmentation with DINOv2: A Cross-Domain Evaluation in Herbicide Research Trials

arXiv:2508.07514v3 Announce Type: replace Abstract: Reliable plant species and damage segmentation for herbicide field research trials requires models that can withstand substantial real-world variati

applicationsarxiv-cs-cv

10 Apr 2026

Cross-subject decoding of imagined speech from scalp EEG: a leakage-controlled evaluation on three public corpora