Search: “evaluation”

ClayBuddy: A Framework, Evaluation, & Mitigation of Coding Agent Failures

arXiv:2606.19380v2 Announce Type: replace-cross Abstract: Software engineering and deployment are increasingly delegated to AI coding agents. The scale of their adoption is surfacing rare, but highly

safetyarxiv-cs-lg

23 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Each Judge Its Own Yardstick: Discovering Per-VLM Taxonomies for Physical Video Evaluation

DGX agent

arXiv:2606.22918v1 Announce Type: new Abstract: Maintaining physical consistency in video generators and world models increasingly relies on vision-language models (VLMs) as automated judges that prov

model-releasesarxiv-cs-cv

23 Jun 2026

Evaluation of Medical Vision Language Models HuluMed and MedGemma, and general purpose chatbots Gemma 3, ChatGPT Plus, and Claude Pro on real previously unseen wound images

model-releasesarxiv-cs-cv

arXiv:2606.20723v1 Announce Type: new Abstract: Chronic wound assessment remains a clinically challenging task that requires accurate interpretation of wound morphology, tissue composition, vascular c

23 Jun 2026

REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation

model-releasesarxiv-cs-cv

arXiv:2606.20736v1 Announce Type: new Abstract: Static visual question answering (VQA) benchmarks age quickly: Once the items leak into training corpora, scores can reflect memorization rather than ge

23 Jun 2026

Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application

arXiv:2606.12191v1 Announce Type: cross Abstract: Environments serve as interactive systems for large language model (LLM) based agents across diverse scenarios and play a crucial role in driving the

agentsarxiv-cs-ai

From Nominal Intensity to Equivalent Rainfall: A Path-Based Credibility Evaluation Framework for Simulated Rainfall in Autonomous-Driving Perception Tests

arXiv:2606.11989v1 Announce Type: new Abstract: Credible simulated-rainfall conditions are essential for identifying perception-system boundaries and supporting SOTIF-oriented risk assessment in autom

agentsarxiv-cs-cv

SDQM: Synthetic Data Quality Metric for Object Detection Dataset Evaluation

arXiv:2510.06596v2 Announce Type: replace-cross Abstract: The performance of machine learning models depends heavily on training data. The scarcity of large-scale, well-annotated datasets poses signif

Soft-Prompt Tuning for Fair and Efficient LLM Benchmark Evaluation

arXiv:2606.12117v1 Announce Type: cross Abstract: Benchmark scores often misrepresent a large language model's (LLM's) knowledge, because they rely, e.g., on the model's ability to follow specific for

Automated Pronunciation Evaluation for Korean Toddler Speech using Speech Diarization and Self-Supervised Learning

arXiv:2606.10213v1 Announce Type: cross Abstract: Speech sound disorders affect approximately 44% of Korean pediatric communication disorder cases, yet automated assessment tools for Korean toddler sp

10 Jun 2026

LLM-Based Code Documentation Generation and Multi-Judge Evaluation

arXiv:2606.09852v1 Announce Type: cross Abstract: High-quality source code documentation is vital yet often neglected, especially in critical domains like healthcare where reliability and maintainabil

10 Jun 2026

MMClima: A Framework for Multimodal Climate Science Data and Evaluation

arXiv:2606.10194v1 Announce Type: cross Abstract: Climate change research increasingly requires AI systems that reason across text, dynamic visual content, and scientific figures, yet existing climate

10 Jun 2026

Activation Steering Induces Emergent Misalignment: A More Comprehensive Evaluation

arXiv:2606.08682v1 Announce Type: cross Abstract: Activation steering has emerged as a popular inference-time technique for modulating the behavior of large language models (LLMs). By constructing a s

Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

arXiv:2606.08840v1 Announce Type: new Abstract: Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance va

Evaluation of ML Resource Utilization Requires Model Life Cycle Assessment

arXiv:2606.07632v1 Announce Type: new Abstract: Proper accounting of the energy requirements and environmental impact of artificial intelligence (AI) systems is necessary for researchers, developers,

safetyarxiv-cs-lg

LATTEArena: An Evaluation Framework for LLM-powered Tabular Feature Engineering (Extended Version)

arXiv:2606.09004v1 Announce Type: new Abstract: Feature engineering remains essential for tabular data analysis, and Large Language Models (LLMs) have emerged as a promising paradigm for automating th

Multi-Turn Evaluation of Deep Research Agents Under Process-Level Feedback

arXiv:2606.09748v1 Announce Type: new Abstract: Existing benchmarks for deep research agents (DRAs) assess only single-shot outputs, ignoring a key question: can DRAs improve their reports when guided

agentsarxiv-cs-ai

Adaptive Band Selection for Hyperspectral Classification with Spatially Disjoint Evaluation

arXiv:2606.06684v1 Announce Type: new Abstract: Hyperspectral band selection methods based on differentiable selectors can be sensitive to initialization and to extracting a final discrete subset, whi

researcharxiv-cs-cv

8 Jun 2026

RECAP: Regression Evaluation for Continual Adaptation of Prompts

model-releasesarxiv-cs-cl

arXiv:2606.06698v1 Announce Type: cross Abstract: Production agentic systems routinely face evolving constraints and must comply from the very next interaction. Scenarios like a tool-call notification

8 Jun 2026

REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference

arXiv:2606.07141v1 Announce Type: cross Abstract: Language models trained for clinical disease inference are trained on patient data, which may include sensitive and private information, and data owne

8 Jun 2026

Do More Agents Help? Controlled and Protocol-Aligned Evaluation of LLM Agent Workflows

arXiv:2606.05670v1 Announce Type: new Abstract: Does adding more agents help an LLM workflow once compared systems share the same benchmark loader, tool access, answer contract, usage accounting, and

6 Jun 2026

SagnacAssisted Enhanced OTDR for Distributed Acoustic Sensing: A Standardized Benchmark and Engineering Evaluation Framework

arXiv:2606.05754v1 Announce Type: cross Abstract: Phase-sensitive optical time-domain reflectometry (phi-OTDR) is widely used in large-scale distributed acoustic sensing (DAS) because it provides dist

6 Jun 2026

SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualization

arXiv:2606.05525v1 Announce Type: new Abstract: Recent advances in agentic visualization have enabled the translation of natural language into executable scientific visualization (SciVis) workflows. W

6 Jun 2026

M^3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks

arXiv:2606.05008v1 Announce Type: cross Abstract: As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developin

4 Jun 2026

Tutorials

A Systematic Evaluation of Current Architectures in Wind Power Forecasting

arXiv:2606.02849v1 Announce Type: new Abstract: Interval wind speed forecasting is essential for the efficient integration of wind energy into power systems, as it accounts for the inherent uncertaint

tutorialsarxiv-cs-lg

3 Jun 2026

GN0: Toward a Unified Paradigm for Generation, Evaluation, and Policy Learning in Visual-Language Navigation

model-releasesarxiv-cs-ro

arXiv:2606.03682v1 Announce Type: new Abstract: Embodied navigation connects intelligent agents with the physical world and is fundamental for general robotic intelligence. Limited availability and qu

3 Jun 2026

Inverting the Generation Process of Denoising Diffusion Implicit Models: Empirical Evaluation and a Novel Method

arXiv:2606.03111v1 Announce Type: new Abstract: This paper studies the problem of inverting the DDIM image generation process to recover latent variables, particularly the initial noise map, from a ge

researcharxiv-cs-cv

3 Jun 2026

A Modelling and Evaluation Framework for EuroCrops-Driven Sentinel-2 Crop Segmentation

arXiv:2606.00676v1 Announce Type: new Abstract: This work presents a configurable pipeline for generating semantic-segmentation-ready agricultural datasets from Sentinel-2 imagery and EuroCrops parcel

safetyarxiv-cs-cv

Community-Aware Assessment of Social Textual Engagement and Resonance: A Human-Centric Perspective on User-Generated Content Evaluation

arXiv:2606.01897v1 Announce Type: new Abstract: Traditional Video Quality Assessment (VQA) focuses narrowly on aesthetic fidelity, overlooking the complex social dynamics that define quality in User-G

IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

arXiv:2606.01260v1 Announce Type: cross Abstract: Despite being home to more than 1300 ethnic groups and 700 indigenous languages, bias in Large Language Models has not been fully studied in Indonesia

SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding

arXiv:2606.00021v1 Announce Type: cross Abstract: Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which a

WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World

model-releasesarxiv-cs-cv

arXiv:2512.10958v2 Announce Type: replace Abstract: Generative world models are reshaping embodied AI, enabling agents to synthesize realistic 4D driving environments that look convincing but often fa

How well does Classification Accuracy capture Concept Drift Detection Quality? An overview of Concept Drift Detection evaluation

arXiv:2605.31186v1 Announce Type: new Abstract: Data streams are nowadays among the most frequently analyzed data structures, with the concept drift posing a major challenge encountered by processing

researcharxiv-cs-lg

1 Jun 2026

When Are Multimodal Predictions Biologically Supported? A Diagnostic Evaluation Framework

arXiv:2605.31504v1 Announce Type: new Abstract: Multimodal models in oncology can produce accurate predictions, but accurate prediction does not reveal whether the model has learned biology that is sh

researcharxiv-cs-lg

1 Jun 2026

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation

arXiv:2605.28830v1 Announce Type: cross Abstract: As Large Language Models (LLMs) are increasingly deployed in safety-critical applications, robust content moderation becomes essential. We present a c

Comparative evaluation of photogrammetric reconstruction methods and 3D Gaussian Splatting for road surface roughness analysis

arXiv:2605.29452v1 Announce Type: new Abstract: Image-based 3D reconstruction offers a low-cost alternative to traditional sensor-based techniques for road surface assessment. This study compares four

safetyarxiv-cs-cv

Resolution Diagnostics for Paired LLM Evaluation

arXiv:2605.30315v1 Announce Type: new Abstract: Across two public LLM leaderboards, many displayed pairwise rankings do not meet a conventional paired-test resolution target under the actual paired ev

researcharxiv-cs-cl

Rethinking Literature Search Evaluation: Deep Research Helps, and Human Citation Lists Are Not a Ground Truth

arXiv:2605.29234v1 Announce Type: new Abstract: We study large-scale literature search from two complementary angles: improving the retrieval pipeline, and stress-testing the human reference list as a

When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis

arXiv:2605.29025v1 Announce Type: new Abstract: Federal agencies are deploying large language models (LLMs) to categorize public comment corpora, where the model's organization of the record shapes wh

A Unified Framework for the Evaluation of LLM Agentic Capabilities

arXiv:2605.27898v1 Announce Type: new Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential. However, reported benchmark scores

28 May 2026

Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability

arXiv:2605.28602v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for tasks that implicitly reduce to Boolean satisfiability (SAT), yet their reasoning ability on SAT

28 May 2026

EEG-FM-Audit: A Systematic Evaluation and Analysis Pipeline for EEG Foundation Models

arXiv:2605.26910v1 Announce Type: cross Abstract: Large EEG Foundation Models (FMs) have shown great potential for decoding EEG signals across diverse cognitive tasks. However, existing EEG-FM studies

27 May 2026

MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation

arXiv:2605.27366v1 Announce Type: new Abstract: Large language model (LLM) agents rely on reusable skills to solve complex tasks. However, existing skill creation approaches treat skills as isolated a

agentsarxiv-cs-ai

27 May 2026

Tutorials

An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods

arXiv:2605.24298v1 Announce Type: cross Abstract: The growing use of Large Language Models (LLMs) for automated code generation has enhanced software development efficiency, but often at the cost of s

tutorialsarxiv-cs-ai

Faithfulness Metrics Don't Measure Faithfulness: A Meta-Evaluation with Ground Truth

model-releasesarxiv-cs-cl

arXiv:2605.25052v1 Announce Type: new Abstract: Chains of thought (CoTs) have become central in interpreting and auditing behaviors of large language models. Yet growing evidence suggests that these t

HoloFair: Unified T2I Fairness Evaluation and Fair-GRPO Debiasing

arXiv:2605.24687v1 Announce Type: cross Abstract: Text-to-Image (T2I) models have made significant strides in visual realism and semantic consistency, yet they often perpetuate and amplify societal bi

Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

arXiv:2605.24247v1 Announce Type: cross Abstract: Many automated labeling pipelines classify inputs into categories defined by a written specification, content moderation being a prominent use case. S

safetyarxiv-cs-ai

MAPLE: Multi-State Aggregated Policy Evaluation for AlphaZero in Imperfect-Information Games

arXiv:2605.24139v1 Announce Type: new Abstract: Imperfect-information games (IIGs) are challenging, as players must make decisions without fully observing the true game state. While AlphaZero has achi

safetyarxiv-cs-ai

Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions

arXiv:2605.25073v1 Announce Type: cross Abstract: Background: Fine-tuning is central to adapting pre-trained Large Language Models (LLMs) to downstream tasks, but its reliance on training data, parame