Search: “evaluation”

AMUSE: Anytime Muon with Stable Gradient Evaluation

arXiv:2605.22432v1 Announce Type: new Abstract: Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimiz

23 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study

DGX agent

arXiv:2605.21566v1 Announce Type: new Abstract: Machine learning models for chronic kidney disease (CKD) risk prediction often post strong discrimination scores on internal test sets. Calibration and

researcharxiv-cs-lg

23 May 2026

Evaluation of Chunking Strategies for Effective Text Embedding in Low-Resource Language on Agricultural Documents

arXiv:2605.22203v1 Announce Type: new Abstract: In this study, we compare the performance of four text chunking approaches: Recursive, Khmer-Aware, Sentence-Based, and LLM-Based within a Retrieval-Aug

22 May 2026

Agents

STRUCTSENSE: A Task-Agnostic Agentic Framework for Structured Information Extraction with Human-In-The-Loop Evaluation and Benchmarking

arXiv:2507.03674v3 Announce Type: replace Abstract: Extracting structured information from scientific literature is critical for accelerating discovery, yet Large Language Models (LLMs) often struggle

agentsarxiv-cs-cl

22 May 2026

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

arXiv:2602.19320v2 Announce Type: replace Abstract: Agentic memory systems enable large language model (LLM) agents to maintain state across long interactions, supporting long-horizon reasoning and pe

21 May 2026

ECUAS{n}: A family of metrics for principled evaluation of uncertainty-augmented systems

model-releasesarxiv-cs-lg

arXiv:2605.20490v1 Announce Type: cross Abstract: In high-stakes automated decision-making, access to predictive uncertainty is essential for enabling users -- human or downstream systems -- to accept

21 May 2026

Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media

arXiv:2605.21338v1 Announce Type: new Abstract: LLMs have demonstrated exceptional proficiency in a wide range of NLP tasks. However, a notable gap remains in practical data analysis scenarios, partic

21 May 2026

A Nonlinear Complexity Index for Wearable PPG Cardiovascular Stability: Multiscale Validation, Systematic Evaluation Correction, and Bayesian Parameter Optimization

arXiv:2605.18802v1 Announce Type: cross Abstract: Cardiovascular stability estimation from wearable photoplethysmography (PPG) requires a principled nonlinear framework, yet major gaps persist in heur

20 May 2026

Interoceptive Divergence in Aesthetic Evaluation and Implications for Human-AI Alignment

arXiv:2605.18759v1 Announce Type: cross Abstract: Artificial intelligence (AI), exemplified by large language models (LLMs), is rapidly approaching and in some cases surpassing human performance acros

safetyarxiv-cs-ai

20 May 2026

White-Balance First, Adjust Later: Cross-Camera Color Constancy via Vision-Language Evaluation

arXiv:2605.19613v1 Announce Type: new Abstract: Color constancy aims to keep object colors consistent under varying illumination. Cross-camera generalization in color constancy remains challenging bec

20 May 2026

Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

arXiv:2605.16996v1 Announce Type: new Abstract: Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To i

19 May 2026

Local Ai

PROTEA: Offline Evaluation and Iterative Refinement for Multi-Agent LLM Workflows

arXiv:2605.18032v1 Announce Type: cross Abstract: Multi-agent LLM workflows -- systems composed of multiple role-specific LLM calls -- often outperform single-prompt baselines, but they remain difficu

local-aiarxiv-cs-ai

19 May 2026

A Causally Grounded Taxonomy for Image Degradation Robustness Evaluation

model-releasesarxiv-cs-cv

arXiv:2605.15906v1 Announce Type: new Abstract: Image degradations can occur during acquisition, processing, and transmission, altering visual appearance and affecting downstream vision tasks. They ar

A Statistical Analysis for Per-Instance Evaluation of Stochastic Optimizers: Avoiding Unreliable Conclusions

arXiv:2503.16589v2 Announce Type: replace Abstract: A key trait of stochastic optimizers is that multiple runs of the same optimizer in attempting to solve the same problem can produce different resul

Community-aware evaluation and threshold calibration for open-set plankton image recognition

arXiv:2605.15835v1 Announce Type: new Abstract: Automated plankton image recognition is increasingly used in aquatic ecosystem monitoring, but deployed classifiers inevitably encounter unseen taxa and

Applications

Does Theory of Mind Improvement Really Benefit Human-AI Interactions? Empirical Findings from Interactive Evaluations

arXiv:2605.15205v1 Announce Type: new Abstract: Improving the Theory of Mind (ToM) capability of Large Language Models (LLMs) is crucial for effective social interactions between these AI models and h

applicationsarxiv-cs-ai

Structure-BiEval: A Self-Supervised, Dual-Track Framework for Decoupling Structure and Content in LLM Evaluation for Web Information Systems

arXiv:2601.19923v2 Announce Type: replace-cross Abstract: As Large Language Models (LLMs) evolve into the core of Web-based autonomous agents and complex Web Information Systems, their ability to fait

A Systematic Evaluation of Imbalance Handling Methods in Biomedical Binary Classification

arXiv:2605.14147v1 Announce Type: new Abstract: Objective: The primary goal of this study was to systematically examine the impact of commonly used imbalance handling methods (IHMs) on predictive perf

15 May 2026

SR-Prominence: A Crowdsourced Protocol and Dataset Suite for Perceptually-Weighted Super-Resolution Artifact Evaluation

model-releasesarxiv-cs-cv

arXiv:2605.14847v1 Announce Type: new Abstract: Modern image super-resolution methods generate detailed, visually appealing results, but they often introduce visual artifacts: unnatural patterns and t

15 May 2026

Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents

arXiv:2605.12894v1 Announce Type: new Abstract: Large Language Model (LLM) agents are increasingly deployed in settings where they interact with a wide variety of people, including users who are uncle

researcharxiv-cs-ai

High-Order Epistasis Detection Using Factorization Machine with Quadratic Optimization Annealing and MDR-Based Evaluation

arXiv:2601.01860v2 Announce Type: replace Abstract: Detecting high-order epistasis is a fundamental challenge in genetic association studies due to the combinatorial explosion of candidate locus combi

Improving Reproducibility in Evaluation through Multi-Level Annotator Modeling

arXiv:2605.13801v1 Announce Type: cross Abstract: As generative AI models such as large language models (LLMs) become more pervasive, ensuring the safety, robustness, and overall trustworthiness of th

safetyarxiv-cs-ai

Large Language Models for Agentic NetOps and AIOps: Architectures, Evaluation, and Safety

arXiv:2605.12729v1 Announce Type: cross Abstract: Large language models are increasingly being used to support network operations (NetOps) and artificial intelligence for IT operations (AIOps), includ

safetyarxiv-cs-ai

Can Nano Banana 2 Replace Traditional Image Restoration Models? An Evaluation of Its Performance on Image Restoration Tasks

arXiv:2604.03061v2 Announce Type: replace Abstract: Recent advances in generative AI raise the question of whether general-purpose image editing models can serve as unified solutions for image restora

DeconDTN-Toolkit: A Library for Evaluation and Enhancement of Robustness to Provenance Shift

arXiv:2605.11237v1 Announce Type: new Abstract: Despite the burgeoning body of work on distribution shifts, provenance shift-where the relationship between data source and label changes at deployment-

How Does Differential Privacy Affect Social Bias in LLMs? A Systematic Evaluation

arXiv:2605.11195v1 Announce Type: new Abstract: Large language models (LLMs) trained on web-scale corpora can memorize sensitive training data, posing significant privacy risks. Differential privacy (

safetyarxiv-cs-cl

MEME: Multi-entity & Evolving Memory Evaluation

arXiv:2605.12477v1 Announce Type: cross Abstract: LLM-based agents increasingly operate in persistent environments where they must store, update, and reason over information across many sessions. Whil

Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering

arXiv:2605.12313v1 Announce Type: new Abstract: Multi-hop question answering (QA) remains a significant challenge in the biomedical domain, requiring systems to integrate information across multiple s

Beyond the Singular: Revealing the Value of Multiple Generations in Benchmark Evaluation

arXiv:2502.08943v4 Announce Type: replace-cross Abstract: Large language models (LLMs) have demonstrated significant utility in real-world applications, exhibiting impressive capabilities in natural l

Can Agent Benchmarks Support Their Scores? Evidence-Supported Bounds for Interactive-Agent Evaluation

arXiv:2605.10448v1 Announce Type: new Abstract: Interactive agent benchmarks map an agent run to a binary outcome through outcome checks. When these checks rely on surface level signals or fail to cap

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents

arXiv:2605.09679v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) and AI agents have made significant progress in learning to analyze and reason about clinical images. However, e

Defense effectiveness across architectural layers: a mechanistic evaluation of persistent memory attacks on stateful LLM agents

arXiv:2605.08442v1 Announce Type: cross Abstract: Persistent memory attacks against LLM agents achieve high attack success rates against open-source models. In these attacks, malicious instructions in

researcharxiv-cs-ai

Generating Leakage-Free Benchmarks for Robust RAG Evaluation

arXiv:2605.08838v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets,

Measure what Matters: Psychometric Evaluation of AI with Situational Judgment Tests

arXiv:2510.22170v2 Announce Type: replace Abstract: Persona conditioning is widely used to steer large language model (LLM) behavior, but it is unclear whether it induces stable behavioral structure o

MolRGen: A Training and Evaluation Setting for De Novo Molecular Generation with Reasonning Models

arXiv:2603.18256v2 Announce Type: replace-cross Abstract: Recent reasoning-based large language models have shown strong performance on tasks with verifiable outcomes, but their use in de novo molecul

Local Ai

Privacy-Preserving Distributed Learning in IoT Systems: A Unified Threat Model and Evaluation Framework

arXiv:2605.09232v1 Announce Type: cross Abstract: The increasing deployment of Internet-of-Things (IoT) devices has accelerated the use of distributed learning frameworks, where data remains local whi

local-aiarxiv-cs-lg

Applications

Rethinking Evaluation of Multiple Sclerosis (MS) Lesion Segmentation Models

arXiv:2605.09666v1 Announce Type: cross Abstract: Multiple Sclerosis (MS) is a chronic autoimmune disease that can significantly reduce the quality of life of a patient. Existing treatment options can

applicationsarxiv-cs-ai

Slum Detection and Density Mapping with AlphaEarth Foundations: A Representation Learning Evaluation Across 12 Global Cities

arXiv:2605.10029v1 Announce Type: new Abstract: Pixel-level slum mapping has long been constrained by limited cross-city generalisation, the absence of continuous density estimation, and weak global c

When Style Similarity Scores Fail: Diagnosing Raw CSD Cosine in Artist-Style Evaluation

arXiv:2605.09030v1 Announce Type: new Abstract: Raw cosine in the 768-dimensional output space of the Contrastive Style Descriptor (CSD) is now widely read as an absolute, calibrated style-fidelity sc

WildClawBench: A Benchmark for Real-World, Long-Horizon Agent Evaluation

arXiv:2605.10912v1 Announce Type: new Abstract: Large language and vision-language models increasingly power agents that act on a user's behalf through command-line interface (CLI) harnesses. However,

Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

arXiv:2510.07926v2 Announce Type: replace Abstract: Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce o

Discovering Ordinary Differential Equations with LLM-Based Qualitative and Quantitative Evaluation

arXiv:2605.07323v1 Announce Type: new Abstract: Discovering governing differential equations from observational data is a fundamental challenge in scientific machine learning. Existing symbolic regres

Streaming Adversarial Robustness in Fuzzy ARTMAP: Mechanism-Aligned Evaluation, Progressive Training, and Interpretable Diagnostics

arXiv:2605.06902v1 Announce Type: new Abstract: Adversarial robustness has been studied extensively for offline deep networks, but less is known about strict single-pass streaming neural learners. Thi

Text-to-CAD Evaluation with CADTests

arXiv:2605.07807v1 Announce Type: cross Abstract: Text-to-CAD has recently emerged as an important task with the potential to substantially accelerate design workflows. Despite its significance, there

The Single-File Test: A Longitudinal Public-Interface Evaluation of First-Output LLM Web Generation with Social Reach Tracking

arXiv:2605.06707v1 Announce Type: cross Abstract: This paper presents an eight-week observational comparison of 68 single-file HTML generations collected across 17 public experiments in the 'HTML AI B

Are Multimodal LLMs Ready for Clinical Dermatology? A Real-World Evaluation in Dermatology

model-releasesarxiv-cs-cv

arXiv:2605.04098v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have demonstrated promise on publicly available dermatology benchmarks. However, benchmark performance may not

7 May 2026

Stop Automating Peer Review Without Rigorous Evaluation

arXiv:2605.03202v1 Announce Type: new Abstract: Large language models offer a tempting solution to address the peer review crisis. This position paper argues that today's AI systems should not be used

researcharxiv-cs-ai

7 May 2026

Towards Self-Referential Analytic Assessment: A Profile-Based Approach to L2 Writing Evaluation with LLMs

arXiv:2605.04298v1 Announce Type: new Abstract: Automated essay scoring (AES) research often relies on rank-based correlation metrics to validate analytic assessment. However, such metrics obscure bot