Search: “evaluation”

Scientific discovery as meta-optimization: a combinatorial optimization case study

arXiv:2606.26728v1 Announce Type: new Abstract: Scientific discovery is fundamentally an optimization problem, defined by a vast 'state space' of theories and experiments, and an evaluation criterion

applicationsarxiv-cs-ai

26 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Randomized Kriging Believer for Parallel Bayesian Optimization with Regret Bounds

DGX agent

arXiv:2603.01470v3 Announce Type: replace Abstract: We consider the optimization problem of an expensive-to-evaluate black-box function, in which we can obtain noisy function values in parallel. For t

researcharxiv-cs-lg

25 Jun 2026

MedBench v5: A Dynamic, Process-Oriented, and Hallucination-Aware Benchmark for Clinical Multimodal Models

model-releasesarxiv-cs-cl

arXiv:2606.24155v1 Announce Type: new Abstract: Existing medical AI benchmarks lack process visibility, atomic skill evaluation, and integrated hallucination detection. We introduce MedBench v5, a red

24 Jun 2026

Quantifying Prior Dominance in RAG Systems

arXiv:2606.23695v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) grounds Large Language Models in external knowledge, yet current evaluations rely on discrete heuristics that suf

researcharxiv-cs-ai

24 Jun 2026

Same Lesson, Different Story: Cross-Lingual Reconstruction of Cultural Narratives in Large Language Models

arXiv:2606.24610v1 Announce Type: new Abstract: The evaluation of cultural grounding context becomes complex when multiple cultures convey the same moral lesson. This challenge is particularly relevan

researcharxiv-cs-cl

24 Jun 2026

A Generative Model for Closed-Loop Microsimulation of Signalized Intersections

arXiv:2606.23588v1 Announce Type: new Abstract: Traffic microsimulators rely on hand-crafted behavior models that reproduce aggregate flow but miss the heterogeneous interactions between vehicles at s

safetyarxiv-cs-ro

Agentic Time Machine as an Infrastructure for Future-Event Forecasting

arXiv:2606.21013v1 Announce Type: cross Abstract: Forecasting future events is a critical challenge for large language model (LLM) agents, spanning domains from elections and monetary policy to financ

safetyarxiv-cs-lg

Gradient-Descent Steps to Success over Mean Accuracy: A Paradigm Shift for ML

arXiv:2606.22053v1 Announce Type: new Abstract: Traditional evaluation of machine learning (ML) models typically focuses on achieving the maximum possible accuracy irrespective of the computational co

HEAS: Hierarchical Evolutionary Agent-Based Simulation Framework for Multi-Objective Policy Search

arXiv:2508.15555v4 Announce Type: replace-cross Abstract: HEAS is a Python framework that connects agent-based simulation, evolutionary search, and scenario-based evaluation in a single reproducible p

safetyarxiv-cs-lg

Kiwano: A Cutting-Edge Open-Source Toolkit for Speaker Verification

arXiv:2606.22369v1 Announce Type: cross Abstract: In this paper, we present Kiwano, an open-source toolkit designed to advance research and evaluation for speaker verification. Kiwano provides a light

MMOU: A Massive Multi-Task Omni Understanding and Reasoning Benchmark for Long and Complex Real-World Videos

model-releasesarxiv-cs-cv

arXiv:2603.14145v2 Announce Type: replace-cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in visual and audio understanding when evaluated in isolation. However,

Specialize Roles, Mix Deployments: Pushing the Cost-Accuracy Frontier of LLM Agent Teams

model-releasesarxiv-cs-lg

arXiv:2606.20629v1 Announce Type: cross Abstract: LLM agents are increasingly deployed as multi-role teams, where tasks are divided across specialized roles such as planner, executor, and verifier. In

When Is an LLM Worth It for Hyperparameter Optimization? A Budget-Matched Study on Tabular Data Finds the Warm-Start Is a Default Configuration, Not the Model

arXiv:2606.21641v1 Announce Type: new Abstract: Large language models (LLMs) have been proposed as hyperparameter-optimization (HPO) advisors that 'warm-start' search from prior knowledge, proposing s

CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detection

model-releasesarxiv-cs-cv

arXiv:2606.11231v1 Announce Type: new Abstract: Vision-language reinforcement learning has recently shown strong target-present localization for camouflaged object detection (COD). Yet localization is

11 Jun 2026

Intelligent Automation for Embodied Benchmark Construction: Pipelines, Embodiments, Simulators, and Trends

arXiv:2606.12207v1 Announce Type: cross Abstract: Embodied intelligence now spans navigation, household assistance, manipulation, autonomous driving, aerial agents, and multimodal large-model control.

11 Jun 2026

Reassessing High-Performing LLMs on Polish Medical Exams: True Competence or Bias-Driven Performance?

model-releasesarxiv-cs-cl

arXiv:2606.12250v1 Announce Type: new Abstract: Large language models (LLMs) in medicine are mainly evaluated using multiple-choice question answering (MCQA), which can overestimate real clinical abil

11 Jun 2026

An (epsilon,elta)-accurate level set estimation with a stopping criterion

arXiv:2503.20272v2 Announce Type: replace-cross Abstract: The level set estimation problem seeks to identify regions within a set of candidate points where an unknown and costly to evaluate function's

Automated Scoring of Arabic Text Using Large Language Models: A Literature Review

arXiv:2606.09830v1 Announce Type: new Abstract: In modern educational systems, Automatic Text Scoring (ATS) plays a central role by enabling scalable and consistent evaluation of learner responses wit

safetyarxiv-cs-cl

Is Fairness Truly Fair? Towards Reliable Lipschitz Fairness in Multi-Task Learning via Fixed-exorpdfstring{elta}{delta} Alignment

arXiv:2606.10632v1 Announce Type: cross Abstract: Lipschitz-style individual fairness formalizes the idea that semantically similar examples should receive similar predictions, but its evaluation in m

When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking

arXiv:2606.10287v1 Announce Type: cross Abstract: Evaluating Knowledge Graph Completion (KGC) models remains challenging because standard assessment relies on isolated rank-based metrics such as MRR,

researcharxiv-cs-cl

Agents

A multi-agent system for spine MRI report generation from multi-sequence imaging

arXiv:2606.08897v1 Announce Type: cross Abstract: Spinal pathology is a leading cause of pain and disability worldwide. Spine MRI is central to clinical evaluation, yet its interpretation remains comp

agentsarxiv-cs-ai

Context Over Compute Human-in-the-Loop Outperforms Iterative Chain-of-Thought Prompting in Interview Answer Quality

arXiv:2603.09995v2 Announce Type: replace-cross Abstract: Behavioral interview evaluation using large language models presents unique challenges that require structured assessment, realistic interview

Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

arXiv:2606.09409v1 Announce Type: new Abstract: Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they rewar

Cranio-Diff: Diffusion-based Cross-domain Craniofacial Reconstruction with 2D X-ray Skull Guidance and Structural Identity Constraints

arXiv:2606.09699v1 Announce Type: new Abstract: The state-of-the-art generative models, such as CycleGAN, Pix2Pix, and diffusion models have demonstrated remarkable performance in the face generation

safetyarxiv-cs-cv

Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis

arXiv:2606.09178v1 Announce Type: cross Abstract: Multilingual safety evaluation of large language models (LLMs) has predominantly relied on direct translation (DT) of English benchmarks into target l

Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs

arXiv:2606.09038v1 Announce Type: new Abstract: Large Language Models (LLMs) have enabled increasingly personalized interactions by adapting to users' preferences, contexts, and long-term histories. H

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

arXiv:2602.15327v2 Announce Type: replace-cross Abstract: Machine learning model performance improvements tend to arise from competition and application. For deployment, we consider prescriptive scali

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

arXiv:2606.07591v1 Announce Type: cross Abstract: AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We presen

SLMJury: Can Small Language Models Judge as Well as Large Ones?

arXiv:2606.07810v1 Announce Type: cross Abstract: Large language models (LLMs) are widely used as judges for evaluating model outputs, but their high cost, latency, and opacity limit scalability. We i

UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL

arXiv:2606.08018v1 Announce Type: new Abstract: Existing text-to-SQL benchmarks are largely centered on SQLite, making it difficult to evaluate whether models can generalize across heterogeneous SQL d

Accelerating Reproducible Research in Synthetic EHR Generation

arXiv:2606.06990v1 Announce Type: new Abstract: The generation of high-fidelity synthetic Electronic Health Records (EHR) is crucial for advancing medical research while preserving patient privacy. Ho

GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting

model-releasesarxiv-cs-lg

arXiv:2606.06881v1 Announce Type: new Abstract: Blood glucose forecasting models are foundational for modern diabetes management systems, as reliable short-term predictions can enable proactive interv

PaperFlow: Profiling, Recommending, and Adapting Across Daily Paper Streams

arXiv:2606.07454v1 Announce Type: cross Abstract: Scientific paper recommendation is typically evaluated as static ranking over a fixed candidate set, yet real scientific reading unfolds as a daily, l

ReclAIm: A Multi-Agent Framework for Monitoring and Correcting Performance Decline in Medical Imaging AI

arXiv:2510.17004v2 Announce Type: replace-cross Abstract: Purpose: To develop and evaluate a multi-agent framework (ReclAIm) for automated monitoring, detection, and correction of performance decline

When Surface Form Changes Moderation Decisions: A Paired Study of Code-Mixed Workflow Instability

arXiv:2606.05654v1 Announce Type: cross Abstract: Hate moderation is often evaluated as classification on clean English inputs, but deployed systems must route content to actions such as ALLOW, FLAG,

researcharxiv-cs-ai

6 Jun 2026

RiskFlow: Fast and Faithful Safety-Critical Traffic Scenario Generation

arXiv:2606.06423v1 Announce Type: new Abstract: Safety-critical traffic scenario generation is essential for evaluating autonomous driving systems under rare but high-risk interactions. Existing diffu

safetyarxiv-cs-ro

5 Jun 2026

Expectations vs. Realities: The Cost of MSE-Optimal Forecasting Under Conditional Uncertainty

arXiv:2606.04342v1 Announce Type: cross Abstract: Multi-step time series forecasting (MSF) is commonly evaluated using point-wise error metrics such as mean squared error (MSE), implicitly treating th

applicationsarxiv-cs-ai

GENEB: Why Genomic Models Are Hard to Compare

model-releasesarxiv-cs-cl

arXiv:2606.04525v1 Announce Type: new Abstract: Progress in genomic foundation models is difficult to assess due to fragmented benchmarks, incompatible evaluation protocols, and task-specific reportin

Measuring Model Robustness via Fisher Information: Spectral Bounds, Theoretical Guarantees, and Practical Algorithms

arXiv:2606.04767v1 Announce Type: cross Abstract: The robustness of deep neural networks is crucial for safety-critical deployments, yet existing evaluation methods are often attack-dependent and lack

safetyarxiv-cs-cv

TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering

arXiv:2602.06911v2 Announce Type: replace-cross Abstract: As increasingly capable open-weight large language models (LLMs) are deployed, improving their tamper resistance against unsafe modifications,

Benchmarking Speech-to-Speech Translation Models

arXiv:2606.03241v1 Announce Type: new Abstract: Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets

applicationsarxiv-cs-cl

COD10K-C: Benchmarking Robustness of Camouflaged Object Detection Under Natural Image Corruptions

model-releasesarxiv-cs-cv

arXiv:2606.02603v1 Announce Type: new Abstract: Camouflaged object detection has improved substantially, but most standard benchmarks evaluate models only on clean images. This is not realistic becaus

Agents

Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks

arXiv:2606.02875v1 Announce Type: new Abstract: Coding-agent benchmarks evaluate whether a single uninterrupted agent can resolve a repository issue. Real software work is messier: tasks are interrupt

agentsarxiv-cs-ai

NewtPhys: Do Foundation Models Understand Newtonian Physics?

arXiv:2606.03986v1 Announce Type: new Abstract: Previous work has evaluated physics reasoning in foundation models using synthetic or semi-synthetic scenes and visual question-answering tasks. However

applicationsarxiv-cs-cv

ProtocolBench: Which LLM MultiAgent Protocol to Choose?

arXiv:2510.17149v3 Announce Type: replace Abstract: As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and

Local Ai

Scalable Derivative Gaussian Processes via Exact Gradient Reduction

arXiv:2606.02909v1 Announce Type: cross Abstract: Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations

local-aiarxiv-cs-lg

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

arXiv:2606.03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment. Statistical tools for de

X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes

arXiv:2603.05290v2 Announce Type: replace Abstract: Large language models (LLMs) achieve promising performance, yet their ability to reason remains poorly understood. Existing evaluations largely emph

researcharxiv-cs-ai