Search: “evaluation”

SciHorizon-eLab: An Agentic Protocol-to-Task Compiler for Scalable Benchmarking of Scientific Embodied Agents

arXiv:2609.30971v1 Announce Type: new Abstract: Embodied agents offer a promising route to automating scientific experimentation, yet their progress is constrained by the lack of reliable and systemat

28 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Towards Understanding LLM-Based Log Anomaly Detection: An Empirical Study of Performance, Efficiency, and Robustness

DGX agent

arXiv:2609.31371v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated promising performance in log anomaly detection, yet how their adaptation strategies, architectures, and d

model-releasesarxiv-cs-lg

28 Sept 2026

When 10,000 Windows Are Not 10,000 Tests: Auditing Statistical Confidence in Sliding-Window Time-Series Classification

arXiv:2609.30721v1 Announce Type: new Abstract: Sliding-window classifiers are often evaluated on thousands of overlapping test windows, even though neighboring predictions share observations and rema

researcharxiv-cs-lg

28 Sept 2026

XPhysICS: Cross-Physical-Domain Threat Grounding for Industrial Control Systems Security

arXiv:2609.30805v1 Announce Type: cross Abstract: Industrial control system (ICS) threats documented for one plant can express cyber-physical effects relevant to another, but semantic similarity alone

28 Sept 2026

A Living Benchmark for Information Retrieval from Electronic Health Records

arXiv:2609.30205v1 Announce Type: new Abstract: Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how cli

Advancing Model Research in AgentX: Long-Horizon Autonomy for Industrial Recommender Systems

arXiv:2609.30001v1 Announce Type: new Abstract: Sustaining industrial recommendation research requires using the results of one experiment to decide what to investigate next. We present AgentX-Model,

Beyond Surface Style: Aligning Multi-Turn User Simulators with Behavioral Consistency

arXiv:2609.28690v1 Announce Type: new Abstract: Faithful user simulation is fundamental to building, evaluating, and improving interactive AI at scale. However, plausible individual responses do not e

Safety

Combining Evasive and Braking Reactions for Safety Reference Models in Automated Vehicles

arXiv:2609.29738v1 Announce Type: new Abstract: Computational models of careful and competent human drivers are essential for scenario-based evaluation of automated driving systems (ADS). However, mos

safetyarxiv-cs-ro

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

arXiv:2603.18612v2 Announce Type: replace Abstract: We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev an

Does DCGAN-Based Synthetic Augmentation Improve Brain Tumor MRI Classification? An Empirical Study

arXiv:2609.28508v1 Announce Type: cross Abstract: Generative adversarial networks (GANs) are increasingly used to augment medical imaging datasets, but synthetic images do not necessarily provide down

researcharxiv-cs-cv

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

arXiv:2609.28876v1 Announce Type: new Abstract: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questi

Safety

Human-AI-Powered Hypothesis Testing: Cost-Aware Selective AI Scoring and Sequential Human Escalation

arXiv:2609.28859v1 Announce Type: new Abstract: Large language models are increasingly used as inexpensive judges to evaluate outputs, label data, and assess whether a system meets a desired quality s

safetyarxiv-cs-ai

Learning to Ideate for Scientific Impact

arXiv:2609.29802v1 Announce Type: new Abstract: Scientific ideation is increasingly mediated by large language models, but current ideation systems are usually trained and evaluated on immediately jud

Reasoning Instructions Can Break Answer Decoding in Vision--Language Models

arXiv:2609.29278v1 Announce Type: cross Abstract: Chain-of-thought (CoT) instructions can distort multiple-choice VLM evaluation when a scorer appends a reasoning cue but reads answer-label logits bef

Sequential knowledge editing breaks a model's ability to tell good evidence from bad, without costing it accuracy

arXiv:2609.29587v1 Announce Type: new Abstract: Knowledge editing is evaluated on whether the edited fact changed, whether paraphrases follow, and whether unrelated answers stayed put. A model can pas

StepCOPS: Closed-Testing Lower-Tail Certificates for Language-Model Policy Selection

arXiv:2609.29549v1 Announce Type: new Abstract: Post-training pipelines must select one language-model policy from many checkpoints, prompts, and decoding rules. Mean evaluator scores can conceal rare

SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories

arXiv:2609.29465v1 Announce Type: new Abstract: Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks,

The Fellowship of the Query: Learning Retrieval Actions

arXiv:2609.28653v1 Announce Type: cross Abstract: Retrieval-augmented question answering requires control decisions about when to decompose a question, search, reformulate, extract evidence, synthesiz

Three Ways Classical Test Theory Misleads for LLM Judges

arXiv:2609.29709v1 Announce Type: cross Abstract: An LLM judge scores a bank of responses against a rubric, and the reliability comes back at 0.52. What has been measured? Judge evaluation has begun b

researcharxiv-cs-cl

Who Holds the Pen? Let Specifications, Not Agents, Sign Off

arXiv:2609.29921v1 Announce Type: new Abstract: Large language model agents increasingly combine generation, decision-making, execution, and self-evaluation within a single agentic loop. Although they

Building Socio-Affective Artificial Intelligence for Interactive Multi-Agent Simulations

arXiv:2609.26927v1 Announce Type: new Abstract: The objective of this article is to provide design principles and a software architecture for enabling interaction between humans and multiple agents in

Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark

arXiv:2609.28449v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in coding tasks, but their ability to reason about code execution remains unclear. Existing reposit

Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court

arXiv:2609.26945v1 Announce Type: new Abstract: Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the

Compressed delayed-information projection for six-degree-of-freedom underwater vehicle navigation under delayed acoustic positioning

arXiv:2609.27439v1 Announce Type: new Abstract: Delayed acoustic positioning packets constrain historical navigation states, but a current-time update evaluates them against a mismatched state, wherea

researcharxiv-cs-ro

EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

model-releasesarxiv-cs-lg

arXiv:2609.28149v1 Announce Type: cross Abstract: Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metr

KATOsuper: Surrogate-accelerated neural topology optimization with sensitivity-consistent Fourier neural operators

arXiv:2609.27216v1 Announce Type: cross Abstract: Topology optimization (TO) remains computationally intensive due to repeated finite element analysis (FEA) evaluations required at each iteration. Whi

Local Ai

LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations

arXiv:2609.28086v1 Announce Type: cross Abstract: We propose LAYERSCOPE, a label-free, layerwise framework that aims to characterize a model's learned representations in video and multimodal settings.

local-aiarxiv-cs-ai

MWE-ECL: Recoverable Long-Range Context Does Not Always Override Local Lexical Priors

arXiv:2609.27590v1 Announce Type: new Abstract: Long-context evaluations often test whether a model can recover distant evidence, but recoverability does not guarantee behavioral influence. We test th

Provably Complete Generalized Planning with LLMs

arXiv:2609.27105v1 Announce Type: new Abstract: Generalized planning aims to compute a plan that solves all instances of a planning domain. Recent work has used LLMs to automatically generate and debu

SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems

arXiv:2603.17373v2 Announce Type: replace Abstract: Large language models are rapidly being deployed as AI tutors, yet current evaluation paradigms assess problem-solving accuracy and generic safety i

Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning

arXiv:2603.08930v2 Announce Type: replace-cross Abstract: This paper introduces a benchmark for evaluating whether vision-language models (VLMs) can generate plant simulation configurations from image

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

arXiv:2508.13680v5 Announce Type: replace Abstract: We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) in

Auditing Proxy-Based Validation Across Text Spans

arXiv:2609.25808v1 Announce Type: cross Abstract: Evaluation scores are often validated by their agreement with inexpensive proxy labels. When the score and the proxy are computed from the same text s

Safety

Biased AI improves human performance but reduces perceived helpfulness

arXiv:2508.09297v4 Announce Type: replace-cross Abstract: Artificial intelligence (AI) increasingly shapes how people think, engage, and evaluate information. To minimize risk, most current systems ar

safetyarxiv-cs-ai

Detecting GPT-Assisted Writing Using Interpretable Stylometric Features

arXiv:2609.26687v1 Announce Type: new Abstract: Distinguishing GPT-assisted from independently authored student writing has become a critical challenge in academia. This paper evaluates the discrimina

researcharxiv-cs-cl

From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation

arXiv:2609.22664v1 Announce Type: cross Abstract: Research on large language model agents for penetration testing is evaluated almost entirely by capability: whether the agent captures a flag or repro

ISA-Bench: A Benchmark for Computational Reasoning Across Instruction Set Architectures

arXiv:2609.22878v1 Announce Type: new Abstract: Large language model code generation benchmarks primarily evaluate well-resourced languages like Python and Java, where models benefit from abundant tra

LLaVA-Assessor: Building the Foundation LMM For Visual Quality Assessment

arXiv:2609.26205v1 Announce Type: new Abstract: Aligning with the human visual system~(HVS) in perceiving and evaluating the quality of visual signals is a central objective of machine-vision-based vi

researcharxiv-cs-cv

Mitigating Sequential Reappearance in Diffusion Data-Point Unlearning

arXiv:2609.25166v1 Announce Type: new Abstract: Diffusion data-point unlearning is typically evaluated immediately after each deletion, even though subsequent requests may repeatedly update the same m

researcharxiv-cs-lg

NavSafe-infty: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments

model-releasesarxiv-cs-ro

arXiv:2609.26618v1 Announce Type: new Abstract: End-to-end (E2E) driving policies have progressed rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy withstands comp

PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI

arXiv:2609.23695v1 Announce Type: new Abstract: Recent advances in Physical AI have accelerated the use of foundation models in autonomous systems such as unmanned aerial vehicles (UAVs), which must p

RoboTwin-Phys: Do WAMs and VLAs Understand the Physical World?

model-releasesarxiv-cs-ro

arXiv:2609.26292v1 Announce Type: new Abstract: Physical-condition diversity is largely missing from current benchmarks for robot manipulation. While large-scale simulation benchmarks increasingly inc

SAMI3D-DW: Interactive Segmentation of Any 3D Medical Images

model-releasesarxiv-cs-cv

arXiv:2609.25743v1 Announce Type: new Abstract: Interactive segmentation of 3D medical images supports quantitative analysis of anatomical structures and disease while allowing users to specify and re

TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents

arXiv:2609.23363v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have led to the emergence of coding agents capable of performing complex engineering tasks, including re

Toward Auditable and Calibrated AI for Dementia-Related Crash Severity Prediction: A Selective Deferral Framework to Support Human Review

arXiv:2609.22694v1 Announce Type: new Abstract: Public crash databases increasingly support automated safety analysis, but crash severity prediction remains difficult to translate into public-sector d

TriWorldBench: A Tri-View Consistency Perspective on Embodied World Models

model-releasesarxiv-cs-ro

arXiv:2609.26314v1 Announce Type: new Abstract: Embodied world models predict the outcomes of robot actions to support learning and planning. For robots equipped with head and wrist cameras, this requ

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

arXiv:2609.12708v2 Announce Type: replace-cross Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whet

ALPINE: Adaptive Localization for Parameter- and Sample-Efficient Few-Shot Learning