Search: “evaluation”

Orchestrating LLM Agents for Scientific Research: A Pilot Study of Multiple Choice Question (MCQ) Generation and Evaluation

arXiv:2602.18891v2 Announce Type: replace-cross Abstract: Advances in large language models (LLMs) are rapidly transforming scientific work, yet empirical evidence on how these systems reshape researc

16 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Rubric Judgments Lack Uncertainty: Conformal Weighting for LLM Evaluation

DGX agent

arXiv:2610.21214v1 Announce Type: new Abstract: Rubric-based evaluation is widely used to assess large language model (LLM) outputs by decomposing quality into multiple criteria and assigning each a b

safetyarxiv-cs-ai

16 Oct 2026

Comparative Evaluation of Deep-Learning based DoA Estimators for Real-Time ISAC Beamforming

arXiv:2610.20326v1 Announce Type: new Abstract: Integrated Sensing and Communication (ISAC) systems rely on accurate Direction-of-Arrival (DoA) estimation for effective beamforming and spectral-effici

DivAlgo: A systematic evaluation of algorithmic diversity for code generation models

arXiv:2610.19157v1 Announce Type: cross Abstract: Code generation models are actively being used in professional and educational settings. However, the competence of large language models (LLMs) to ge

What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the ag

Whose Data, Whose Benchmark? A User-Centric Framework for Auditing Evaluation Resources

arXiv:2610.19268v1 Announce Type: new Abstract: Benchmarks shape how model capabilities are measured and interpreted, yet their scores can obscure whose conventions, contexts, and judgments define suc

Local Ai

Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluations

arXiv:2601.22318v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models lo

local-aiarxiv-cs-lg

PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

arXiv:2610.15722v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physic

Local Ai

ProWorld: Multi-View World Model with Proprioceptive Feedback for Closed-Loop Policy Evaluation

arXiv:2610.15294v1 Announce Type: new Abstract: Evaluating robot manipulation policies through repeated physical trials is costly. We study whether an action-conditioned world model can serve as a clo

local-aiarxiv-cs-ro

Which Trajectory Is Safer? Combining Rule and Learning for Safety Evaluation

arXiv:2610.17806v1 Announce Type: new Abstract: Collision rate remains a dominant metric for evaluating autonomous driving trajectory safety, but its binary nature obscures substantial safety differen

safetyarxiv-cs-ro

FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems

arXiv:2610.13806v1 Announce Type: cross Abstract: Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not

12 Oct 2026

Closed-loop evaluation of LLM agents for embedded software development

arXiv:2610.11447v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair s

9 Oct 2026

GaussianBench: Physics-Fidelity Evaluation for Gaussian Scene Representations

arXiv:2610.10554v1 Announce Type: cross Abstract: 3D Gaussian Splatting has evolved from static reconstruction toward physics-integrated representations meant to predict how scenes change under intera

9 Oct 2026

Efficient Best-of-N policy evaluation for inference-time alignment

arXiv:2610.09250v1 Announce Type: new Abstract: Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating

safetyarxiv-cs-lg

8 Oct 2026

Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics

arXiv:2610.08831v1 Announce Type: cross Abstract: As the use of smart devices continues to increase, their potential to capture sensitive speech content raises growing privacy concerns. It is therefor

safetyarxiv-cs-cl

8 Oct 2026

A Trust Layer for Agent Evaluation

arXiv:2610.07274v1 Announce Type: new Abstract: Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second

Data, Numbers, and Geometry: Three Tutorials on Numerical Methods, Machine Learning, and Evaluation

arXiv:2610.07220v1 Announce Type: new Abstract: We present three practical tutorials on numerical computation and machine learning for mathematical research, developed for the DANGER: Data, Numbers, a

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi

The Missing Minimal Pair: Stereotype Evaluation in LLMs

arXiv:2610.08747v1 Announce Type: new Abstract: A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that su

safetyarxiv-cs-cl

Construction and Evaluation of Machine Learning Models for Near-Real-Time Fire Detection from MTG FCI Imagery

arXiv:2610.05154v1 Announce Type: new Abstract: Geostationary satellite observations are important for wildfire detection and monitoring. The current study evaluates machine learning models for MTG FC

researcharxiv-cs-cv

6 Oct 2026

Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation

model-releasesarxiv-cs-ro

arXiv:2610.06171v1 Announce Type: new Abstract: Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. Ho

6 Oct 2026

Evaluation format, not model capability, drives measured triage failure in the assessment of consumer health AI

arXiv:2603.11413v4 Announce Type: replace-cross Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage po

5 Oct 2026

Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

arXiv:2610.01645v1 Announce Type: new Abstract: Fairness conclusions in clinical prediction can depend strongly on both the metrics reported and the demographic resolution at which performance is eval

safetyarxiv-cs-lg

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

arXiv:2610.01670v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model t

MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI

arXiv:2610.02136v1 Announce Type: cross Abstract: Unsupervised anomaly detection (UAD) methods for brain MRI are ranked by a single score, yet that score rests on choices that are rarely reported: how

researcharxiv-cs-ai

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

arXiv:2610.01497v1 Announce Type: new Abstract: Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology. As AI enters this workflow,

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

arXiv:2609.33153v2 Announce Type: replace-cross Abstract: Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative re

researcharxiv-cs-ai

When Does Exercise-Specific Joint Selection Help? An Audit of Evaluation and Control Design

arXiv:2610.01188v1 Announce Type: new Abstract: Exercise-specific joint selection can improve skeleton-based correctness classification, but what does that gain establish? We audit 1,057 repetitions f

researcharxiv-cs-ai

EnergyEminence: Source-Aware Environmental Calibration and Evaluation in a Physics-Grounded Grid Digital Twin

arXiv:2609.36215v1 Announce Type: new Abstract: Power-grid digital twins must combine data-driven prediction with physically meaningful state evolution while preserving the provenance of environmental

30 Sept 2026

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

arXiv:2609.35815v1 Announce Type: cross Abstract: Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated con

30 Sept 2026

CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings

arXiv:2609.35645v1 Announce Type: cross Abstract: Code-switching (CS), a seamless alternation between languages within a single utterance, remains a critical challenge in automatic speech recognition

Decision Readouts for Text-Mediated Video Anomaly Detection: An Exploratory Evaluation of Jev and Qwen

arXiv:2609.34180v1 Announce Type: cross Abstract: How much does the decision readout matter when video-derived textual evidence is held fixed? We evaluate Jev typed decisions and three Qwen readouts o

LLM Unlearning Evaluation with TRIAGE

arXiv:2609.32103v1 Announce Type: new Abstract: Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving

Role Support in Knowledge-Graph Error Ranking: Predictor Regimes and Evaluation Policies

arXiv:2512.22318v3 Announce Type: replace Abstract: When do role-specific training counts improve knowledge-graph error ranking? We evaluate fifteen from-scratch predictors, nine development-selected

safetyarxiv-cs-lg

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

arXiv:2609.34228v1 Announce Type: new Abstract: Evaluating statistical discovery by large language model (LLM) agents requires verifiable analytical ground truth. Establishing such ground truth for re

Accounting for Bias Enables Sustainable LLM Evaluation

arXiv:2609.31184v1 Announce Type: new Abstract: LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

arXiv:2609.30979v1 Announce Type: new Abstract: Vision-language models (VLMs) have demonstrated excellent performance in visual tasks, but their visual causal reasoning capabilities still lack reliabl

Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation

arXiv:2609.31186v1 Announce Type: new Abstract: Artificial intelligence is advancing rapidly, with increasingly capable systems taking larger roles in reasoning, decision-making, scientific discovery,

Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

arXiv:2609.30918v1 Announce Type: cross Abstract: Robust counterfactual explanations promise recourse that still works after the model behind it changes. Whether they keep that promise depends on what

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

arXiv:2609.29669v1 Announce Type: cross Abstract: Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions a

Evaluation of Multi-Turn Consistency in LLM Agents: Survival Analysis and Failure-Rationale Taxonomy

arXiv:2609.29508v1 Announce Type: new Abstract: Large language model (LLM) agents may perform well on isolated tasks yet drift into inconsistency over extended interaction. We evaluate temporal consis

Tutorials

Evaluation of OpenAI o1: Opportunities and Challenges of AGI

arXiv:2409.18486v5 Announce Type: replace Abstract: This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, sp

tutorialsarxiv-cs-cl

Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluation

arXiv:2609.26176v2 Announce Type: replace-cross Abstract: Encoded-prompt attacks are evaluated almost entirely on their harmful arm: a benchmark sends obfuscated harmful requests and reports how often

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

arXiv:2609.28725v1 Announce Type: cross Abstract: Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models lear

Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

arXiv:2508.04899v3 Announce Type: replace Abstract: Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inc

24 Sept 2026

Observer Choice and Threshold Selection in Retinal Vessel Segmentation: A Subject-Separated Evaluation

arXiv:2609.25597v1 Announce Type: new Abstract: The annotation used to select a segmentation threshold is part of the evaluation protocol, yet its effect is easily conflated with model quality. We exa

researcharxiv-cs-cv

23 Sept 2026

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

arXiv:2609.25001v1 Announce Type: new Abstract: Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, an

22 Sept 2026

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlled