Search: “evaluation”

Which Trajectory Is Safer? Combining Rule and Learning for Safety Evaluation

arXiv:2610.17806v1 Announce Type: new Abstract: Collision rate remains a dominant metric for evaluating autonomous driving trajectory safety, but its binary nature obscures substantial safety differen

safetyarxiv-cs-ro

14 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FORTE: A Production Framework for Feature Evaluation and Deprecation in Industrial Recommender Systems

DGX agent

arXiv:2610.13806v1 Announce Type: cross Abstract: Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not

model-releasesarxiv-cs-ai

12 Oct 2026

Closed-loop evaluation of LLM agents for embedded software development

arXiv:2610.11447v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair s

9 Oct 2026

GaussianBench: Physics-Fidelity Evaluation for Gaussian Scene Representations

arXiv:2610.10554v1 Announce Type: cross Abstract: 3D Gaussian Splatting has evolved from static reconstruction toward physics-integrated representations meant to predict how scenes change under intera

9 Oct 2026

Efficient Best-of-N policy evaluation for inference-time alignment

arXiv:2610.09250v1 Announce Type: new Abstract: Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating

8 Oct 2026

Is Word Error Rate Enough? Rethinking Privacy Evaluation in Speech with Entity-Aware Metrics

arXiv:2610.08831v1 Announce Type: cross Abstract: As the use of smart devices continues to increase, their potential to capture sensitive speech content raises growing privacy concerns. It is therefor

safetyarxiv-cs-cl

8 Oct 2026

A Trust Layer for Agent Evaluation

arXiv:2610.07274v1 Announce Type: new Abstract: Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second

Data, Numbers, and Geometry: Three Tutorials on Numerical Methods, Machine Learning, and Evaluation

arXiv:2610.07220v1 Announce Type: new Abstract: We present three practical tutorials on numerical computation and machine learning for mathematical research, developed for the DANGER: Data, Numbers, a

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

model-releasesarxiv-cs-cl

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi

The Missing Minimal Pair: Stereotype Evaluation in LLMs

arXiv:2610.08747v1 Announce Type: new Abstract: A common approach to measuring bias in Large Language Models is to compare the log-likelihoods of two contrastive stereotype sentences. We argue that su

safetyarxiv-cs-cl

Construction and Evaluation of Machine Learning Models for Near-Real-Time Fire Detection from MTG FCI Imagery

arXiv:2610.05154v1 Announce Type: new Abstract: Geostationary satellite observations are important for wildfire detection and monitoring. The current study evaluates machine learning models for MTG FC

researcharxiv-cs-cv

6 Oct 2026

Controllable and Photorealistic Pedestrian Risky Motion Generation for End-to-End Driving Safety Evaluation

model-releasesarxiv-cs-ro

arXiv:2610.06171v1 Announce Type: new Abstract: Evaluating end-to-end autonomous driving under rare, safety-critical vehicle-pedestrian interactions requires photorealistic, sensor-level scenarios. Ho

6 Oct 2026

Evaluation format, not model capability, drives measured triage failure in the assessment of consumer health AI

arXiv:2603.11413v4 Announce Type: replace-cross Abstract: A recent Nature Medicine study reported that ChatGPT Health under-triages 51.6% of emergencies and concluded that consumer-facing AI triage po

5 Oct 2026

Beyond Demographic Balance: Multi-Metric and Intersectional Evaluation of Fairness in MIMIC-IV Mortality Prediction

arXiv:2610.01645v1 Announce Type: new Abstract: Fairness conclusions in clinical prediction can depend strongly on both the metrics reported and the demographic resolution at which performance is eval

Do MLLM Judges Judge the Edit? Auditing Bias in Image Editing Evaluation with Verified Quality Preservation

arXiv:2610.01670v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as automated judges for instruction-based image editing and as reward signals for model t

MIRTO: a registration-gated, multiverse-tested evaluation protocol for unsupervised anomaly segmentation in brain MRI

arXiv:2610.02136v1 Announce Type: cross Abstract: Unsupervised anomaly detection (UAD) methods for brain MRI are ranked by a single score, yet that score rests on choices that are rarely reported: how

OpenMTB-Audit: Exposing Over-Refusal and Clinical Expert Perspectives in LLM-Based Molecular Tumor Board Safety Evaluation

arXiv:2610.01497v1 Announce Type: new Abstract: Molecular tumor boards integrate genomic findings, clinical context, and therapeutic evidence to support precision oncology. As AI enters this workflow,

What Does a Skill Actually Do? Estimands and Evaluation Validity for Tool and Skill Use in LLM Agents: A Critical Review

arXiv:2609.33153v2 Announce Type: replace-cross Abstract: Reported improvements from tools and reusable skills in large language model agents refer to different comparisons. This critical narrative re

When Does Exercise-Specific Joint Selection Help? An Audit of Evaluation and Control Design

arXiv:2610.01188v1 Announce Type: new Abstract: Exercise-specific joint selection can improve skeleton-based correctness classification, but what does that gain establish? We audit 1,057 repetitions f

EnergyEminence: Source-Aware Environmental Calibration and Evaluation in a Physics-Grounded Grid Digital Twin

arXiv:2609.36215v1 Announce Type: new Abstract: Power-grid digital twins must combine data-driven prediction with physically meaningful state evolution while preserving the provenance of environmental

30 Sept 2026

How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats

arXiv:2609.35815v1 Announce Type: cross Abstract: Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated con

30 Sept 2026

CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings

arXiv:2609.35645v1 Announce Type: cross Abstract: Code-switching (CS), a seamless alternation between languages within a single utterance, remains a critical challenge in automatic speech recognition

Decision Readouts for Text-Mediated Video Anomaly Detection: An Exploratory Evaluation of Jev and Qwen

arXiv:2609.34180v1 Announce Type: cross Abstract: How much does the decision readout matter when video-derived textual evidence is held fixed? We evaluate Jev typed decisions and three Qwen readouts o

LLM Unlearning Evaluation with TRIAGE

arXiv:2609.32103v1 Announce Type: new Abstract: Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving

Role Support in Knowledge-Graph Error Ranking: Predictor Regimes and Evaluation Policies

arXiv:2512.22318v3 Announce Type: replace Abstract: When do role-specific training counts improve knowledge-graph error ranking? We evaluate fifteen from-scratch predictors, nine development-selected

SleuthBench: Benchmarking Statistical LLM Evaluation Using Tabular Hidden Signals

arXiv:2609.34228v1 Announce Type: new Abstract: Evaluating statistical discovery by large language model (LLM) agents requires verifiable analytical ground truth. Establishing such ground truth for re

Accounting for Bias Enables Sustainable LLM Evaluation

arXiv:2609.31184v1 Announce Type: new Abstract: LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

arXiv:2609.30979v1 Announce Type: new Abstract: Vision-language models (VLMs) have demonstrated excellent performance in visual tasks, but their visual causal reasoning capabilities still lack reliabl

Evolutionary Safety of Recursive Self-Improving AI: Taxonomy, Risk Discovery, and Evaluation

arXiv:2609.31186v1 Announce Type: new Abstract: Artificial intelligence is advancing rapidly, with increasingly capable systems taking larger roles in reasoning, decision-making, scientific discovery,

Robust to Which Model Change? A Unified Evaluation of Robust Counterfactual Explanations

arXiv:2609.30918v1 Announce Type: cross Abstract: Robust counterfactual explanations promise recourse that still works after the model behind it changes. Whether they keep that promise depends on what

Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence

arXiv:2609.29669v1 Announce Type: cross Abstract: Robot learning now advances along two tracks that rarely meet. On one side, direct Vision-Language-Action (VLA) policies map observations to actions a

Evaluation of Multi-Turn Consistency in LLM Agents: Survival Analysis and Failure-Rationale Taxonomy

arXiv:2609.29508v1 Announce Type: new Abstract: Large language model (LLM) agents may perform well on isolated tasks yet drift into inconsistency over extended interaction. We evaluate temporal consis

Tutorials

Evaluation of OpenAI o1: Opportunities and Challenges of AGI

arXiv:2409.18486v5 Announce Type: replace Abstract: This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, sp

tutorialsarxiv-cs-cl

Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluation

arXiv:2609.26176v2 Announce Type: replace-cross Abstract: Encoded-prompt attacks are evaluated almost entirely on their harmful arm: a benchmark sends obfuscated harmful requests and reports how often

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

arXiv:2609.28725v1 Announce Type: cross Abstract: Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models lear

Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection

arXiv:2508.04899v3 Announce Type: replace Abstract: Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inc

24 Sept 2026

Observer Choice and Threshold Selection in Retinal Vessel Segmentation: A Subject-Separated Evaluation

arXiv:2609.25597v1 Announce Type: new Abstract: The annotation used to select a segmentation threshold is part of the evaluation protocol, yet its effect is easily conflated with model quality. We exa

researcharxiv-cs-cv

23 Sept 2026

GameHorizon Suite: Multi-Horizon Data and Evaluation in Gameplay

arXiv:2609.25001v1 Announce Type: new Abstract: Modern video games provide a measurable testbed for AI models, combining abilities of visual understanding, instruction decomposition, goal planning, an

22 Sept 2026

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

model-releasesarxiv-cs-cl

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlled

22 Sept 2026

Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluation

arXiv:2509.22754v2 Announce Type: replace-cross Abstract: Autonomous driving remains a highly active research domain that seeks to enable vehicles to perceive dynamic environments, predict the future

21 Sept 2026

Rethinking Human-Aligned Evaluation: An Analysis of Semantic Metrics Beyond WER

model-releasesarxiv-cs-cl

arXiv:2609.21663v1 Announce Type: new Abstract: Word Error Rate (WER), the most commonly used metric for Automatic Speech Recognition (ASR), treats every lexical deviation from the reference as equall

21 Sept 2026

A Table-Free Index for Tapered Memoization Grids: Compact Out-of-Core Evaluation of Functions of Sorted Arguments

arXiv:2609.20276v1 Announce Type: cross Abstract: Many applications must repeatedly evaluate an expensive function f of a sorted score vector whose influence decays with rank: Plackett-Luce choice pro

18 Sept 2026

KoNeoBench: A Curated Evaluation Dataset for LLM Understanding of Korean Neologisms

arXiv:2609.19916v1 Announce Type: cross Abstract: Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging word

18 Sept 2026

Model-based Bootstrap for Offline Policy Evaluation in Tabular Reinforcement Learning

arXiv:2609.20389v1 Announce Type: cross Abstract: Offline policy evaluation (OPE) is crucial in high-stakes reinforcement learning applications, where new policies must be assessed reliably before dep

18 Sept 2026

Beyond Pixel Similarity: Task-Aware Evaluation of GAN-Based Synthetic Sonar Data for Robotic Perception

arXiv:2609.18100v1 Announce Type: cross Abstract: Synthetic data can reduce the cost of collecting and annotating training data for robotic perception, but generating sensor observations that preserve

17 Sept 2026

Large Language Models Versus Physicians in Traditional Chinese Medicine: A Real-World Clinical Case Evaluation

arXiv:2609.17544v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being explored for clinical applications, yet their assessment for real-world traditional Chinese medicine

safetyarxiv-cs-cl

17 Sept 2026

The Uneven Impact of Generative AI on Student Learning: Examining the Roles of Reliance, Evaluation Literacy, and Course Policy in AI-related Courses

arXiv:2609.18676v1 Announce Type: new Abstract: Generative artificial intelligence (GenAI) is changing how students learn, yet the roles of course context, cognitive reliance, evaluation literacy, and

17 Sept 2026

Beyond In-Distribution Metrics: A Systematic Out-of-Distribution Evaluation of Congenital Heart Disease Segmentation

arXiv:2609.17068v1 Announce Type: cross Abstract: Congenital heart disease (CHD) diagnosis and surgical planning often require patient-specific 3D anatomical models, but manual segmentation is labor-i