Search: “evaluation”

Capabilities Ain't All You Need: Measuring Propensities in AI

arXiv:2602.18182v5 Announce Type: replace-cross Abstract: AI evaluation has primarily focused on measuring capabilities, with formal approaches inspired from Item Response Theory (IRT) being increasin

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

CompMat-Bench: Benchmarking AI Agents for Computational Materials Science

DGX agent

arXiv:2610.00636v1 Announce Type: new Abstract: Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In c

model-releasesarxiv-cs-ai

2 Oct 2026

Local Ai

Compositional Embedding Architecture for Physical Field Prediction in Componentized Aerospace Systems

arXiv:2610.00237v1 Announce Type: cross Abstract: Spacecraft thermal design requires repeated evaluation of how variations in the number and spatial arrangement of heat-generating components and in th

local-aiarxiv-cs-lg

Detecting Inconsistencies in Model Specifications with LLM-as-Verifier Reasoning

arXiv:2610.01847v1 Announce Type: cross Abstract: Model specifications define how large language models (LLMs) should behave, guiding alignment training, inference-time behavior, and evaluation. Yet t

DiVid: Diagnosing Dimension-Specific Diversity Collapse in Video Generation Models

arXiv:2610.01661v1 Announce Type: new Abstract: Despite remarkable progress, video generation models often produce highly similar outputs when repeatedly sampled from the same prompt, limiting their u

GIFTBench: Diagnosing Generalization in Image Forgery Localization and Informing Model Design

arXiv:2610.01778v1 Announce Type: new Abstract: Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cove

InterviewSim: A Scalable Framework for Interview-Grounded Personality Simulation

arXiv:2602.20294v2 Announce Type: replace-cross Abstract: Simulating real personalities with large language models requires grounding generation in authentic personal data. Existing evaluation approac

Tutorials

Prediction-powered Neural Architecture Search

arXiv:2610.01317v1 Announce Type: new Abstract: Evaluating candidate architectures in neural architecture search (NAS) faces an inherent trade-off: on the one hand, reliable performance labels are lim

tutorialsarxiv-cs-lg

Local Ai

Towards Subject Consistency over Dynamic Subject Sets in Video Generation

arXiv:2610.01052v1 Announce Type: new Abstract: We argue that as video generation extends to longer durations, subject consistency should be evaluated over extit{dynamic subject sets}. We therefore in

local-aiarxiv-cs-cv

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

arXiv:2609.38409v1 Announce Type: new Abstract: Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable re

E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch

arXiv:2609.38335v1 Announce Type: cross Abstract: Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. Ho

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

arXiv:2609.38473v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of ret

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular int

Do Evidence-Reading Diagnostics Improve Interface Selection in Small LLM Recommenders?

arXiv:2609.37472v1 Announce Type: cross Abstract: Behavioral tests measure how a language model reads evidence. We ask whether those measurements help choose a recommendation interface. We evaluate si

researcharxiv-cs-lg

HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models

arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real huma

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

arXiv:2609.26550v3 Announce Type: replace Abstract: LLM-as-a-judge scales evaluation, but reasoning judges are slow and costly. We study JEV-as-a-Judge: evaluation with JEV, a decision-only judge that

researcharxiv-cs-ai

Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs

arXiv:2609.38070v1 Announce Type: new Abstract: As the chain-of-thought reasoning capabilities of large language models improve, evaluating and calibrating their reasoning confidence is becoming incre

Reliability Testing of Medical Model Performance under Distributed Deployment

arXiv:2609.36525v1 Announce Type: cross Abstract: Distributed inference has become an indispensable part of deploying medical models under practical latency, memory, and throughput constraints. Althou

Tutorials

SYNCR: Diagnosing and Learning Cross-Video Reasoning from Simulation

arXiv:2609.37918v1 Announce Type: new Abstract: Reasoning across videos requires aligning events, matching identities, comparing motion, and integrating partial observations. Evaluating these capabili

tutorialsarxiv-cs-cv

A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

arXiv:2609.32890v1 Announce Type: new Abstract: Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured

ControlTrace: Recovering Control Fields for Hidden-Content Recognition

arXiv:2609.34807v1 Announce Type: new Abstract: Spatially conditioned diffusion models can embed words and contours in natural-looking images, but vision-language models (VLMs) may fail to recognize t

CTP-FL: Common-Trajectory Gradient Prediction for Federated Learning

arXiv:2609.35130v1 Announce Type: new Abstract: Communication-efficient federated optimization commonly spends several gradient evaluations between server updates. Existing local-update methods use th

safetyarxiv-cs-lg

FestDPO: Few-step Generator Alignment with Direct Preference Optimization

arXiv:2609.34673v1 Announce Type: new Abstract: Few-step generative models can generate high-fidelity samples within a few function evaluations. Despite this efficiency, generated samples may not exhi

safetyarxiv-cs-lg

From Language to Task Maps: Compiling Semantic Relations While Preserving Task-Relevant Freedom

arXiv:2609.34412v1 Announce Type: new Abstract: Natural-language manipulation instructions specify qualitative relations, whereas continuous controllers require state-evaluable task quantities, differ

researcharxiv-cs-ro

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

arXiv:2609.35052v1 Announce Type: new Abstract: Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video ref

ROTE: Benchmarking Neural Memorization on Complexity-Controlled Symbolic Sequences

arXiv:2609.31918v1 Announce Type: new Abstract: We introduce ROTE (RollOut Testing of Exact memorization), a benchmarking protocol for evaluating symbolic memorization of neural architectures. We stud

SolveEdit: Benchmarking Visual Problem Solving in Generative Models

arXiv:2609.35504v1 Announce Type: new Abstract: Machine intelligence is often evaluated through abstract reasoning problems, yet many real-world problems are visual, such as arranging objects, repairi

BioEVAL: A global, multi-institutional benchmark of large language and multimodal models for bioengineering

arXiv:2609.30489v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated historic breakthroughs in general reasoning with early successes in biomedical science. However, existing

28 Sept 2026

Rank-Reliable Teacher-Guided Fitness Approximation for Expensive Evolutionary Optimization: A TinyML Architecture Search Study

arXiv:2609.30553v1 Announce Type: new Abstract: Expensive evolutionary search does not always need an exact fitness estimate for every candidate. It often needs a reliable answer to a simpler question

28 Sept 2026

TrafficImag: A Benchmark for Counterfactual Roadside Traffic Video Generation

arXiv:2609.30722v1 Announce Type: cross Abstract: Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video gener

28 Sept 2026

Contact as a Decision Variable: Capability-Tradeoff Contact Selection for Legged Loco-Manipulation

arXiv:2609.30140v1 Announce Type: new Abstract: In this paper, we study the joint selection of an environmental support contact and a whole-body configuration for a prescribed loco-manipulation task.

researcharxiv-cs-ro

Hard Stop: Kernel-Level Preemption and Containment for Rogue Agentic Execution

arXiv:2609.29808v1 Announce Type: cross Abstract: In July 2026, an unconstrained autonomous agent participating in a frontier AI cybersecurity evaluation harness breached its evaluation sandbox, estab

RoboRecover: Benchmarking Robot Policy Recovery under Execution Deviations

model-releasesarxiv-cs-ro

arXiv:2609.28952v1 Announce Type: new Abstract: Robot-policy benchmarks increasingly cover diverse tasks and preset out-of-distribution conditions, but typically evaluate complete trajectories from pr

Local Ai

Same Bit Width, Different Outcomes: Post-Training Quantization of Text-to-Speech Across Architectures

arXiv:2609.28974v1 Announce Type: cross Abstract: Post-training quantization (PTQ) reduces the cost of on-device text-to-speech (TTS), but published evaluations cover one system or method. We evaluate

local-aiarxiv-cs-lg

When Misalignment Becomes Supervision: Structured Label Noise in Supervised Synthetic CT Generation

arXiv:2609.29387v1 Announce Type: new Abstract: Supervised synthetic CT (sCT) generation is commonly trained and evaluated as voxel-wise regression against registered reference CT images. In practice,

safetyarxiv-cs-cv

A hierarchy of faithfulness criteria for knowledge base completion

arXiv:2609.27863v1 Announce Type: new Abstract: Knowledge graph completion is evaluated by ranking observed triples above randomly corrupted ones, which treats every unobserved fact as false. When the

researcharxiv-cs-ai

Beyond Overlap: Estimating the Causal Effect of Benchmark Exposure

model-releasesarxiv-cs-cl

arXiv:2609.27176v1 Announce Type: new Abstract: Evidence that evaluation material entered training does not reveal how much it affected evaluation. This distinction leaves a contaminated benchmark sco

Dissecting Advantage-Guided Post-Training for Vision-Language-Action Policies

arXiv:2609.28161v1 Announce Type: new Abstract: Advantage-guided reinforcement learning provides a practical way to post-train vision-language-action (VLA) policies using limited robot data. However,

safetyarxiv-cs-ro

Limiting-Kernel Q(lambda): Bridging Short and Long Horizons

arXiv:2609.27741v1 Announce Type: new Abstract: In value-based reinforcement learning, improving the accuracy of policy evaluation has been shown to improve downstream policy optimization performance.

The Recall Ceiling of LLM Recommendation Reranking

arXiv:2609.27953v1 Announce Type: cross Abstract: Some LLM-based recommendation rerankers are evaluated under an oracle protocol that guarantees the ground-truth item is present in the scored set, eit

Artificial Neural Networks as Surrogate Models in Black Box Optimization

arXiv:2609.22329v1 Announce Type: cross Abstract: Black-Box Optimization (BBO) is often applied in several engineering fields and can utilize an advancement of numerical measure- ments and simulation

researcharxiv-cs-ai

23 Sept 2026

Counterfactual Tool Ranking under Utility, Cost, and Privilege Constraints

arXiv:2609.22819v1 Announce Type: new Abstract: Counterfactual tool evaluation must distinguish authority, historical support, and what a comparison actually estimates. We study these distinctions wit

IMPLICIT-Bench: Measuring Implicit Bias in Text-to-Image Models under Neutral Prompts

arXiv:2609.24228v1 Announce Type: new Abstract: Text-to-image (T2I) models are typically evaluated for bias using slot-based templates such as ``a photo of a [profession]''. Such templates probe only

RSPDBench: Benchmarking Vision Foundation Models on Earth Observation Tasks Under Physically Grounded Remote-Sensing Product Degradations

arXiv:2609.23427v1 Announce Type: new Abstract: Vision foundation models targeting Earth observation (EO) tasks are commonly evaluated on clean downstream benchmarks, but operational EO products can a

safetyarxiv-cs-cv

When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs

arXiv:2609.24799v1 Announce Type: new Abstract: Post-training quantization (PTQ) enables efficient deployment of large language models, and PTQ methods are usually optimized and evaluated with generic

researcharxiv-cs-cl

Clinician-Grounded Quality Assurance for AI-Assisted Psychiatric Intake

arXiv:2609.21149v1 Announce Type: new Abstract: Before patients can use AI-assisted psychiatric intake systems, health systems need practical ways to routinely evaluate these tools against their clini

21 Sept 2026

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

arXiv:2609.22069v1 Announce Type: new Abstract: Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omn

21 Sept 2026

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

arXiv:2609.21465v1 Announce Type: cross Abstract: We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly