Search: “evaluation”

arXiv:2610.05578v1 Announce Type: cross Abstract: Multi-camera streaming perception is increasingly deployed on heterogeneous edge platforms shared with co-resident workloads, yet accelerator placemen

hardwarearxiv-cs-cv

6 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Counterfactual Action Evaluation, Observation Bottlenecks, and Representation Geometry in Joint-Embedding Predictive World Models

DGX agent

arXiv:2610.02860v1 Announce Type: new Abstract: Low latent prediction error does not establish that a world model distinguishes the consequences of its actions. We introduce an evaluation protocol tha

researcharxiv-cs-lg

5 Oct 2026

Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

arXiv:2610.02267v1 Announce Type: new Abstract: Agent harnesses make many small, typed decisions per task: which model to call, which tool to use, whether retrieved text is relevant, whether an input

5 Oct 2026

SceneFactory-3D: Lifting 2D Traffic Scenes into 3D Physical Counterfactuals for Scalable Physically Grounded Safety Evaluation

arXiv:2610.02874v1 Announce Type: new Abstract: Scalable driving simulators typically execute vehicle commands using prescribed behavioral or kinematic rules, overlooking the physics of tire-road inte

safetyarxiv-cs-ro

5 Oct 2026

When Predicting Nothing Beats SAM 3: Revisiting Evaluation in Video Object Segmentation

arXiv:2610.02946v1 Announce Type: new Abstract: Video Object Segmentation (VOS) in complex and long videos is increasingly important for real-world applications, where target objects often appear only

5 Oct 2026

Beyond Leaderboard Scores: A Deployment-Focused Protocol for Interpretable Tracking Evaluation in Pedestrian-Centric Environments

arXiv:2610.01682v1 Announce Type: cross Abstract: Mobile robots operating among pedestrians need trajectories that become available quickly, remain spatially credible through missed observations, pres

2 Oct 2026

Code Owns the Simulation, Jev Owns the Evaluation

arXiv:2610.01834v1 Announce Type: new Abstract: Judgment models such as jev{} return, in a single call and without reasoning text, a probability for each described option. This makes them attractive a

2 Oct 2026

Hardware

ibUMAP: Coherent and Scalable Field Evaluation for UMAP Optimization

arXiv:2610.01445v1 Announce Type: new Abstract: UMAP achieves scalable layout optimization through stochastic negative sampling. However, this stochasticity can lead to unstable embeddings across reru

hardwarearxiv-cs-lg

2 Oct 2026

Automated Evaluation of Multi-Turn Dialogues in In-Car Conversational Assistants

arXiv:2609.35812v1 Announce Type: new Abstract: In-car conversational assistants (ICAs) are increasingly integrated into vehicles to support route planning, vehicle control, and information access. En

safetyarxiv-cs-cl

Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers

arXiv:2609.37097v1 Announce Type: new Abstract: The rapid growth of submissions and reviewing workload has accelerated the use of large language models (LLMs) in peer review. Prior studies suggest tha

Complexity-Aware Evaluation of LLM Comprehension

arXiv:2609.37405v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks that require understanding existing source code, including behavior

REALHOP: Rethinking Multi-Hop Reasoning Evaluation via Behavioral Auditing

arXiv:2609.36984v1 Announce Type: new Abstract: Complex questions often require multi-hop reasoning that connects facts distributed across sources or distant regions of a long context through intermed

SalamahBench: Dialect and Category Level Safety Evaluation of Arabic Language Models

arXiv:2603.04410v3 Announce Type: replace-cross Abstract: While different stakeholders are trying to leverage Arabic Language Models (ALMs), safety alignment in ALMs remains largely underexplored, hin

A Free Knob: Decoupling Calibration and Predictive Skill in Threshold-Based Evaluation

arXiv:2609.33457v1 Announce Type: new Abstract: Many dense-prediction benchmarks evaluate rare events by pooling prediction and target over spatial blocks, thresholding each, and scoring the contingen

Local Ai

JET: Justification Evaluation in Transformer

arXiv:2609.33874v1 Announce Type: new Abstract: JET uses pretrained language and vision-language models to select among a finite set of answers without additional training. It evaluates candidate like

local-aiarxiv-cs-lg

Measurement Boundaries in LLM Financial Agent Evaluation: Fixed-Tape Execution and Multi-Defect Auditing

arXiv:2609.32379v1 Announce Type: new Abstract: What controls are needed to interpret execution performance and audit scores in LLM agent evaluations? We study two limits on these interpretations in a

agentsarxiv-cs-lg

Offline Policy Evaluation via Mixed Bellman Residuals and Adaptive Critic Representations

arXiv:2609.33186v1 Announce Type: cross Abstract: Evaluating a target policy using data generated by a different behavior policy remains a fundamental challenge in reinforcement learning. While most e

SCLATE: a Substrate for Continual-Learning Agent Training and Evaluation

arXiv:2609.32391v1 Announce Type: cross Abstract: Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requir

Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection

arXiv:2609.32691v1 Announce Type: cross Abstract: LLM agents that invoke privileged tools are vulnerable to indirect prompt injection (IPI), in which adversarial instructions embedded in retrieved dat

Do LLMs Understand Context? A Knowledge Graph-Based Evaluation Framework

arXiv:2609.30484v1 Announce Type: new Abstract: While large language models (LLMs) have achieved remarkable linguistic capabilities, a profound question lingers at their core: do these models truly co

Evaluation Is All You Need for Multi-Modal Autonomous Driving

arXiv:2609.30818v1 Announce Type: cross Abstract: Multi-modal planning is promising for autonomous driving by representing multiple plausible behaviors in ambiguous and long-tail scenarios. Existing m

safetyarxiv-cs-ai

SlideLab: Audience-Centered Scientific Slide Generation and Evaluation

arXiv:2609.30294v1 Announce Type: cross Abstract: Scientific presentations are more than summaries of research papers. They need to present the work in a coherent sequence, explain the main ideas clea

Applications

Where Does Retrieval-Based Open-Ended Evaluation Fail? Automatic Taxonomy Induction from Long-Form Medical Answer Factuality Verification

arXiv:2609.30467v1 Announce Type: new Abstract: Retrieval-based factuality evaluation, where LLM-generated claims are verified against evidence from authoritative medical corpora, has become the domin

applicationsarxiv-cs-cl

Words Speak Louder Than Order: A Behavioral Evaluation of Gemma 4

arXiv:2609.30716v1 Announce Type: cross Abstract: When a language model receives two conflicting documents as input, how does it decide which one to prioritize? Does it rely on how the sources are fra

Generating Interesting Scientific Ideas using Knowledge Graphs and LLMs: Evaluations with 100 Research Group Leaders

arXiv:2405.17044v4 Announce Type: replace Abstract: The rapid growth of scientific literature makes it increasingly challenging for researchers to identify novel and impactful ideas, especially across

25 Sept 2026

Med-AR: Autoregressive Vision-Language Pretraining for Long-Tailed Chest X-Ray Classification and Uncertainty-Aware Evaluation

arXiv:2609.29156v1 Announce Type: cross Abstract: Long-tailed chest X-ray classification requires visual representations that capture both common abnormalities and subtle, infrequent findings. We prop

25 Sept 2026

Connecting the Dots in Agentic AI Security: A Cross-Dimensional Threat Taxonomy, Evaluation Maturity, and Open Challenges

arXiv:2609.23894v1 Announce Type: cross Abstract: Agentic AI extends LLM security beyond generated content to persistent state, autonomous actions, tool use, and interactions with humans and other age

Contact-Stable Deformable Tissue Simulation Using Implicit Integration and Live-Pose Grasp Constraints for Laparoscopic Surgery Robot Policy Evaluation

arXiv:2609.25642v1 Announce Type: new Abstract: Closed-loop evaluation of surgical robots requires tissue that deforms, can be grasped and lifted, and reproduces the anatomy in which the robot will op

safetyarxiv-cs-ro

From Offline Proxies to Online Decisions: A Layered Engagement Evaluation Framework for Conversational AI

arXiv:2609.25408v1 Announce Type: cross Abstract: Online A/B experiments are the decision standard for user engagement, but traffic and readout time limit how many conversational-AI changes can be tes

researcharxiv-cs-lg

On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality

arXiv:2609.26388v1 Announce Type: cross Abstract: Recent advances in large language models (LLMs) have made them widely used for code-related tasks. Identifier names are statistically informative in n

researcharxiv-cs-cl

Protocol before progress: leakage-aware evaluation of AIS trajectory prediction

arXiv:2609.25827v1 Announce Type: new Abstract: Reported gains in vessel-trajectory prediction from Automatic Identification System (AIS) data are credited to new architectures, but the evaluation pro

An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond

arXiv:2609.24170v1 Announce Type: new Abstract: Embodied AI systems are often organized into System 1 and System 2. System 1 is typically a pretrained policy that generates actions at high frequency,

Authority-Preserving Evaluation of Medical Vision-Language Assistants

arXiv:2609.22302v1 Announce Type: new Abstract: Medical vision-language models can propose how urgently a skin lesion should be reviewed, but the local service retains authority to accept or replace t

Applications

Deep Persona: A Psychologically Grounded Architecture and Evaluation Framework for Role-Playing Agents and Simulations

arXiv:2609.22255v1 Announce Type: new Abstract: Existing approaches to persona simulation with Large Language Models (LLMs) mostly rely on shallow character descriptions that fail to sustain coherent

applicationsarxiv-cs-cl

Local Ai

Reconstructed holograms and explanation-aware evaluation for low-cost computational pollen analysis in veterinary cytology

arXiv:2609.23017v1 Announce Type: new Abstract: Automated pollen analysis supports veterinary cytology, but brightfield microscopy is costlier and more complex than lens-less digital in-line holograph

local-aiarxiv-cs-cv

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

model-releasesarxiv-cs-cl

arXiv:2609.20850v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easil

21 Sept 2026

Reliability-Centered Evaluation of Sparse Longitudinal CT Lesion-Size Forecasting with Conformal Interval Calibration and Gompertz-Inspired Regularization

arXiv:2609.21197v1 Announce Type: new Abstract: Sparse longitudinal CT follow-up limits lesion-size forecasting when only a few prior observations are available. We constructed a five-visit DLT-derive

21 Sept 2026

The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators

arXiv:2609.21133v1 Announce Type: cross Abstract: SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We o

21 Sept 2026

A Unified Evaluation Framework for Trustworthy Large Language Models, Agentic AI, and Multimodal Systems

arXiv:2609.19524v1 Announce Type: new Abstract: Benchmark scores alone provide an incomplete basis for assessing the trustworthiness of modern artificial intelligence systems. Large language models (L

18 Sept 2026

Refuse, Decompose, Refresh: A Claim-Safe Protocol for Closed-Loop AI Evaluation

arXiv:2609.20538v1 Announce Type: new Abstract: An AI evaluation can be perfectly reproducible and still support the wrong claim. This risk is acute in closed-loop systems: policy determines visited s

safetyarxiv-cs-ai

18 Sept 2026

Replan, Repair, or Edit? A Unified Empirical Evaluation of Travel Agents for Itinerary Revision under Resource Disruptions

arXiv:2609.19654v1 Announce Type: new Abstract: Travel-planning agents generate itineraries that may become infeasible after acceptance because of flight cancellations, hotel unavailability, or attrac

18 Sept 2026

A Systematic Evaluation of the COTQ Provincial Land Cover Product: Structural Consistency, Spectral Separability, and Relative Positioning Against ESA, ESRI, and Google Products

arXiv:2609.17731v1 Announce Type: new Abstract: High-resolution land use and land cover (LULC) products derived from Sentinel-2 imagery are widely used for environmental monitoring and land management

Face-voice Association across LAnguages and Gender (FLAG) 2027 Challenge Evaluation Plan

arXiv:2609.17913v1 Announce Type: new Abstract: Face--voice association models may rely on language or gender cues in the voice rather than on speaker-specific voice characteristics, which can lead to

Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

model-releasesarxiv-cs-cl

arXiv:2609.19101v1 Announce Type: new Abstract: As models scale, reward hacking becomes more frequent, more sophisticated, and more consequential. Does it leave a telltale signature in model represent

Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale

arXiv:2609.18769v1 Announce Type: new Abstract: Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document i

Execution Flexibility in Automated Planning: A Comparative Evaluation of Deordering and Reordering Strategies

arXiv:2609.16822v1 Announce Type: new Abstract: This study covers foundational concepts for enhancing plan-execution flexibility, including partial-order planning, the producer-consumer-threat formali

16 Sept 2026

Bridging the Gap in ECG-Based Emotion Recognition: A Unified Evaluation of Deep Learning Models

arXiv:2609.15055v1 Announce Type: cross Abstract: Deep learning has led to numerous proposed architectures for Automated Emotion Recognition (AER) from electrocardiogram (ECG) data, but inconsistencie

15 Sept 2026

The MODA General Attribute Suite: A Four-Track Evaluation Benchmark for Fashion Attribute Extraction

arXiv:2609.13279v1 Announce Type: new Abstract: Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different proble