Search: “evaluation”

A Policy Profile for Croissant: Refusal as a Property of the Dataset

arXiv:2609.19640v1 Announce Type: new Abstract: Croissant is the de facto machine-readable descriptor for ML datasets: JSON-LD over schema.org. Since version 1.1 it also carries data use conditions, r

safetyarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

F^{2}DR: A Fine-Grained Full-Pipeline Reward Framework for DeepSearch Workflows

DGX agent

arXiv:2609.19827v1 Announce Type: new Abstract: With the widespread industrial deployment of Large Language Models (LLMs), DeepSearch has emerged as the dominant paradigm for resolving complex user qu

model-releasesarxiv-cs-cl

18 Sept 2026

Seeing Abnormal from Normal: Glomerular Abnormality in Representations of Normal Renal Morphology

arXiv:2609.19444v1 Announce Type: new Abstract: Fine-grained evaluation of glomerular pathology must distinguish normal glomeruli from abnormalities such as global and segmental glomerulosclerosis, ob

safetyarxiv-cs-cv

When Does Retrieval Help Time-Series Forecasting?

arXiv:2609.20193v1 Announce Type: new Abstract: Retrieval plug-ins supply a deep forecaster with information its lookback window cannot carry. Published evaluations report consistent gains, and each c

researcharxiv-cs-lg

'Your Robot Was Trained on a Lie': Collision Mesh Poisoning Attacks on Robotic Manipulation

arXiv:2609.18122v1 Announce Type: cross Abstract: Learning-enabled robotic manipulation increasingly relies on robot simulators for policy training and evaluation before real-world deployment. Inside

safetyarxiv-cs-ro

Compositional Policy Violations: When Step-Level Compliance Fails In Agentic AI Workflows

arXiv:2609.18820v1 Announce Type: new Abstract: Agentic workflows now make consequential decisions in regulated settings, and the governance placed around them is almost entirely step-scoped: input-ou

safetyarxiv-cs-ai

Do Frontier Models Seek Safety Evidence Before Acting?

arXiv:2609.17865v1 Announce Type: new Abstract: Frontier models are often evaluated on how they respond to safety information once it is already in context. We study an earlier decision point: whether

FIERCE: From Generalist Robot Policies to Fast Specialists via Progress-Failure Feedback

model-releasesarxiv-cs-ro

arXiv:2609.18651v1 Announce Type: new Abstract: Generalist robot policies offer useful initialization, but refining compact specialists through limited physical interaction requires informative learni

MIRAGE: How Conversation State Shapes Historical Evidence Use in Multimodal Personal Agents

arXiv:2609.19059v1 Announce Type: cross Abstract: Multimodal large language model (MLLM) agents are increasingly used as personal assistants for long-running tasks. Their utility depends on continuity

Modular Deep Learning Mechanisms for Auditable Next-Day Wildfire Spread Prediction

model-releasesarxiv-cs-lg

arXiv:2609.17763v1 Announce Type: new Abstract: Next-day wildfire prediction requires models whose forecasts can be evaluated alongside the assumptions and historical evidence used in their computatio

MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education

arXiv:2609.19088v1 Announce Type: new Abstract: Large vision-language models have achieved remarkable progress in multi-modal understanding, yet their capabilities in educational settings remain insuf

PBEBench: A Multi-Step Programming by Examples Reasoning Benchmark inspired by Historical Linguistics

arXiv:2505.23126v5 Announce Type: replace Abstract: Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangl

REVERSAL-BENCH: A Reversibility Axis and Reset Oracle for Measuring the Reset-Free RL Cliff

arXiv:2609.17745v1 Announce Type: cross Abstract: A central goal of autonomous reinforcement learning is continuous policy training without external resets. However, existing paradigms largely depend

Tutorials

TACTICS: Taxonomy-Aware Intelligent Corpus Sampling for Machine Translation

arXiv:2609.17956v1 Announce Type: new Abstract: Large-scale machine-translation (MT) systems are typically evaluated on random samples from a corpus whose distributional composition is an artifact of

tutorialsarxiv-cs-cl

What Counts as Strategic Reasoning? A Systematic Mapping of Chess Research on Humans, Engines, and Language Models

arXiv:2609.18286v1 Announce Type: new Abstract: Chess has long served as a model domain for studying search, expertise, decision-making, and artificial intelligence. The emergence of large language mo

A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance

arXiv:2609.16592v1 Announce Type: new Abstract: This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with

CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design

arXiv:2609.16251v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engin

Diagnosing the Fact-Grounding Gap in Multi-Hop Question Answering

arXiv:2609.17043v1 Announce Type: cross Abstract: Multi-hop question answering requires combining information from multiple documents to answer complex questions. These systems have grown increasingly

researcharxiv-cs-ai

DoubleAgents: Human-Agent Alignment in a Socially Embedded Workflow

arXiv:2509.12626v4 Announce Type: replace-cross Abstract: Aligning agentic AI with user intent is critical for delegating complex, socially embedded tasks, yet user preferences are often implicit, evo

safetyarxiv-cs-ai

EviScope: Paired Counterfactual Evidence Diagnostics for Faithful and Efficient Grounded Language Models

arXiv:2609.17081v1 Announce Type: new Abstract: Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or p

Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

arXiv:2607.18063v2 Announce Type: replace-cross Abstract: LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. We present a 21-scenario benchmark f

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

arXiv:2609.15478v1 Announce Type: new Abstract: Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmark

CAL-MOS: Bridging Layers with Adapters for Robust MOS Prediction Across Speech Foundation Models

arXiv:2609.14956v1 Announce Type: cross Abstract: Speech Quality Assessment (SQA) is essential for modern speech technologies, and recent non-intrusive SQA predictors increasingly rely on Speech Found

Failure-Guided Co-Evolution of Prompts and Training Data

arXiv:2609.15209v1 Announce Type: cross Abstract: Automatic prompt optimization (APO) improves language-model programs by revising prompts from task feedback, yet it typically holds its training data

researcharxiv-cs-ai

FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learning

arXiv:2609.15625v1 Announce Type: cross Abstract: Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone te

GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

arXiv:2609.13308v1 Announce Type: new Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language mo

Grounded Adjudication of Variations across Extracted TimeLines (GAVEL): Comparing Clinical Timelines Against Their Case Reports

arXiv:2609.13475v1 Announce Type: new Abstract: Existing pipelines for clinical timeline extraction from case reports are evaluated using an expert reference and are limited by imperfect reference ann

Merging the Knowledge of LLMs for Automatic Speech Recognition

arXiv:2609.15743v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-o

researcharxiv-cs-cl

MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

arXiv:2609.14992v1 Announce Type: new Abstract: Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute,

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

arXiv:2609.15695v1 Announce Type: new Abstract: Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains ch

PARSE: Provenance-Aware Retrieval Sanitization for Professional Domain LLM Agents

arXiv:2606.17467v3 Announce Type: replace-cross Abstract: Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and inte

Post-Reasoning: Improving the Performance of Non-Thinking Models at No Cost

arXiv:2605.06165v2 Announce Type: replace Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribute

Sanity Checking Causal Representation Learning on a Simple Real-World System

arXiv:2502.20099v3 Announce Type: replace-cross Abstract: We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL. The

SkillLift: Learning Dense Rubrics from Sparse Oracles for Efficient Skill Evolution

arXiv:2609.15396v1 Announce Type: new Abstract: LLM-based agents increasingly rely on persistent skills, i.e., reusable procedural prompts, to adapt without weight updates. Existing skill self-evoluti

TotalSynth: Robust Whole-Body Synthetic CT from MRI and CBCT

arXiv:2609.13838v1 Announce Type: new Abstract: Purpose: To develop and evaluate TotalSynth, a reusable pretrained framework for whole-body synthetic CT (sCT) generation from MRI and cone-beam CT (CBC

TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

arXiv:2609.14762v1 Announce Type: cross Abstract: Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk

Two-Stage Personalized Gait Phase Estimation in Stroke Survivors During Exoskeleton-Assisted Walking: An Offline Feasibility Study

arXiv:2609.14984v1 Announce Type: new Abstract: This study evaluated personalized gait phase estimation for stroke survivors using functional inertial measurement unit (IMU) alignment and two-stage se

safetyarxiv-cs-ro

What Makes a 3D Scene Editable? A Factorized Benchmark of Fidelity, Locality, Consistency, and Preservation

arXiv:2609.14899v1 Announce Type: new Abstract: Neural 3D scene editing is often evaluated by semantic alignment alone, although a convincing result may alter unrelated content or become inconsistent

When does a scaling result justify a different allocation? A critical review of resource-allocation evidence for AI systems

arXiv:2609.14500v1 Announce Type: new Abstract: AI scaling studies increasingly evaluate systems that combine a pretrained model with retrieval, search, verification, tools, and interaction. Yet a hig

Hardware

When Faster VLA Deployment Changes Closed-Loop Behavior: Task Success-Latency Analysis of SmolVLA Across PyTorch and ONNX Variants

arXiv:2609.14146v1 Announce Type: cross Abstract: Vision-language-action (VLA) deployment can reduce inference latency while changing closed-loop task behavior. We evaluate HuggingFaceVLA/smolvla_libe

hardwarearxiv-cs-lg

Confidence-Gated Transductive Test Generation for Code Reranking

arXiv:2609.12489v1 Announce Type: cross Abstract: Test case synthesis is crucial for evaluating and ranking programs generated by large language models (LLMs). However, constructing high-quality test

researcharxiv-cs-cl

Context-Aware Causal Gaze Forecasting for Human-Vehicle Interaction During In-Cabin Tracking Dropouts

arXiv:2609.12374v1 Announce Type: new Abstract: Dashboard-mounted gaze trackers often lose sight of the driver's eyes during large head rotations, including shoulder checks, mirror glances, and inters

researcharxiv-cs-cv

Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents

arXiv:2609.13117v1 Announce Type: new Abstract: Full-duplex evaluation often emphasizes whether an agent keeps speaking or stops. That binary cannot express a third response humans use routinely: cont

agentsarxiv-cs-cl

GRACE-DS: a Guarded Reward-guided Agent Correction Environment in Data Science

arXiv:2606.16000v3 Announce Type: replace Abstract: We introduce GRACE-DS, a Guarded Reward-guided Agent Correction Environment in Data Science for pre-deployment evaluation of LLM-powered AutoML agen

safetyarxiv-cs-cl

PRISMA-LLM: An Empirical Reporting Framework for AI-Assisted Systematic Reviews

arXiv:2609.11559v1 Announce Type: cross Abstract: Large language models (LLMs) and AI-enabled software increasingly participate in systematic-review decisions, yet the information needed to audit thes

researcharxiv-cs-cl

ProactiveBench: Can Streaming Video Models Really Interact Like Humans?

arXiv:2609.12658v1 Announce Type: new Abstract: Streaming video understanding requires models to process continuous multimodal input while maintaining temporal context. Existing evaluations are predom

researcharxiv-cs-lg

Revisiting Multi-Object Tracking Baselines: Hyperparameter Optimization with Multi-Fidelity Greedy Coordinate Search

arXiv:2609.12261v1 Announce Type: new Abstract: Multi-object tracking (MOT) is dominated by the tracking-by-detection paradigm, whose methods typically rely on a small set of hyperparameters that are

researcharxiv-cs-cv

Robust Policy Optimization via Adversarial Importance Sampling

arXiv:2609.13044v1 Announce Type: new Abstract: Significant progress has been made in safeguarding deep reinforcement learning (DRL) policies against input perturbations. Developing robust DRL involve

safetyarxiv-cs-lg