Search: “evaluation”

AgentBug-Smith: Automatically Reproducing Real-World Harness Bugs in Agentic Systems

arXiv:2609.37864v1 Announce Type: cross Abstract: Agent harness bugs exhibit unique characteristics and remain challenging for state-of-the-art software agents to repair. Progress in this area is furt

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Auditable Long-Term Memory: A Deterministic Retrieval Chain Measured at 479/475 of 500 on LongMemEval-S

DGX agent

arXiv:2609.38021v1 Announce Type: cross Abstract: We evaluate an auditable long-term memory system on LongMemEval-S. Its retrieval chain uses hybrid candidate retrieval, cross-encoder reranking, cover

model-releasesarxiv-cs-ai

30 Sept 2026

Benchmarking Automatic Speech Recognition Tools for Iberian Languages

model-releasesarxiv-cs-cl

arXiv:2609.36920v1 Announce Type: new Abstract: Comprehensive evaluations of automatic speech recognition (ASR) for Iberian languages remain limited, and low-resource languages, biases, and efficiency

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

arXiv:2609.35841v1 Announce Type: cross Abstract: Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate l

Concept Direction Reliability Across Languages with Different Tokenizer Fertility

arXiv:2609.36194v1 Announce Type: new Abstract: Extracted sentiment directions can vary across samples even when downstream sentiment classification remains accurate. To evaluate direction reproducibi

researcharxiv-cs-cl

Local Ai

Distill Locally, Schedule Globally: Flow Maps for Few-Step Text-to-Speech

arXiv:2609.36324v1 Announce Type: cross Abstract: Flow-matching text-to-speech (TTS) models achieve high synthesis quality but require many neural function evaluations (NFEs) to integrate their genera

local-aiarxiv-cs-lg

From Dead Code and Static Requirements to Working Engines: Software Revival with Coding Agents

arXiv:2609.36161v1 Announce Type: cross Abstract: Can coding agents restore software that no longer runs while preserving its underlying methods, and reconstruct industrial software engines from open

InterBias-SV: Compound Conditions in Speaker Verification

arXiv:2609.36500v1 Announce Type: cross Abstract: Speaker verification systems encounter combinations of noise, channel distortion, and changes in speech. Evaluating each condition separately does not

Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL

arXiv:2609.38065v1 Announce Type: cross Abstract: Training agents to follow arbitrary instructions is an important goal of multi-task reinforcement learning (RL). Linear temporal logic (LTL) provides

Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection

arXiv:2609.37586v1 Announce Type: cross Abstract: Continual audio deepfake detection requires learning newly emerging deepfake methods while retaining discrimination of previously encountered speech.

researcharxiv-cs-ai

Learning Nonlinear Finite Element Solution Operators using Multilayer Perceptrons and Energy Minimization

arXiv:2412.04596v3 Announce Type: replace Abstract: We develop and evaluate a method for learning solution operators to nonlinear problems governed by partial differential equations (PDEs). The approa

LLM unbranding: Erasing Commercial Identity while Preserving Generic Utility

model-releasesarxiv-cs-cl

arXiv:2609.37127v1 Announce Type: new Abstract: Establishing unbranding as a critical practice to prevent visual logos from acquiring negative connotations is standard in image generation. Large Langu

Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG

arXiv:2609.36700v1 Announce Type: cross Abstract: When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up t

researcharxiv-cs-ai

Multichannel Audio Quality Assessment: Extending Pretrained Perceptual Models to Spatial Audio

arXiv:2609.37116v1 Announce Type: cross Abstract: Accurate perceptual quality assessment is essential for evaluating and optimizing spatial audio, where perceived quality depends on both signal fideli

researcharxiv-cs-ai

NIDAR: NIR-Guided Intrinsic Decomposition for Scalable Scene-Agnostic LiDAR Intensity Reconstruction

arXiv:2609.36878v1 Announce Type: new Abstract: LiDAR return intensity provides complementary surface-response cues for robotic perception and state estimation, yet many simulation pipelines omit it o

researcharxiv-cs-ro

Retrieval Sensitivity to Identity Signals in Queries

arXiv:2609.36534v1 Announce Type: new Abstract: Dense retrievers decide which documents reach users and the language models that use them, yet they are typically evaluated with neutral queries. We ask

safetyarxiv-cs-cl

RoboChrono: A Real Robot Benchmark for Streaming Task Understanding

model-releasesarxiv-cs-ro

arXiv:2609.36605v1 Announce Type: new Abstract: Understanding ongoing robot manipulation requires models to interpret visual observations in relation to interaction history and task progress. We intro

SQUARE: Structured Quantum Representation Adapters as Compact Quadratic Feature Maps for Frozen Language Models

arXiv:2609.37134v1 Announce Type: cross Abstract: Frozen language models (LMs) are increasingly used as fixed feature extractors for downstream reranking, scoring, and preference modeling, raising a p

Think Before You Score: Thinking Reward Model for Visual Generation

arXiv:2609.37372v1 Announce Type: new Abstract: Visual reward models are essential for evaluating and improving visual generation models, yet existing approaches typically map task conditions and cand

safetyarxiv-cs-cv

Agents

UpliftMem: Learning Set-Level Uplift for Agent Memory Retrieval

arXiv:2609.36805v1 Announce Type: new Abstract: Large language model (LLM) agents reuse external memory to guide new tasks, but effective retrieval requires learning which memory sets improve executio

agentsarxiv-cs-ai

VeriWeave Govern: Evidence-Gated Deterministic Runtime Governance for Enterprise AI Agents

arXiv:2609.37457v1 Announce Type: new Abstract: Enterprise artificial-intelligence agents increasingly call tools, modify infrastructure, and process protected data, creating a need to separate action

WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

arXiv:2609.36635v1 Announce Type: cross Abstract: Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness a

WorldLine: Action-Driven Visual Simulation for Robotic Manipulation

arXiv:2609.38059v1 Announce Type: cross Abstract: Real-world robot learning is constrained by the cost of collecting experience and evaluating candidate behaviors. Video generation models offer a scal

safetyarxiv-cs-cv

You Cannot Recover What Was Never Measured: Quantifying the Information Ceiling of Ultra-Low-Field MRI Super-Resolution

arXiv:2609.36837v1 Announce Type: new Abstract: Generative super-resolution models can turn portable 64 mT MRI into images that look like 3T scans, and the field evaluates them with PSNR, SSIM, and pi

AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making

arXiv:2609.32740v1 Announce Type: new Abstract: Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient state

safetyarxiv-cs-cv

Are We Making Progress in Multimodal Domain Generalization? A Comprehensive Benchmark Study

arXiv:2605.06643v2 Announce Type: replace Abstract: Despite the growing popularity of Multimodal Domain Generalization (MMDG) for enhancing model robustness, it remains unclear whether reported perfor

Concept Score Relearning: A Unified Cross-Architecture Attack on Concept Erasure

arXiv:2609.33445v1 Announce Type: new Abstract: Concept erasure aims to suppress undesirable knowledge in text-to-image generative models. However, existing robustness evaluations typically rely on re

ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts

model-releasesarxiv-cs-ro

arXiv:2609.31792v1 Announce Type: cross Abstract: While Vision-Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Exist

Contamination, Prior, or Evidence? Decomposing and Training Evidence Use in Whole-Slide Vision-Language Models

arXiv:2609.32185v1 Announce Type: new Abstract: Pathology vision-language models (VLMs) are conventionally evaluated by accuracy, but accuracy alone does not measure evidence use: it may conflate data

Convex Optimization Is Free When Accuracy Is Expensive

arXiv:2609.35418v1 Announce Type: cross Abstract: This paper studies convex optimization when the gradient cannot be evaluated exactly, but only approximated by a hierarchy of algorithms whose compute

safetyarxiv-cs-lg

Deep Weighted Bellman Residual Minimization for Q^* Estimation

arXiv:2609.34593v1 Announce Type: new Abstract: Off-policy evaluation is a foundational component of offline reinforcement learning, aiming to assess and optimize policy performance using pre-collecte

safetyarxiv-cs-lg

Disentangle and Drop: Robust Universal Removal of Image Watermarks via Reconstructive Grayscale Residual Decomposition

arXiv:2609.31693v1 Announce Type: new Abstract: Invisible image watermarks are commonly evaluated against benign postprocessing operations such as compression, resizing, blur, and color changes. These

researcharxiv-cs-cv

DS-VLA: A Dendritic-inspired Vision-Language-Action Model for Robust Action Control

arXiv:2609.32253v1 Announce Type: new Abstract: Vision-language-action (VLA) models have achieved strong performance in language-conditioned manipulation, yet success under nominal evaluation does not

researcharxiv-cs-ro

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

model-releasesarxiv-cs-lg

arXiv:2609.31632v1 Announce Type: new Abstract: Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence a

EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning

arXiv:2609.33261v1 Announce Type: new Abstract: Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical q

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

arXiv:2609.32352v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insuf

Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Models

arXiv:2609.32069v1 Announce Type: new Abstract: Vision--language--action (VLA) models provide strong priors for robotic manipulation but are typically deployed as frozen policies, unable to improve fr

safetyarxiv-cs-ro

From One-Shot Generation to Incremental Music Composition: Adapting a General-Purpose Instruction LLM for Persistent Symbolic Editing

model-releasesarxiv-cs-lg

arXiv:2609.34994v1 Announce Type: cross Abstract: Most music-generation systems are still framed and evaluated primarily as producers of complete outputs, whereas composition often proceeds through su

JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

arXiv:2609.35032v1 Announce Type: cross Abstract: In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be d

Learning High-Risk High-Precision Motion Control

arXiv:2609.34851v1 Announce Type: new Abstract: Deep reinforcement learning (DRL) algorithms for movement control are typically evaluated and benchmarked on sequential decision tasks where imprecise a

safetyarxiv-cs-lg

Learning to Act under Visual Interruptions with Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2609.35003v1 Announce Type: new Abstract: Vision-language-action (VLA) models have demonstrated strong capabilities in robotic manipulation, but they are typically developed and evaluated with a

Agents

M3OS: A Monte Carlo Graph Search-Orchestrated Multi-Agent LLM System for Evidence-Traced Molecular Optimization

arXiv:2609.34491v1 Announce Type: new Abstract: Small-molecule optimization integrates medicinal-chemistry reasoning and computational evidence through iterative, multi-objective decisions. When large

agentsarxiv-cs-lg

Tutorials

Observability-Informed Optimal Sensor Placement for Soft Robots

arXiv:2609.33681v1 Announce Type: new Abstract: This paper presents the application and experimental evaluation of a systematic method for optimal sensor placement in soft robots. Existing methods eit

tutorialsarxiv-cs-ro

Patient-level validation of a foundation-model pipeline for pediatric lung sounds: physician-annotated adventitious events are recognized, disease groups are not reliably predicted

arXiv:2603.15688v2 Announce Type: replace-cross Abstract: Background: Lung-sound classifiers are usually evaluated on recording- or event-level splits, although each child contributes many recordings.

PDMD: Projected Distribution Matching Distillation for Video Diffusion Models

arXiv:2609.35768v1 Announce Type: new Abstract: Modern video diffusion models require tens of denoising evaluations over long spatiotemporal token sequences. Distribution Matching Distillation (DMD) r

researcharxiv-cs-cv

Recovery-Directed Symbolic Distillation of Neural Likelihoods

model-releasesarxiv-cs-lg

arXiv:2609.32409v1 Announce Type: new Abstract: Amortized neural likelihoods enable computationally expensive inference for models with analytically intractable or unspecified likelihoods, but their b

Recursive LLM Degradation in Biomedical Question Answering: A Cross-Generation Study

arXiv:2609.34257v1 Announce Type: cross Abstract: Repeatedly training language models on their own generated data may create a synthetic-data feedback loop in which errors and distributional biases ar

Riccati State Space Models: Non-iterative Parallelization for Nonlinear Sequence Modeling

arXiv:2609.35441v1 Announce Type: new Abstract: State space models (SSMs) achieve efficient sequence processing because their affine state updates are closed under composition and can therefore be eva