Search: “evaluation”

Banger paper from MIT and Sakana AI. They show that self-improving coding agents work. The best part is that their approach, Self-Improvemen…

Banger paper from MIT and Sakana AI. They show that self-improving coding agents work. The best part is that their approach, Self-Improvement via Fast Tree-search (SIFT), runs at a tenth of the CPU ho

model-releasesdair-ai--x

19 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Local Ai

A Simulation Platform for AUV Fault Recovery: Exploring LLM-Based Diagnostic Strategies

DGX agent

arXiv:2609.20620v1 Announce Type: cross Abstract: Autonomous underwater vehicles (AUVs) operating beyond reliable communications must recover from failures without human intervention. We investigate a

local-aiarxiv-cs-ai

18 Sept 2026

An Empirical Study of Harness Design for Coding Agents

arXiv:2609.20804v1 Announce Type: new Abstract: Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work t

AnyviewMeter: Adapting Robotic Reward Models with Camera Geometry and Multi-View Attention

arXiv:2609.20106v1 Announce Type: new Abstract: Robotic reward models evaluate task execution from visual observations, but their predictions can change with camera viewpoint and occlusion even when t

CellRFT: Reinforcement Fine-Tuning for Single-Cell Perturbation Modeling

arXiv:2609.19970v1 Announce Type: new Abstract: Predicting cellular responses to perturbations supports the study of gene function, disease mechanisms, and therapeutic strategies. Despite advances in

safetyarxiv-cs-lg

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

arXiv:2506.17277v2 Announce Type: replace-cross Abstract: The retrieval stage of retrieval-augmented generation (RAG) for scientific question answering depends on how documents are segmented and how c

Agents

EconSkills: Studying Skill Transfer and Retrieval for Web Agents on Live Economic Data

arXiv:2609.19523v1 Announce Type: new Abstract: Web agents often revisit the same sites, yet most evaluations discard the procedures learned in earlier successful interactions. We introduce EconSkills

agentsarxiv-cs-ai

MaSCoD: A Multi-Agent Framework for Structural-Context-Guided Candidate Causal Graph Generation

arXiv:2609.19944v1 Announce Type: new Abstract: Large language models (LLMs) have been applied to causal discovery, but candidate-graph generation rarely treats premature omission of potentially relev

NS3Learn: Transferring 5G NR Mode-2 Reception Realism from ns-3 to the Veins/SUMO Stack for Connected-Vehicle Safety Assessment

arXiv:2609.20578v1 Announce Type: cross Abstract: Connected-vehicle safety evaluations rely on coupled traffic and network simulations, but standard channel models ignore radio resource competition in

safetyarxiv-cs-lg

Self Improvement via Fast Tree-search

arXiv:2609.19526v1 Announce Type: new Abstract: Coding agents can recursively modify their own implementations, forming a loop of self-improvement. While prior work shows this can boost performance on

SlugTrails: An Egocentric Benchmark for Floor Plan Localization in Large Buildings

arXiv:2609.19876v1 Announce Type: new Abstract: Floor-plan-based indoor visual localization enables infrastructure-free positioning, but most methods are developed and evaluated in small residential e

Agents

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks

arXiv:2609.19182v1 Announce Type: new Abstract: Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how ev

agentsarxiv-cs-ai

When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models

arXiv:2603.26556v3 Announce Type: replace-cross Abstract: Converting a pretrained Transformer into a more efficient hybrid model through distillation offers a promising approach to reducing inference

AMIGO: Agentic Multi-Image Grounding Oracle Benchmark

arXiv:2603.28662v2 Announce Type: replace-cross Abstract: Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn c

Local Ai

ASLEval: Measuring Privacy Exposure Displacement in LLM Agent Sessions

arXiv:2609.18864v1 Announce Type: cross Abstract: Privacy evaluations of tool-using LLM agents often inspect a designated action, final response, or attacker report. These local proxies can miss unaut

local-aiarxiv-cs-ai

Benchmarking Visual-Inertial Odometry in Subterranean Environments Under Sensor Degradation, Miscalibration, and Dynamic Occlusion

model-releasesarxiv-cs-ro

arXiv:2609.18628v1 Announce Type: new Abstract: Visual-inertial odometry (VIO) is a core capability for autonomous operation in GPS-denied subterranean environments, yet its reliability can degrade sh

Cultural Competence in Context: A Large Language Model Passes the Turing Test in Finland

arXiv:2609.18394v1 Announce Type: new Abstract: We report the results of a Turing Test conducted in Finland in the Finnish language. Because languages and cultural contexts are unevenly represented in

researcharxiv-cs-ai

CzechTopic: A Benchmark for Zero-Shot Topic Localization in Historical Czech Documents

arXiv:2603.03884v2 Announce Type: replace-cross Abstract: Topic localization aims to identify spans of text that express a given topic defined by a name and description. To study this task, we introdu

Faking Good and Faking Bad in LLMs: Response Distortion Across Dark Triad Personality Traits

arXiv:2609.17534v1 Announce Type: new Abstract: Social desirability and impression management are pervasive sources of response distortion in human personality assessment, yet their effects on Large L

safetyarxiv-cs-cl

Agents

Market Signal Injection: Adversarial Context Manipulation of LLM Pricing Agents

arXiv:2609.18357v1 Announce Type: new Abstract: Large language model (LLM) pricing agents may respond to how market data is presented, even when its numerical values remain unchanged. We introduce mar

agentsarxiv-cs-ai

Prosthesis-Aware 3D Human Pose Estimation: A Dataset and Benchmark for RSP Users

arXiv:2609.18406v1 Announce Type: new Abstract: Recovering 3D human body motion from video is important for applications such as rehabilitation assessment and sports performance evaluation. For prosth

Reading Between the Lines: Can LLMs Discover the Question Behind the Text?

model-releasesarxiv-cs-cl

arXiv:2609.19070v1 Announce Type: new Abstract: This paper introduces ``question archaeology'', a specific evaluation task focused on inferring the single, authentic 'genesis question' that motivated

Hardware

Risk-Aware World Modeling with Flow-Guided Occupancy Evolution for Selective Trajectory Planning in Automated Driving

arXiv:2609.18442v1 Announce Type: new Abstract: Safe motion planning in automated driving requires anticipating evolving traffic risks and deciding when to revise the current planned trajectory. We in

hardwarearxiv-cs-ai

Selective Prediction and Uncertainty-Aware Referral for Pap Smear Classification

arXiv:2609.17545v1 Announce Type: new Abstract: Deep learning models for cervical cytology are almost always evaluated as if every prediction must be acted upon, yet a screening system deployed alongs

researcharxiv-cs-cv

Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA

arXiv:2609.18562v1 Announce Type: new Abstract: Track 2 of the AI City Challenge 2026 requires both visual question answering (VQA) and traffic event description generation under a challenging synthet

Too Good to Be Real? Diagnosing and Reducing the Gap Between AI Preference and Real User Engagement

arXiv:2609.18282v1 Announce Type: new Abstract: Large language models are increasingly used to generate and evaluate online content, yet it remains unclear whether the qualities they associate with hi

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

arXiv:2609.18585v1 Announce Type: cross Abstract: Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essenti

safetyarxiv-cs-lg

When Is Graph Structure Worth Its Cost? The Case for Structure Pricing in Retrieval-Augmented Generation

arXiv:2609.18099v1 Announce Type: new Abstract: Graph-based retrieval-augmented generation (RAG) can help answer questions that require information from many documents. However, building a graph often

researcharxiv-cs-ai

A Sentinel-2 benchmark dataset for deep-learning active-fire segmentation across 25 California wildfires

arXiv:2609.16199v1 Announce Type: cross Abstract: This article describes an open image dataset for developing and evaluating active-fire segmentation methods in satellite imagery. The dataset contains

BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents

arXiv:2609.16305v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly operate over long-horizon interactions involving tool use, persistent state, evolving authorization, and

Challenges of Auditing: Variability in Outputs of Large Language Models for Health

arXiv:2609.16590v1 Announce Type: new Abstract: People increasingly use frontier AI models for health advice, but via different access modes (e.g., ChatGPT, ChatGPT Health, APIs) with varying settings

Comment on arXiv:2607.01233: Survivorship Bias in Published-Paper Baselines for Research-Idea Distributions

arXiv:2609.15996v1 Announce Type: new Abstract: Chen, Zhao, and Cohan introduce a valuable distributional evaluation of LLM-generated research ideas. This comment raises a narrower identification conc

safetyarxiv-cs-cl

DiaWhisper-DPO: Role-Attributed Transcription of Clinical Interviews via Failure-Mined Preference Optimization

arXiv:2609.16661v1 Announce Type: new Abstract: Automated depression screening from clinical interviews requires attribution of utterances to the clinician or patient. We evaluate two datasets: DAIC-W

ECHO: A Matched-Contrast Benchmark for Context-Sensitive Turn-Taking in Full-Duplex Dialogue

model-releasesarxiv-cs-cl

arXiv:2609.17360v1 Announce Type: new Abstract: Full-duplex spoken dialogue systems must distinguish interruptions that require yielding the floor from backchannels that permit continued speaking. Exi

Efficient 3D Whole-Body PET Image Denoising via Conditional Rectified Flow With Optimized Sampling Strategy

arXiv:2609.16690v1 Announce Type: new Abstract: Reducing radiation exposure in Positron Emission Tomography (PET) is important for patient safety; however, ultra-low-dose imaging suffers from severe n

safetyarxiv-cs-cv

FairLint-DL: An IDE-Native Tool for Fairness Debugging of Deep Learning Software

arXiv:2609.16321v1 Announce Type: cross Abstract: Existing fairness analysis tools predominantly operate as post-training evaluation frameworks, requiring practitioners to complete the full model deve

safetyarxiv-cs-ai

Quantifying Organizational Environmental Action from Web Data and Large Language Models

arXiv:2609.16627v1 Announce Type: new Abstract: Quantifying organizational environmental action from publicly available web content remains a challenging environmental data science problem because rel

RoleBreak: Benchmarking Long-Horizon Role-Playing Robustness in Spoken Dialogue

arXiv:2609.16614v1 Announce Type: cross Abstract: Speech-to-speech dialogue models increasingly support persona control, yet existing spoken role-playing benchmarks remain largely character-centric an

Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act

arXiv:2609.16268v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These to

safetyarxiv-cs-cl

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

model-releasesarxiv-cs-cl

arXiv:2609.16532v1 Announce Type: new Abstract: Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source co

the design of http://evaluatorbench.com grew out of a summer project, http://epistemedia.org (which i still need to write about), and runs o…

model-releasesyohei-nakajima--x

the design of http://evaluatorbench.com grew out of a summer project, http://epistemedia.org (which i still need to write about), and runs on @activegraphai the pattern across all three projects is th

Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?

arXiv:2609.17248v1 Announce Type: new Abstract: Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked

What Breaks Local Watermarks? A Robustness Benchmark for Local Invisible Image Watermarking

arXiv:2609.16832v1 Announce Type: cross Abstract: Local image watermarking embeds an invisible signal into selected image regions rather than spreading it across the entire image, enabling payload rec

What Does Layer-Importance Reveal About Transformers and State-Space Models?

arXiv:2609.16537v1 Announce Type: cross Abstract: Transformers and state-space models (SSMs) are the two dominant families of sequence models, and a central open question is how far the analytical kno

safetyarxiv-cs-ai

3D Gait-Based Autism Classification Using Attention-Enhanced Deep Learning with Cross-Fold Statistical Stability Analysis

arXiv:2609.14159v1 Announce Type: new Abstract: Autism Spectrum Disorder (ASD) is a neurodevelopmental condition whose early diagnosis remains challenging because conventional clinical assessments are

A latent dimension of Condorcet's jury theorem for multiple AI advisers

arXiv:2609.14438v1 Announce Type: cross Abstract: When the same question is asked of multiple AI advisers, as in self-consistency and LLM-as-a-judge panels, Condorcet's jury theorem predicts that addi

researcharxiv-cs-ai

Accelerating Transfer-Learning-Based Autotuning with Predictive LLVM IR Performance Ranking

arXiv:2609.15807v1 Announce Type: cross Abstract: As the complexity of High Performance Computing (HPC) ecosys- tems continually increases, achieving optimal performance becomes a challenge. Tradition

researcharxiv-cs-lg

Advancing Automated Speaking Assessment Leveraging Multifaceted Relevance and Grammar Information

arXiv:2506.16285v2 Announce Type: replace Abstract: Current automated speaking assessment (ASA) systems for use in multi-aspect evaluations often fail to make full use of content relevance, overlookin