Search: “evaluation”

arXiv:2610.03817v1 Announce Type: cross Abstract: Purpose: To evaluate the performance-feasibility tradeoffs of foundation models (FMs) and task-specific convolutional neural networks (CNNs) trained f

hardwarearxiv-cs-cv

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Odyssey: A Closed-Loop Benchmark for Long-Horizon Real-World Driving with Explicit Navigation Routes

model-releasesarxiv-cs-ro

arXiv:2610.06469v1 Announce Type: new Abstract: Closed-loop evaluation of end-to-end driving requires continuous rollouts that reveal how earlier decisions affect subsequent driving. However, existing

SOL: Measuring Gaps between Text Distributions by Double Sliced Wasserstein Metrics

arXiv:2610.06513v1 Announce Type: cross Abstract: Evaluating text generation requires measuring how well the generated distribution matches the data distribution. For autoregressive models, this is do

researcharxiv-cs-lg

Training and Scaling Compute-Optimal Physiological Waveform Foundation Models

arXiv:2610.05649v1 Announce Type: new Abstract: We investigate the scaling laws and compute-optimal training of physiological waveform foundation models (FMs). We train Aether, a family of over one hu

researcharxiv-cs-lg

When Is Enough Enough in Self-Evolving LLM Systems?

arXiv:2610.04756v1 Announce Type: cross Abstract: Self-evolving large language model (LLM) systems repeatedly propose, evaluate, and incorporate updates to prompts, skills, or other persistent artifac

Benchmarking Literature Retrieval for a Model Organism: A Dictyostelium Case Study

arXiv:2610.03130v1 Announce Type: cross Abstract: Biological literature retrieval systems are often developed and evaluated using broad biomedical corpora and general-purpose search tasks. However, ma

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

arXiv:2610.03639v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems o

Language Models that Play Chess and Explain Their Moves

arXiv:2610.03695v1 Announce Type: new Abstract: Modern chess engines are silent experts: they play at a superhuman level, but do not offer explanations for their play. On the other hand, language mode

Large language models exhibit unreliable updating of clinical judgment as patient evidence evolves

arXiv:2610.02684v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for clinical reasoning, but whether they appropriately revise judgments as patient evidence evolv

researcharxiv-cs-ai

NegT2IBench: When Negation Changes the Picture. A Polarity Benchmark for Text-to-Image Models

arXiv:2610.03084v1 Announce Type: cross Abstract: Text-to-image (T2I) models are judged by benchmarks that measure whether requested content appears, but these benchmarks largely overlook the compleme

OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation

arXiv:2610.02781v1 Announce Type: cross Abstract: Many useful language-model tasks cannot be evaluated by exact outcome verification. Rubric-based reinforcement learning (RL) addresses this issue by s

safetyarxiv-cs-ai

OpenGameEval: Benchmarking Agentic Programming and Exploration in a Stateful Game Engine

arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in

Tutorials

Reliable Self-Evolution with Imperfect Proxy Rewards

arXiv:2610.02975v1 Announce Type: new Abstract: Large language model (LLM)-based self-evolving search is a promising approach to scientific discovery. However, high-fidelity evaluation of every candid

tutorialsarxiv-cs-ai

SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation

arXiv:2610.02304v1 Announce Type: cross Abstract: Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish

Understanding Clinical Cognitive Dialogues Using Large Language Models

arXiv:2609.34125v2 Announce Type: replace Abstract: In-person cognitive assessment is both a test and an interaction. Clinicians explain tasks, repair misunderstandings, and adapt to patient responses

From Pixels to Policy: A Multi-Agent System for Intervention and Geo-Spatial Decision Support

arXiv:2610.01870v1 Announce Type: new Abstract: Urban environments are shaped by design choices with long-term implications for health, safety, and quality of life, yet evaluating proposed interventio

safetyarxiv-cs-cv

In Vino Veritas and Vulnerabilities: Examining LLM Safety via Drunk Language Inducement

arXiv:2601.22169v2 Announce Type: replace-cross Abstract: Humans are susceptible to undesirable behaviours and privacy leaks under the influence of alcohol. This paper investigates drunk language, i.e

safetyarxiv-cs-ai

Interpreting Reasoning of Large Language Models via Partial Information Decomposition

arXiv:2610.00571v1 Announce Type: cross Abstract: Large reasoning models (LRMs) have achieved substantial improvements in solving complex mathematical problems, but often produce lengthy, repetitive,

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

model-releasesarxiv-cs-ro

arXiv:2610.01351v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphas

KaliBench: A Fine-Grained Benchmark for Cybersecurity Tool Use on Kali Linux with Runtime-Free Verifiable Rewards

arXiv:2610.02206v1 Announce Type: cross Abstract: LLMs are increasingly applied to cybersecurity workflows, where they are expected to translate analysts' intent into tool invocations. However, existi

Learning to Sell: Reinforcement Learning for Strategic Large Language Model Agents in Multi-Product Markets

arXiv:2609.33289v2 Announce Type: replace Abstract: Autonomous large language model (LLM) agents operating in multi-product markets must make sequential decisions under information asymmetry and resou

Hardware

LESS: Lightweight Evolutionary Supernet Search in Minutes

arXiv:2610.01468v1 Announce Type: cross Abstract: Low-cost NAS must both explore high-performing architectures and identify them reliably, yet reducing evaluation cost often weakens the fidelity of ca

hardwarearxiv-cs-lg

M^2Weather: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting

arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather vari

PhysVista: Benchmarking Physical Intelligence in VLMs via a Perception-Reasoning-Assessment Loop

model-releasesarxiv-cs-cv

arXiv:2610.00559v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities, yet whether they truly capture the physical consistency underlying re

SALD: Self-Referenced Advantage Learning for Diffusion Models

arXiv:2610.01496v1 Announce Type: new Abstract: Recent work on language-model adaptation has shown that single models can obtain informative training signals by evaluating their behavior in demonstrat

researcharxiv-cs-cv

TabJoinBench: A Benchmark for Joinable Table Discovery

arXiv:2610.00817v1 Announce Type: cross Abstract: Join discovery aims to identify tables from large data repositories that can augment a query table with complementary information, enabling downstream

Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting

arXiv:2609.40117v1 Announce Type: new Abstract: Time-series forecasting models achieve strong benchmark performance but exhibit severe systematic bias in industrial deployments. This train--deploy gap

Agents

cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

arXiv:2609.40284v1 Announce Type: cross Abstract: Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on m

agentsarxiv-cs-ai

Disagreement-Regularized Imitation Learning for Image-Based Continuous Control with Gaussian and Beta Policies

arXiv:2609.38407v1 Announce Type: new Abstract: Purpose: Behavior cloning can accumulate errors when a learned controller visits states outside the demonstrated distribution. This study evaluates whet

safetyarxiv-cs-lg

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

arXiv:2609.39013v1 Announce Type: new Abstract: EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, sele

researcharxiv-cs-cl

Tutorials

From Search to Signal: Online Post-Training in Automatic Heuristic Design

arXiv:2609.39383v1 Announce Type: cross Abstract: Large language model (LLM)-based automatic heuristic design (AHD) iteratively proposes and refines heuristics, pairing design rationales with executab

tutorialsarxiv-cs-ai

MADBench: Benchmarking the Security of Multi-Agent Debate

arXiv:2609.39146v1 Announce Type: new Abstract: Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same

MM-FinEval: A Multi-Task Multimodal Benchmark for Real-World Financial Forecasting

arXiv:2609.38523v1 Announce Type: cross Abstract: Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle com

OpenCollab: A Multi-Agent Coding Framework with Programmable Collaboration and Controllable Runtime

arXiv:2609.38345v1 Announce Type: cross Abstract: Multi-agent coding systems are designed to tackle complex software engineering tasks through collaboration. However, existing evaluations typically as

QuantCode Model: Specializing Language Models for Executable Algorithmic Trading Code

arXiv:2609.39420v1 Announce Type: new Abstract: Large language models are strong general-purpose code generators, but executable algorithmic trading remains a demanding specialization target: a model

RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection

arXiv:2609.39229v1 Announce Type: cross Abstract: Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary fro

RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

arXiv:2609.39551v1 Announce Type: new Abstract: Auto-research agents, LLM systems that propose, implement, train, and evaluate model changes across iterations, promise to automate applied ML's experim

Role-guided Speaker Deletion Verification in Clinical Psychiatry Speech Recordings with Audio Language Models

arXiv:2609.38491v1 Announce Type: new Abstract: Clinical research in psychiatry increasingly relies on large scale collection of spoken language data to identify acoustic and linguistic biomarkers. Ye

Search Shapes Conclusions: Auditing Evidence Selection Bias in Deep Research Agents

arXiv:2609.39026v1 Announce Type: new Abstract: Deep Research agents synthesize evidence into cited reports, yet a well-cited report can still reach a misleading conclusion. Citation correctness check

SETA: Scaling Environments for Terminal Agents

arXiv:2607.10891v2 Announce Type: replace Abstract: Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interf

Agents

SimTrace: Grounded Multimodal User Trajectories Generation for Online User Modeling

arXiv:2609.38397v1 Announce Type: new Abstract: Virtual clients offer a cost-effective approach to support applications such as A/B testing, recommender system development, and interface evaluation. H

agentsarxiv-cs-ai

Agents

TAGGRAPH: Tag-Augmented Graphs for Graph Retrieval of Agent Persistent Histories

arXiv:2609.38353v1 Announce Type: cross Abstract: Long-term memory lets LLM agents recall past interactions and remain consistent across sessions, but memory systems are hard to compare because they o

agentsarxiv-cs-ai

ViTeX-Bench: Benchmarking High-Fidelity Video Scene Text Editing

arXiv:2609.40356v1 Announce Type: cross Abstract: Recent video generation is increasingly realistic and controllable, yet video editing remains less developed, particularly for precise local edits tha

AeroManip-VLA: Scalable Vision-Language-Action Learning for Aerial Manipulation with RL-Generated Demonstrations

arXiv:2609.36915v1 Announce Type: cross Abstract: Aerial manipulators extend robotic manipulation into 3D workspaces that are difficult for ground-based robots to access, creating new opportunities fo

Authority Bias in Language Models: Source Deference and User Agreement Are Not Interchangeable

arXiv:2609.37616v1 Announce Type: cross Abstract: Language models tend to agree with whatever a user asserts, and post-training increasingly targets this sycophancy so that models evaluate claims on t

safetyarxiv-cs-cl

CAD-Native Transformer Operators for AI-Aided Engineering

arXiv:2609.36806v1 Announce Type: new Abstract: Modern engineering systems, from automobiles to aircraft, are designed by using precise, continuous parametric computer-aided design (CAD) models. Evalu

researcharxiv-cs-ai

CruxBench: A Benchmark of Information Discovery

arXiv:2609.35879v1 Announce Type: cross Abstract: Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many comp

Tutorials

Do LLMs Really Forget? Hidden-State Leakage in Model Unlearning and How to Fix it

arXiv:2609.36612v1 Announce Type: new Abstract: Unlearning in large language models (LLMs) is typically evaluated at the output level, where a model appears to suppress sensitive or undesirable conten

tutorialsarxiv-cs-lg