Search: “evaluation”

arXiv:2609.21257v1 Announce Type: cross Abstract: Large language model (LLM) agents can propose, implement, and evaluate model changes. Autoresearch loops demonstrate this capability through minutes-s

hardwarearxiv-cs-ai

21 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

When Better Turns Do Not Make Better Agents: Diagnosing the Gap Between Next-Turn Metrics and Workflow Success

DGX agent

arXiv:2609.21187v1 Announce Type: new Abstract: Agent models are frequently evaluated one decision at a time, where the model predicts the next action based on the gold interaction history, which is s

model-releasesarxiv-cs-cl

21 Sept 2026

AntiGrounding: Executable Robot Trajectories as Visual Prompts for VLM-Guided Manipulation

arXiv:2506.12374v3 Announce Type: replace-cross Abstract: Natural-language manipulation instructions specify the task goal but leave the underlying robot trajectory unspecified. We present AntiGroundi

safetyarxiv-cs-ai

18 Sept 2026

It ain’t independent if you are partnering with them and chose the partner.

It ain’t independent if you are partnering with them and chose the partner. We’re partnering with Accenture on independent evaluation of frontier AI—part of our recent commitment to embed evaluators a

safetygary-marcus--x

18 Sept 2026

Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcuts

arXiv:2609.18366v1 Announce Type: new Abstract: Reliable agent evaluation is complicated by automatic harness optimization, which repeatedly uses a released benchmark B_{rel} to guide a Proposer that

Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

arXiv:2609.18909v1 Announce Type: cross Abstract: Agent benchmarks are substantially more costly to evaluate than conventional LLM benchmarks. Benchmark compression is therefore a natural solution, ye

Do Social Patterns Hold in Synthetic Data? Analyzing Cyberbullying Dynamics in LLM-Generated and Authentic Dialogues

arXiv:2609.17549v1 Announce Type: new Abstract: Cyberbullying (CB) is a complex social phenomenon characterized by repeated aggression, power imbalance, and multi-party interaction. Although large lan

GYROval: A Robust Benchmark for Cultural Value Orientation in Large Language Models

arXiv:2609.18384v1 Announce Type: cross Abstract: We present a robust benchmark for measuring cultural value orientation in large language models on the two Inglehart-Welzel axes over several domains

Made in Hungary: Comments on the performance of generative language models

arXiv:2609.18284v1 Announce Type: new Abstract: In recent years, three initiatives have emerged to develop generative language models in Hungary. The motivation behind them is the same. For Hungarian,

researcharxiv-cs-cl

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

arXiv:2609.18805v1 Announce Type: cross Abstract: Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents ma

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

arXiv:2609.16233v1 Announce Type: cross Abstract: Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benc

Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?

arXiv:2507.23248v2 Announce Type: replace Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task

The Neverwhere Visual Parkour Benchmark Suite

arXiv:2609.16443v1 Announce Type: cross Abstract: State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world pe

Which Pretext Task Transfers? Self-Supervised Pretraining Objectives for Lung Ultrasound

arXiv:2609.16551v1 Announce Type: new Abstract: Self-supervised learning (SSL) can reduce the need for labelled medical images, but the choice of pretext objective remains unclear for lung ultrasound

researcharxiv-cs-cv

A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion

arXiv:2609.13511v1 Announce Type: cross Abstract: Large language models are increasingly benchmarked against classical machine learning for network intrusion detection (NIDS), almost always using same

researcharxiv-cs-ai

Hardware

Accelerating HKTex without Mesh Eigensystems: Local Unfolding and Randomized Thermal Features

arXiv:2609.14105v1 Announce Type: new Abstract: Heat Kernel Textures (HKTex) represent surface appearance with intrinsic anisotropic kernels, but evaluate them using 50 global Laplace-Beltrami eigende

hardwarearxiv-cs-cv

Are LLMs Good Financial User Simulators? A Preliminary Study

arXiv:2609.15727v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as user simulators, but their ability to reproduce evolving individual financial decisions remains un

researcharxiv-cs-ai

Building a Production Greek-English Speech Recognizer

arXiv:2609.13498v1 Announce Type: cross Abstract: We report a multi-month engineering program to build Sophea, a production bilingual Greek-English automatic speech recognition system. We evaluate the

Comparing Trajectories from Positions Alone: Curvature-Based Time Alignment and Drift Error Metric

arXiv:2609.14936v1 Announce Type: new Abstract: In field robotics, acquiring independent large-scale reference trajectories more accurate than the evaluated estimates remains an open challenge. The do

safetyarxiv-cs-ro

DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

arXiv:2609.15122v1 Announce Type: cross Abstract: Generative CAD models are expected to remain behaviorally correct after parameter edits, so increasing the number of edit checks is often treated as a

Designing UNICORN: a Unified Benchmark for Imaging in Computational Pathology, Radiology, and Natural Language

arXiv:2603.02790v2 Announce Type: replace Abstract: Foundation models are changing the way we develop medical artificial intelligence. By learning broadly generalizable features across diverse data mo

Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Models

arXiv:2609.13556v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized te

E2A-Bench: Benchmarking Evidence-to-Action Reliability in Financial Chart Reasoning

arXiv:2609.14302v1 Announce Type: new Abstract: Can financial vision-language models (VLMs) turn chart evidence into reliable action recommendations? Existing hallucination evaluations are mostly clai

Agents

From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaboration

arXiv:2609.13261v1 Announce Type: cross Abstract: LLM agents are increasingly used for collaborative problem solving and human-group simulation. This makes outcome-only evaluation insufficient: if LLM

agentsarxiv-cs-ai

Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Models

arXiv:2509.19375v2 Announce Type: replace-cross Abstract: Large language models are increasingly used for clinical text classification, where overconfident misclassifications can directly affect patie

ZAPS: Zero-Cost Active Proxy Search for Neural Architecture Search

arXiv:2609.14184v1 Announce Type: new Abstract: Neural Architecture Search (NAS) automates network design, but evaluating a single candidate requires training it to convergence, making exhaustive sear

researcharxiv-cs-lg

Can We Trust LLM Judges: A Study of Capability-Dependent Biases and Multi-Judge Ensemble for Bias Calibration

arXiv:2609.12002v1 Announce Type: new Abstract: LLMs are increasingly used as automated judges for model training and evaluation, yet individual judges exhibit systematic biases that undermine reliabi

safetyarxiv-cs-lg

Certifying Concept Unlearning in Text-to-Image Diffusion Models

arXiv:2609.12163v1 Announce Type: new Abstract: Existing evaluations of concept unlearning in text-to-image (T2I) diffusion models primarily rely on attack success rates obtained through automated adv

safetyarxiv-cs-lg

ChitraMiti: Benchmarking Visual Grounding and Modality Reliance in Bengali Geometric Reasoning

arXiv:2609.12509v1 Announce Type: new Abstract: Evaluation of vision-language models (VLMs) for multimodal mathematical reasoning remains limited for low-resource languages and for geometry problems t

Is Bash All You Need? An Empirical Study of Tool Interfaces for Enterprise Digital Worker Agents

arXiv:2609.11999v1 Announce Type: cross Abstract: In this study, we examine whether a general shell can outperform specialized tools on enterprise tasks. Shell-based agents have shown strong results i

RiPPLE: Cross-Space Performance Prediction from Early Training for Neural Architecture Search

arXiv:2609.12418v1 Announce Type: cross Abstract: Neural architecture search (NAS) evaluates candidate networks, but fully training enough architectures to rank an entire space is expensive. Zero-cost

Compute-Bounded Security Assurance - Coverage, Verification, and Response under Resource Constraints

arXiv:2609.09229v1 Announce Type: cross Abstract: Additional inference compute can increase the number of correctly resolved security-assurance tasks, but repeated success, unique coverage, accepted e

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

arXiv:2609.09798v1 Announce Type: cross Abstract: Large language models (LLMs) have been ex- ploited to generate malware, but the effective- ness of guardrails for code generation secu- rity remains u

SoK: Privacy Attacks on Machine Learning via Explainable AI

model-releasesarxiv-cs-lg

arXiv:2609.10627v1 Announce Type: cross Abstract: Machine learning explanations reveal model behavior beyond predictions, creating attack surfaces for model confidentiality and data privacy. We system

Temporal and Multimodal Deep Learning for Cyberattack Detection in LEO Satellite Systems

arXiv:2609.10746v1 Announce Type: cross Abstract: The growing reliance on Low-Earth Orbit (LEO) satellite communication systems has increased the need for intelligent methods capable of detecting cybe

researcharxiv-cs-lg

The Era by Eon Benchmark: A Generated Enterprise Estate with Exact Ground Truth for Benchmarking LLM Agents

arXiv:2609.09853v1 Announce Type: new Abstract: LLM agents for enterprise systems of record cannot be evaluated on customer production data, and no existing substitute provides ground truth. We presen

ACEA: An Adversarial Co-Evolution Arena for Head-to-Head Red-Team and Blue-Team LLM Testing

arXiv:2609.08256v1 Announce Type: cross Abstract: Automated red-team attacks and blue-team defenses for large language models (LLMs) are advancing quickly. However, attackers and defenders are built a

safetyarxiv-cs-ai

AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

arXiv:2609.07611v1 Announce Type: new Abstract: Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Exis

AtomCite: Verification and Correction of Supplied Page-Level Citations in Multi-Page Documents

arXiv:2609.05802v1 Announce Type: cross Abstract: Large language models answering questions over multi-page documents are expected to cite the supporting pages, yet supplied citations are sometimes in

Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of ge

NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures

arXiv:2609.06831v1 Announce Type: new Abstract: AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI

OntologyBench: Can Dense Retrieval Satisfy Structured Biomedical Constraints?

arXiv:2609.08174v1 Announce Type: new Abstract: We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs ac

SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

arXiv:2609.09349v1 Announce Type: new Abstract: Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating g

Tutorials

uFlowCSP: Crystal Structure Prediction using Mean flow generative models

arXiv:2609.09799v1 Announce Type: cross Abstract: Crystal structure prediction (CSP) is fundamental to computational materials discovery. Generative models including CDVAE, DiffCSP, FlowMM, and Crysta

tutorialsarxiv-cs-lg

Charts Are Beyond Pixels: Probing for Layer-Wise Chart Understanding and Editing

arXiv:2609.08657v1 Announce Type: new Abstract: Charts are structured visual compositions whose elements have distinct functional roles, semantic correspondences, and visibility relations. This struct

9 Sept 2026

STP-BENCH: A Unified Systematic Benchmark for Virtual Spatial Transcriptomics from Histopathology Images

arXiv:2609.05956v1 Announce Type: new Abstract: Spatial transcriptomics (ST) provides unprecedented insights into tumor heterogeneity by capturing spatially resolved gene expression, yet its high expe

9 Sept 2026

Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Tasks

arXiv:2512.14779v2 Announce Type: replace Abstract: Standard weather forecast evaluations focus on the forecaster's perspective and on a statistical assessment comparing forecasts and observations. In

researcharxiv-cs-lg

7 Sept 2026

NavArena: Automated Construction of Goal-Oriented Navigation Benchmarks from 3D Gaussian Splatting Reconstructions