Search: “benchmark”

What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

arXiv:2605.21404v1 Announce Type: new Abstract: We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was ru

21 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

RTL-BenchMT: Dynamic Maintenance of RTL Generation Benchmark Through Agent-Assisted Analysis and Revision

DGX agent

arXiv:2605.15537v1 Announce Type: new Abstract: This paper introduces RTL-BenchMT, an agentic framework for dynamically maintaining RTL generation benchmarks. Large Language Models (LLMs) assisted aut

model-releasesarxiv-cs-ai

18 May 2026

Talk is (Not) Cheap: A Taxonomy and Benchmark Coverage Audit for LLM Attacks

arXiv:2605.15118v1 Announce Type: cross Abstract: We introduce a reusable framework for auditing whether LLM attack benchmarks collectively cover the threat surface: a 4imes6 Target imes Technique mat

15 May 2026

MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents

arXiv:2512.12634v3 Announce Type: replace Abstract: Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer inte

14 May 2026

Generating Leakage-Free Benchmarks for Robust RAG Evaluation

arXiv:2605.08838v1 Announce Type: cross Abstract: Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets,

12 May 2026

Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models

arXiv:2508.04325v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However,

30 Apr 2026

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

arXiv:2604.20087v1 Announce Type: new Abstract: Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how

23 Apr 2026

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

arXiv:2604.19533v1 Announce Type: cross Abstract: We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of th

22 Apr 2026

Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

arXiv:2510.07143v3 Announce Type: replace Abstract: Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectivene

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

arXiv:2601.17230v2 Announce Type: replace Abstract: Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where tr

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

arXiv:2604.18584v1 Announce Type: cross Abstract: Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in

ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering

arXiv:2604.17944v1 Announce Type: new Abstract: Developing agents capable of navigating fragmented, multi-source information remains challenging, primarily due to the scarcity of benchmarks reflecting

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

arXiv:2604.17771v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflate

MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents

arXiv:2509.06477v2 Announce Type: replace Abstract: Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, fostering a promising hybrid paradigm for ML

17 Apr 2026

Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality

arXiv:2604.05510v2 Announce Type: replace Abstract: Augmented reality (AR) has rapidly expanded over the past decade. As AR becomes increasingly integrated into daily life, its security and reliabilit

14 Apr 2026

Robust Reasoning Benchmark

arXiv:2604.08571v1 Announce Type: cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly ov

13 Apr 2026

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

arXiv:2604.02022v2 Announce Type: replace Abstract: Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions

FinTruthQA: A Benchmark for AI-Driven Financial Disclosure Quality Assessment in Investor -- Firm Interactions

arXiv:2406.12009v5 Announce Type: replace Abstract: Accurate and transparent financial information disclosure is essential for market efficiency, investor decision-making, and corporate governance. Ch

MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

arXiv:2604.06846v1 Announce Type: cross Abstract: Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients,

Toward Memory-Aided World Models: Benchmarking via Spatial Consistency

arXiv:2505.22976v2 Announce Type: replace-cross Abstract: The ability to simulate the world in a spatially consistent manner is a crucial requirements for effective world models. Such a model enables

X-RIC: An Agentic RAN-Intelligent Controller Benchmark for Enhancing Reinforcement Learning

arXiv:2610.22269v1 Announce Type: new Abstract: The O-RAN ALLIANCE introduces architectural flexibility through disaggregation and AI-driven control. While the RAN Intelligent Controller (RIC) enables

16 Oct 2026

AuthGateBench: Benchmarking LLMs as Authorization Gate

arXiv:2610.16426v1 Announce Type: new Abstract: Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classi

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

arXiv:2610.15634v1 Announce Type: new Abstract: Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribu

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

arXiv:2610.20012v1 Announce Type: new Abstract: Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

arXiv:2610.19165v1 Announce Type: cross Abstract: Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively t

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

arXiv:2608.28281v2 Announce Type: replace Abstract: Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitione

PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

arXiv:2610.16538v1 Announce Type: cross Abstract: We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluatio

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

arXiv:2610.16412v1 Announce Type: new Abstract: As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizi

Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

arXiv:2610.16127v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regul

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

arXiv:2610.16067v1 Announce Type: cross Abstract: Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such s

HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution

model-releasesarxiv-cs-ro

arXiv:2610.17694v1 Announce Type: new Abstract: We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typicall

How to Build a Benchmark That Actually Measures Taste

arXiv:2610.14707v1 Announce Type: cross Abstract: Expert-aligned 'taste' benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benc

InCLAD: A Continual Learning Benchmark for Industrial Visual Anomaly Detection

arXiv:2610.15032v1 Announce Type: new Abstract: Industrial anomaly detection benchmarks have advanced rapidly, yet they remain largely static and therefore underrepresent deployment settings in which

LABIT: Long-Horizon Robotic Assembly Benchmark for Industrial Tasks

model-releasesarxiv-cs-ro

arXiv:2610.16398v1 Announce Type: new Abstract: Robot learning has undergone significant advancements in the domain of compliant control and robotic assembly. The development of novel methodologies ha

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

arXiv:2610.18829v1 Announce Type: new Abstract: With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly import

PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models

arXiv:2610.14635v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress across a wide range of multimodal tasks. However, object hallucination remains

SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

arXiv:2601.04824v3 Announce Type: replace Abstract: Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video r

Coding-Agent Benchmarks Should Match Their Users' Task Flows

arXiv:2610.09633v1 Announce Type: cross Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engin

8 Oct 2026

LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems

arXiv:2610.07580v1 Announce Type: new Abstract: Aerospace electrical-design revisions can contain multiple genuine changes, although an engineering request may authorize only a subset. Propagating eve

ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

arXiv:2610.08662v1 Announce Type: new Abstract: As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing

PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation

arXiv:2610.08346v1 Announce Type: new Abstract: Polarization imaging provides physical cues beyond intensity imaging but typically requires specialized hardware. Recent methods infer polarization from

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi

Towards a Unified Misuse Monitoring Benchmark

arXiv:2610.07089v1 Announce Type: cross Abstract: LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request

Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)

arXiv:2610.04392v1 Announce Type: new Abstract: Existing Anomaly Detection benchmarks are saturated and often unrealistic. As part of the VAND 4.0 Challenge, we introduce a hidden-test, application-dr

6 Oct 2026

InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

arXiv:2610.05775v1 Announce Type: new Abstract: A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision f

6 Oct 2026

Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

arXiv:2610.03825v1 Announce Type: cross Abstract: A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the s

6 Oct 2026

EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents

arXiv:2610.03153v1 Announce Type: cross Abstract: Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources

5 Oct 2026

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent: