Search: “benchmark”

Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps

arXiv:2604.19533v1 Announce Type: cross Abstract: We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of th

22 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Are We Using the Right Benchmark: An Evaluation Framework for Visual Token Compression Methods

DGX agent

arXiv:2510.07143v3 Announce Type: replace Abstract: Recent efforts to accelerate inference in Multimodal Large Language Models (MLLMs) have largely focused on visual token compression. The effectivene

model-releasesarxiv-cs-cv

21 Apr 2026

CaseFacts: A Benchmark for Legal Fact-Checking and Precedent Retrieval

arXiv:2601.17230v2 Announce Type: replace Abstract: Automated Fact-Checking has largely focused on verifying general knowledge against static corpora, overlooking high-stakes domains like law where tr

I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For exampl…

model-releasesethan-mollick--x

I find that open weights models over-perform on benchmarks compared to actual real-world usage, and Kimi feels like no exception. For example, a small amount of use will show that Kimi is not as good

MathNet: a Global Multimodal Benchmark for Mathematical Reasoning and Retrieval

arXiv:2604.18584v1 Announce Type: cross Abstract: Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in

ReCoQA: A Benchmark for Tool-Augmented and Multi-Step Reasoning in Real Estate Question and Answering

arXiv:2604.17944v1 Announce Type: new Abstract: Developing agents capable of navigating fragmented, multi-source information remains challenging, primarily due to the scarcity of benchmarks reflecting

SPENCE: A Syntactic Probe for Detecting Contamination in NL2SQL Benchmarks

arXiv:2604.17771v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on natural language to SQL (NL2SQL) benchmarks, yet their reported accuracy may be inflate

MAS-Bench: A Unified Benchmark for Shortcut-Augmented Hybrid Mobile GUI Agents

arXiv:2509.06477v2 Announce Type: replace Abstract: Shortcuts such as APIs and deep-links have emerged as efficient complements to flexible GUI operations, fostering a promising hybrid paradigm for ML

17 Apr 2026

Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality

arXiv:2604.05510v2 Announce Type: replace Abstract: Augmented reality (AR) has rapidly expanded over the past decade. As AR becomes increasingly integrated into daily life, its security and reliabilit

14 Apr 2026

Robust Reasoning Benchmark

arXiv:2604.08571v1 Announce Type: cross Abstract: While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly ov

13 Apr 2026

ATBench: A Diverse and Realistic Agent Trajectory Benchmark for Safety Evaluation and Diagnosis

arXiv:2604.02022v2 Announce Type: replace Abstract: Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions

FinTruthQA: A Benchmark for AI-Driven Financial Disclosure Quality Assessment in Investor -- Firm Interactions

arXiv:2406.12009v5 Announce Type: replace Abstract: Accurate and transparent financial information disclosure is essential for market efficiency, investor decision-making, and corporate governance. Ch

MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

arXiv:2604.06846v1 Announce Type: cross Abstract: Interactive medical dialogue benchmarks have shown that LLM diagnostic accuracy degrades significantly when interacting with non-cooperative patients,

Toward Memory-Aided World Models: Benchmarking via Spatial Consistency

arXiv:2505.22976v2 Announce Type: replace-cross Abstract: The ability to simulate the world in a spatially consistent manner is a crucial requirements for effective world models. Such a model enables

AuthGateBench: Benchmarking LLMs as Authorization Gate

arXiv:2610.16426v1 Announce Type: new Abstract: Automated approval systems let coding agents execute high-risk actions with minimal human oversight by delegating the authorization decision to a classi

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

arXiv:2610.15634v1 Announce Type: new Abstract: Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribu

ChemArgus: Automated Scientific Diagnostics for Lab-Grade Multimodal Chemistry Reasoning: A Frontier Benchmark

arXiv:2610.20012v1 Announce Type: new Abstract: Chemistry evaluation has moved past multiple-choice probes toward open-ended expert problems, yet the benchmarks that define the current frontier still

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

arXiv:2610.19165v1 Announce Type: cross Abstract: Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively t

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

arXiv:2608.28281v2 Announce Type: replace Abstract: Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitione

PatTransBench: an ipc-stratified, multi-dimensional benchmark for patent machine translation evaluation

arXiv:2610.16538v1 Announce Type: cross Abstract: We present PatTransBench, a 12,000-item Chinese-English/Japanese/Korean patent machine translation (PatMT) benchmark with a four-dimensional evaluatio

SLS-Bench: A Benchmark for Incident Log Summarization with Synthetic Observability Data

arXiv:2610.16412v1 Announce Type: new Abstract: As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizi

Towards a Benchmark for Fire Safety Code Interpretation in the Hong Kong AEC Industry

arXiv:2610.16127v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly explored in the Architecture, Engineering, and Construction (AEC) domain, yet their reliability in regul

When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark

arXiv:2610.16067v1 Announce Type: cross Abstract: Agent evaluation harnesses are software systems that run test suites, generate inputs, and decide outcomes with test oracles. This paper audits such s

HelpMeOut: A Neuropsychology-Grounded Benchmark for Error-Aware Monitoring of Household Task Execution

model-releasesarxiv-cs-ro

arXiv:2610.17694v1 Announce Type: new Abstract: We introduce HelpMeOut, an embodied AI benchmark for error-aware monitoring of household task execution. Existing embodied household benchmarks typicall

How to Build a Benchmark That Actually Measures Taste

arXiv:2610.14707v1 Announce Type: cross Abstract: Expert-aligned 'taste' benchmarks should reward craft judgment, not the ability to game the test. We introduce TasteBench, a pairwise good-vs-bad benc

InCLAD: A Continual Learning Benchmark for Industrial Visual Anomaly Detection

arXiv:2610.15032v1 Announce Type: new Abstract: Industrial anomaly detection benchmarks have advanced rapidly, yet they remain largely static and therefore underrepresent deployment settings in which

LABIT: Long-Horizon Robotic Assembly Benchmark for Industrial Tasks

model-releasesarxiv-cs-ro

arXiv:2610.16398v1 Announce Type: new Abstract: Robot learning has undergone significant advancements in the domain of compliant control and robotic assembly. The development of novel methodologies ha

OmniCompass: A Holistic Benchmark for Omni-Modal Video Grounding and Understanding

arXiv:2610.18829v1 Announce Type: new Abstract: With the rapid development of multimodal large language models (MLLMs), evaluating their video understanding capabilities has become increasingly import

PVFH-Bench: Benchmarking Premise-Induced Visual Fixation Hallucinations in Vision-Language Models

arXiv:2610.14635v1 Announce Type: cross Abstract: Large Vision-Language Models (LVLMs) have achieved remarkable progress across a wide range of multimodal tasks. However, object hallucination remains

SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models

arXiv:2601.04824v3 Announce Type: replace Abstract: Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video r

Coding-Agent Benchmarks Should Match Their Users' Task Flows

arXiv:2610.09633v1 Announce Type: cross Abstract: The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engin

8 Oct 2026

LOGIC: An LLM Benchmark for Intent-Grounded Change Impact in Aerospace Electrical Systems

arXiv:2610.07580v1 Announce Type: new Abstract: Aerospace electrical-design revisions can contain multiple genuine changes, although an engineering request may authorize only a subset. Propagating eve

ParanoiaEval: Benchmarking Unnecessary Defensive Work in Agentic Coding

arXiv:2610.08662v1 Announce Type: new Abstract: As coding agents increasingly undertake real-world work autonomously, judging whether their risk treatments are warranted has become important. Existing

PolarScale: A Physics-Grounded Benchmark for Radiometrically Consistent RGB-to-Stokes Estimation

arXiv:2610.08346v1 Announce Type: new Abstract: Polarization imaging provides physical cues beyond intensity imaging but typically requires specialized hardware. Recent methods infer polarization from

The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation

arXiv:2610.08026v1 Announce Type: new Abstract: In recent years, several methods for detecting when large language models (LLMs) hallucinate have been developed. These methods are often benchmarked wi

Towards a Unified Misuse Monitoring Benchmark

arXiv:2610.07089v1 Announce Type: cross Abstract: LLM agents increasingly act in multi-actor environments, exposing them to misuse from multiple sources: decomposition attacks, where a harmful request

Detecting Defects that Matter: An Application-Driven Benchmark for Anomaly Detection in Manufacturing and Retail Logistics (VAND 4.0 Challenge)

arXiv:2610.04392v1 Announce Type: new Abstract: Existing Anomaly Detection benchmarks are saturated and often unrealistic. As part of the VAND 4.0 Challenge, we introduce a hidden-test, application-dr

6 Oct 2026

InteractionBench: A Real-Time Interaction Benchmark for Streaming Video Systems

arXiv:2610.05775v1 Announce Type: new Abstract: A video assistant must speak when its instruction warrants a response and stay silent otherwise. We introduce a benchmark that evaluates this decision f

6 Oct 2026

Same Output, Different Gold: Measuring How Reference Choice Moves a Multilingual Benchmark Score

arXiv:2610.03825v1 Announce Type: cross Abstract: A benchmark score compares a system output against a reference, and methodological attention falls almost entirely on the first term. We measure the s

6 Oct 2026

EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents

arXiv:2610.03153v1 Announce Type: cross Abstract: Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources

5 Oct 2026

MintEval: Do LLMs Implement the Trading Strategy You Asked For? A Behavioural-Equivalence Benchmark for Natural-Language-to-Strategy Code

arXiv:2610.03080v1 Announce Type: cross Abstract: Large language models are moving from producing trading signals to writing the code that executes them. The failure mode of the second role is silent:

5 Oct 2026

RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

arXiv:2610.02242v1 Announce Type: cross Abstract: Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize y

5 Oct 2026

Benchmarking Prompt Optimization of Large Language Models With Chess

arXiv:2610.00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamin

CompMat-Bench: Benchmarking AI Agents for Computational Materials Science

arXiv:2610.00636v1 Announce Type: new Abstract: Evaluating AI agents on scientific research tasks is constrained by the time and resources required for the underlying experiments or calculations. In c

Cross-Benchmark Transfer from RL on Agentic Coding Tasks

arXiv:2610.00890v1 Announce Type: cross Abstract: Coding agents often fail in the last mile: they build most of a feature but drop a requirement, test only the cases their implementation already handl

DAYJOB: A Benchmark for Long-Horizon Professional Work

arXiv:2610.01306v1 Announce Type: new Abstract: Professional work often starts with a brief request that leaves the professional to work out what is needed, which documents matter, and whether the req

Detect, Explain, Interpret: An End-to-End Benchmark for Time Series Anomaly Detection, Explainability and Interpretability

arXiv:2610.01168v1 Announce Type: cross Abstract: Time Series Anomaly Detection has received increasing attention, driven by the growing availability of complex time series data. This surge has led to

M^2Weather: A Benchmark for Joint Multi-Station and Multi-Variable Weather Forecasting

arXiv:2610.00370v1 Announce Type: new Abstract: Station weather forecasting is fundamentally shaped by both complex spatial dependencies across stations and strong physical coupling among weather vari