Search: “benchmark”

Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

arXiv:2609.21267v1 Announce Type: new Abstract: Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation fo

21 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

KoUniTalk: A Lightweight Articulation-Centered Korean-English 3D Talking Face Benchmark

DGX agent

arXiv:2609.19840v1 Announce Type: new Abstract: High-quality 3D talking face datasets remain largely English- centric, and Korean 3D facial motion data are difficult to combine with standard English b

model-releasesarxiv-cs-cv

18 Sept 2026

PetriBench: Benchmarking LLM Reasoning over Dynamic State Spaces

arXiv:2609.19883v1 Announce Type: cross Abstract: Characterizing LLM reasoning remains an open challenge, as many existing benchmarks isolate specific reasoning skills, rely on external knowledge, or

18 Sept 2026

Agents

What Do We Expect from LLMs? Mapping the Design of LLM Benchmarks

arXiv:2609.19182v1 Announce Type: new Abstract: Benchmarks are central to how progress in large language models (LLMs) is assessed and communicated. Yet model rankings alone reveal little about how ev

agentsarxiv-cs-ai

18 Sept 2026

HearInContext: A Benchmark for Implicit Context in Speech Recognition

arXiv:2609.18680v1 Announce Type: new Abstract: Contextual ASR can benefit from semantic cues or from target words explicitly provided in the context. We introduce HearInContext, a Mandarin--English b

17 Sept 2026

Japanese Stroke LLM Evaluation: A Conversational Benchmark for Safe Stroke Care in Japanese Using Large Language Models

arXiv:2609.16739v1 Announce Type: new Abstract: Background: Large language models (LLMs) have achieved physician-comparable performance on multiple-choice medical knowledge examinations, but their cap

16 Sept 2026

MechReason: Benchmarking Multi-Image Multi-Hop Reasoning in Mechanical Engineering

arXiv:2609.16012v1 Announce Type: new Abstract: Despite significant progress in general visual question answering and cross-modal understanding, multimodal large language models still face a pronounce

16 Sept 2026

Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands

arXiv:2609.15726v1 Announce Type: cross Abstract: Tactile sensing provides contact information that can be difficult to infer from vision alone, but tactile hardware for dexterous hands has not conver

Evaluating Contextual Bias in CNN Image Classification: Evidence from Agricultural Benchmark Datasets

arXiv:2609.14654v1 Announce Type: new Abstract: Convolutional neural networks (CNNs) are typically evaluated using held-out classification accuracy, an approach that presupposes predictions are based

From Density to Biopsy Decisions and Malignancy Prediction: A Benchmark Study of Multimodal Large Language Models Against Radiologists in Digital and Contrast-Enhanced Mammography

arXiv:2609.14676v1 Announce Type: new Abstract: Purpose: To compare four multimodal large language models (MLLMs) with radiologists of varying expertise in breast density assessment, BI-RADS assessmen

From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting

arXiv:2609.13606v1 Announce Type: cross Abstract: Multi-arm robotic harvesting offers a promising path to improve harvesting efficiency and reduce reliance on manual labor. However, practical deployme

K-Bench: A Benchmark for LLM Unlearning in Agentic Deployments

arXiv:2609.12808v2 Announce Type: replace Abstract: Unlearning benchmarks such as TOFU and MUSE certify forgetting by reading the model's final answer, where a model that refuses to answer already cou

KnowBench: Effort Reduction as a Unified, Deployment-Grounded Benchmark for Clinical AI

arXiv:2609.15794v1 Announce Type: new Abstract: Clinical AI systems are evaluated with instruments built for research settings (reference-based similarity metrics and expert rubric panels) that measur

One Example Is Enough to Pass Fairness Benchmarks: Rethinking Fairness Evaluation for Aligned LLMs

arXiv:2609.14860v1 Announce Type: cross Abstract: Warning: This submission studies stereotypes and biases, and contains toxic and offensive examples, used for illustration purposes only. Fairness benc

TriCalRAG: A Three-Strategy, Retrieval-Augmented Benchmark for On-Premise LLM-Based Root Cause Analysis in AIOps

arXiv:2609.14762v1 Announce Type: cross Abstract: Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk

Research

A Dataset and Benchmarks for Atrial Fibrillation Detection from Electrocardiograms of Intensive Care Unit Patients

arXiv:2512.18031v2 Announce Type: replace Abstract: Objective: Atrial fibrillation (AF) is the most common cardiac arrhythmia experienced by intensive care unit (ICU) patients and can cause adverse he

researcharxiv-cs-lg

14 Sept 2026

FINESSE: An Agent-Based Simulator and Benchmark Dataset for Multimodal Financial Event Sequences

model-releasesarxiv-cs-lg

arXiv:2609.11993v1 Announce Type: new Abstract: Machine learning research in financial services is limited by the scarcity of representative open-source datasets. Existing resources are often narrowly

14 Sept 2026

Hardware

HoliBench: A Cross-Platform Benchmarking and Deployment Toolkit for Foundation Models in CPS-IoT Applications

arXiv:2609.12412v1 Announce Type: cross Abstract: Foundation models, including large language models, vision-language models, and time-series foundation models, are increasingly deployed on embedded a

hardwarearxiv-cs-lg

14 Sept 2026

SemVerBench: Benchmarking LLM Comprehension of Version-Constraint Resolution Semantics

arXiv:2609.11180v1 Announce Type: new Abstract: Large language model (LLM) coding agents constantly decide whether a version satisfies a constraint such as ^1.2.3 or >=2.0,1.2 means >=1.3.0) traps eve

12 Sept 2026

When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting

arXiv:2609.11282v1 Announce Type: new Abstract: Multimodal forecasting models that combine time series with text annotations promise richer prediction through textual context, but how do we know wheth

12 Sept 2026

CS-Guard: Benchmarking LLM Guardrails for Code Generation Security

arXiv:2609.09798v1 Announce Type: cross Abstract: Large language models (LLMs) have been ex- ploited to generate malware, but the effective- ness of guardrails for code generation secu- rity remains u

DeFiFlowBench: Benchmarking and Improving Safe Executability in Natural-Language DeFi Workflow Synthesis

model-releasesarxiv-cs-lg

arXiv:2609.11504v1 Announce Type: new Abstract: A structurally valid DeFi workflow can still authorize a costly trade. We introduce DeFiFlowBench, a benchmark of 207 team-authored prompts for natural-

Does YOLO26 Truly Offer Advantages Over Its Predecessors for Edge Deployment? A Benchmark Study in Aquaculture

arXiv:2607.09835v2 Announce Type: replace Abstract: The recently introduced YOLO26 architecture incorporates NMS-free end-to-end inference and is optimized for deployment on resource-constrained CPU-b

MMGait: Benchmarking and Unifying Gait Recognition across Heterogeneous Modalities

arXiv:2609.11601v1 Announce Type: new Abstract: Gait recognition is commonly studied using RGB videos or their derived silhouettes and poses. Yet human walking produces heterogeneous photometric, geom

Shedding Light: A Benchmark for Evaluating Lighting Understanding in Generative Image Models

arXiv:2609.10787v1 Announce Type: new Abstract: Accurate modelling of illumination is central to realistic image synthesis and scene understanding. Yet, there is little exploration into whether image

Attribution in Scientific Literature: New Benchmark and Methods

arXiv:2405.02228v4 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly generate citation-backed responses, yet citation hallucination remains a major challenge for trustwo

10 Sept 2026

AVSRBench: A Multi-Condition AVSR Benchmark

arXiv:2609.10366v1 Announce Type: cross Abstract: While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true g

10 Sept 2026

ProcArena: A Multi-Scenario Benchmark for LLMs on Direct and Interactive PL/SQL Development from Natural Language

arXiv:2609.06527v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasin

10 Sept 2026

Benchmarking Dexterity of Multifingered Robot Hands: A Review and Perspective

model-releasesarxiv-cs-ro

arXiv:2609.05585v1 Announce Type: new Abstract: Robot hands are a key interface between AI and the physical world, making advances in robotic dexterity essential to realizing the vision of physical AI

GAN-Blot: A Controllable Structure-Style Synthesis Benchmark for Western Blot Forensics

arXiv:2609.06619v1 Announce Type: new Abstract: Western blot (WB) images are widely used as key evidence in biomedical research. Recent scientific misconduct cases reveal that WB imagery is increasing

LANTERN: A Closed-Loop Benchmark for VLM-Based Cooperative Driving with Temporally Grounded Warnings

model-releasesarxiv-cs-ro

arXiv:2609.06368v1 Announce Type: new Abstract: We present LANTERN, a closed-loop benchmark for temporally grounded cooperative warnings. LANTERN separates warning onset, hazard onset, warning termina

NEO-BENCH: A New Multi-Source Benchmark for Generalizable Astronomical Streak Detection

arXiv:2609.06774v1 Announce Type: cross Abstract: Near-Earth Objects (NEOs) can appear as faint streaks in long-exposure astronomical images. Detecting these streaks across diverse observatories requi

RevalExo: A Functional Daily-Activity Benchmark for Inertial and Visual Locomotion Mode Recognition in Older Adults and Clinical Cohorts

arXiv:2609.08090v1 Announce Type: cross Abstract: Assistive devices for people with mobility impairments, such as powered exoskeletons, rely on accurate locomotion mode recognition to adapt control st

Towards Embodied Air-Ground Cooperative Object Search: Benchmark, Dataset and Agentic Method

arXiv:2609.08402v1 Announce Type: new Abstract: Air-Ground Object Search (AGOS) in urban environments is a challenging embodied task, which requires an Unmanned Aerial Vehicle (UAV) and an Unmanned Gr

PetQA: Benchmarking Veterinary Knowledge and Clinical Reasoning

arXiv:2609.04598v1 Announce Type: cross Abstract: We introduce PetQA, a Korean long-form question-answering (QA) benchmark for evaluating veterinary knowledge and clinical reasoning in large language

7 Sept 2026

PRISM-Bench: An Audio-Centric Diagnostic Benchmark for Text-to-Audio-Video Generation

arXiv:2609.04867v1 Announce Type: cross Abstract: Text-to-audio-video (T2AV) generation has advanced rapidly, but its evaluation still underestimates the audio modality. Existing benchmarks either tre

7 Sept 2026

TIER: Threat Implicitness Benchmark for Evaluating LLM Safety Behaviors

arXiv:2609.05117v1 Announce Type: cross Abstract: Current LLM safety benchmarks largely rely on binary metrics, overlooking how models respond to harmful prompts with varying threat implicitness. We i

7 Sept 2026

Benchmarking Machine Translation on Chinese Social Media Texts

arXiv:2601.22931v2 Announce Type: replace Abstract: The prevalence of rapidly evolving slang, neologisms, and highly stylized expressions in informal user-generated text, particularly on Chinese socia

4 Sept 2026

EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models

arXiv:2609.01611v1 Announce Type: new Abstract: Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave different

3 Sept 2026

TAPVid-MV: A Benchmark for Tracking Any Point in 3D Across Multiple Views

arXiv:2609.01899v1 Announce Type: new Abstract: Multi-camera systems are increasingly practical for robotics, AR/VR, and autonomous driving because complementary views reduce depth ambiguity and prese

3 Sept 2026

CoLT-Drive: Counterfactual Long-Tail Benchmarking and Knowledge-Preserving Adaptation for Driving Affordance Prediction

arXiv:2609.00242v1 Announce Type: cross Abstract: Long-tail autonomous driving failures are often framed as rare-object recognition errors. We argue that this view is incomplete: the decision-critical

DramaChain Bench: An End-to-End Benchmark for Short-Drama Generation

arXiv:2609.00646v1 Announce Type: new Abstract: Commercial short-drama production follows a multi-stage chain: script, storyboard, keyframe imagery, shot-level video, and the finished short drama. Mos

HugAgent: A Human Simulation Benchmark for Individual-Level Reasoning

arXiv:2510.15144v4 Announce Type: replace Abstract: Simulating human reasoning in open-ended tasks has long been a central aspiration in AI and cognitive science. While large language models now appro

LLM-Driven Autonomous Vehicles Inherit Human Driver Biases in Pedestrian Yielding: Results and Implications From A New Benchmark

arXiv:2609.00192v1 Announce Type: new Abstract: Public trust in Autonomous Vehicles (AVs) may depend not only on technical success but also on the fairness of their decision making. While a recent tre

Research

Benchmarking Peptide-Protein Affinity Prediction Across Peptide and Target Shifts

arXiv:2608.30175v1 Announce Type: new Abstract: Peptide-protein affinity models are often evaluated with a single data split, obscuring whether they interpolate among measurements for observed targets

researcharxiv-cs-lg

Do Small Models Use the Law You Give Them? Measuring Context Use on a Bilingual Bangladesh Legal Benchmark

arXiv:2608.30327v1 Announce Type: new Abstract: Fine-tuning can improve legal question-answering accuracy without improving how models use law supplied in context. We study this distinction in bilingu

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

arXiv:2608.30678v1 Announce Type: new Abstract: Text-rich image understanding requires multimodal large language models (MLLMs) to organize OCR (Optical Character Recognition)-grounded evidence across

SpatialTrust: A Benchmark for Environmental Risk Recognition in Secure Authentication

arXiv:2608.29489v1 Announce Type: cross Abstract: Visual environmental risk recognition plays an important role in secure authentication, where a user's surroundings may reveal sensitive information o