Search: “benchmark”

Benchmarking Open-Ended Multi-Agent Coordination in Language Agents

arXiv:2606.08340v1 Announce Type: new Abstract: As language models are increasingly deployed as autonomous agents, they must coordinate with others over long horizons in open-ended interactive tasks.

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Beyond FLOPs: Benchmarking Real Inference Acceleration of LLM Pruning under a GEMM-Centric Taxonomy

DGX agent

arXiv:2606.09080v1 Announce Type: new Abstract: Pruning has emerged as a dominant paradigm for accelerating large language model (LLM) inference, spanning a broad spectrum of methods that remove compu

researcharxiv-cs-lg

9 Jun 2026

Declarative Outcome-Conformant Synthesis: Exact, Closed-Form Specification Satisfaction and a Conformance Benchmark

arXiv:2606.08736v1 Announce Type: new Abstract: We study a capability the dominant paradigm in synthetic tabular data does not provide: exact satisfaction of a declared analytical outcome with no sour

DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation

arXiv:2511.18421v2 Announce Type: replace-cross Abstract: Existing Test-time Adaptation (TTA) studies rely heavily on static and homogeneous corruption protocols, such as ImageNet-C and CIFAR-10-C/100

GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models

arXiv:2606.08194v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) integrate audio perception and language understanding within a unified framework, enabling a wide range of real-wo

Safety

Human-Centered Benchmarking of Driver Monitoring Models

arXiv:2606.08123v1 Announce Type: cross Abstract: Vision-based driver monitoring systems are increasingly deployed in safety-critical intelligent transportation settings, yet they are almost always co

safetyarxiv-cs-ai

IGenBench: Benchmarking the Reliability of Text-to-Infographic Generation

arXiv:2601.04498v2 Announce Type: replace-cross Abstract: Infographics are composite visual artifacts that combine data visualizations with textual and illustrative elements to communicate information

MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science

arXiv:2510.12171v2 Announce Type: replace Abstract: Large Language Models have shown strong scientific reasoning ability, but their performance on materials science problems remains less studied. To f

MedVision: Benchmarking Quantitative Medical Image Analysis

arXiv:2511.18676v2 Announce Type: replace-cross Abstract: Current vision-language models (VLMs) in medicine are primarily designed for categorical question answering (e.g., 'Is this normal or abnormal

Real-Time Industrial Defect Detection on Edge Hardware Using Fine-Tuned YOLOv8: A Systematic Benchmark on the NEU Surface Defect Database and MVTec AD with Automotive & Battery Manufacturing Extensions

arXiv:2606.07659v1 Announce Type: new Abstract: Automated surface defect detection is critical for ensuring rigorous quality control in high-speed manufacturing environments. While deep learning model

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

arXiv:2606.07591v1 Announce Type: cross Abstract: AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We presen

Semi-supervised Source Detection in Astronomical Images: New Benchmark and Strong Baseline

arXiv:2606.09219v1 Announce Type: new Abstract: Source detection in modern observational astronomy is a cornerstone for localizing and identifying stellar sources accurately. It is crucial for studies

TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs

arXiv:2606.09578v1 Announce Type: new Abstract: Large Language Models (LLMs) and Vision-Language Models (VLMs) are increasingly evaluated on table reasoning tasks, but the role of table representation

When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark

arXiv:2606.09705v1 Announce Type: new Abstract: Scientific generative modeling often requires size transfer, where models trained on small systems are evaluated on larger ones. While translation-invar

XCR-Bench: Benchmarking Cross-Cultural Reasoning in LLMs via Culture-Specific Items and Hall's Triad

arXiv:2601.14063v2 Announce Type: replace-cross Abstract: Cross-cultural competence in large language models (LLMs) requires understanding and adapting Culture-Specific Items (CSIs) across varying cul

Benchmarking Language Modeling for Lossless Compression of Full-Fidelity Audio

arXiv:2603.08683v2 Announce Type: replace-cross Abstract: Autoregressive 'language' models (LMs) trained on raw waveforms can be repurposed for lossless audio compression, but prior work is limited to

GlucoFM-Bench: Benchmarking Time-Series Foundation Models for Blood Glucose Forecasting

arXiv:2606.06881v1 Announce Type: new Abstract: Blood glucose forecasting models are foundational for modern diabetes management systems, as reliable short-term predictions can enable proactive interv

OpenHalDet: A Unified Benchmark for Hallucination Detection across Diverse Generation Scenarios

arXiv:2606.06959v1 Announce Type: cross Abstract: Hallucination detection is essential for the reliable deployment of large language models (LLMs). However, existing evaluations face two core challeng

SW-A^2-Bench: Benchmarking Autonomous Software Agent Generation for Agentic Web

arXiv:2604.04226v2 Announce Type: replace-cross Abstract: The Agentic Web is emerging as a paradigm in which autonomous software agents interact with online resources and with each other to accomplish

Which Anatomy Matters Under Limited Labels? A Data-Efficient Anatomy-Aware Benchmark for Cardiac Pathology Prediction

arXiv:2606.06509v1 Announce Type: cross Abstract: Numerous medical imaging problems must be solved under limited labels and constrained compute, yet it remains unclear whether performance gains are dr

Research

Benchmarks in Leipzig

arXiv:2606.05818v1 Announce Type: cross Abstract: Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of

researcharxiv-cs-ai

Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs

arXiv:2606.05966v1 Announce Type: cross Abstract: Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) sti

DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions

arXiv:2606.06322v1 Announce Type: new Abstract: GUI agents - vision-based models that control desktops, web browsers, and mobile devices through graphical user interfaces - promise to automate a wide

SentinelBench: A Benchmark for Long-Running Monitoring Agents

arXiv:2606.05342v1 Announce Type: new Abstract: AI agents are increasingly asked to carry out work that spans minutes, hours, or longer. Yet the default model of agent behavior is continuous action: i

CollabBench: Benchmarking and Unleashing Collaborative Ability of LLMs with Diverse Players via Proactive Engagement

model-releasesarxiv-cs-cl

arXiv:2606.05793v1 Announce Type: new Abstract: While LLM-based agents excel at individual tasks, effective collaboration with realistic human partners remains challenging. Most of the existing conver

Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understanding

model-releasesarxiv-cs-cl

arXiv:2505.05026v5 Announce Type: replace Abstract: User interface (UI) design goes beyond visuals to shape user experience (UX), underscoring the shift toward UI/UX as a unified concept. While recent

Hardware

Gotta Grow Fast: Design and Benchmarking of a Tip Mount for High-Speed Vine Robots

arXiv:2606.06040v1 Announce Type: new Abstract: Soft, growing vine robots extend through tip eversion, a mechanism that enables navigation through cluttered environments. However, integrating cameras

hardwarearxiv-cs-ro

Is This Edit Correct? A Multi-Dimensional Benchmark for Reasoning-Aware Image Editing

arXiv:2606.05172v1 Announce Type: cross Abstract: Diffusion-based image editing has achieved strong visual fidelity under natural language instructions, yet most existing systems still operate at the

Agents

OdysseyArena: Benchmarking Large Language Models For Long-Horizon, Active and Inductive Interactions

arXiv:2602.05843v2 Announce Type: replace Abstract: The rapid advancement of Large Language Models (LLMs) has catalyzed the development of autonomous agents capable of navigating complex environments.

agentsarxiv-cs-cl

V2V-Bench: A Comprehensive Benchmark for Video-to-Video Generation Evaluation

arXiv:2606.05665v1 Announce Type: new Abstract: Video-to-video (V2V) generation is difficult to evaluate because outputs must both follow editing instructions and preserve frame-level correspondence w

Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms

model-releasesarxiv-cs-cl

arXiv:2606.04701v1 Announce Type: cross Abstract: GUI agents today assume a static screen, where the world is frozen between two actions. However, real interfaces such as short-video applications viol

DLO-Lab: Benchmarking Deformable Linear Object Manipulations with Differentiable Physics

model-releasesarxiv-cs-ro

arXiv:2606.04206v1 Announce Type: new Abstract: We address the challenge of enabling robots to manipulate deformable linear objects (DLOs), such as ropes, cables, and rubber bands. Prior work has prim

GroupToM-Bench: Benchmarking Group Theory of Mind and Nonlinear Social Emergence in MLLMs

arXiv:2606.04184v1 Announce Type: new Abstract: True general intelligence requires not only a model of the physical world but also a social world model: the capacity to infer how individual mental sta

New Benchmarking Shows Limited Generalization Power of TCR Antigenic Epitope Prediction Models

arXiv:2606.04994v1 Announce Type: new Abstract: Accurate computational prediction of T cell receptor (TCR) antigen specificity would transform the study of T cell biology and enable scalable immune en

When Seeing Is Not Believing -- A Benchmark for Search-Grounded Video Misinformation Detection

arXiv:2606.04098v1 Announce Type: new Abstract: Video misinformation increasingly operates at the semantic and evidential level: authentic footage may be selectively edited, temporally reordered, spli

Applications

Benchmarking Speech-to-Speech Translation Models

arXiv:2606.03241v1 Announce Type: new Abstract: Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol: studies report non-overlapping metric subsets

applicationsarxiv-cs-cl

GeoDrive-Bench: Benchmarking Region-Specific Multimodal Reasoning in Autonomous Driving

arXiv:2606.02774v1 Announce Type: new Abstract: Vision-language models (VLMs) for autonomous driving have shown promising performance, but their ability to handle region-specific traffic rules remains

GTBench: A Curriculum-Grounded Benchmark for Evaluating LLMs as Mathematical Research Assistants in Graph Theory

arXiv:2606.03144v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as self-study assistants in technical disciplines, yet their reliability as mathematical reasoning as

Plan2Map: A Multimodal Benchmark for Document-Grounded Geospatial Boundary Reconstruction from Planning Records

arXiv:2606.02747v1 Announce Type: cross Abstract: Planning records define restrictions over geographic areas, but their source documents often provide only indirect spatial evidence rather than machin

SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation

arXiv:2606.03788v1 Announce Type: new Abstract: Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directl

TadA-Bench: A Million-Variant Benchmark for Future-Round Discovery Toward Agentic Protein Engineering

arXiv:2606.02624v1 Announce Type: cross Abstract: AI for scientific discovery is entering an agentic era, where protein-engineering systems are expected to prioritize future wet-lab experiments rather

Agents

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601.08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets perform

agentsarxiv-cs-ai

TurtleAI: Benchmarking Multimodal Models for Visual Programming in Turtle Graphics

arXiv:2606.03626v1 Announce Type: cross Abstract: Vision-language models (VLMs) have been explored for visual programming, where they generate code to solve visual tasks. However, most prior work focu

3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code

arXiv:2606.01057v1 Announce Type: cross Abstract: Procedural 3D modeling through code is emerging as a versatile paradigm, offering deterministic, engine-ready, and precisely editable assets that neur

A Systematic Benchmark of Intraoperative Ultrasound-to-MR Synthesis for Brain Tumour Surgery

arXiv:2606.00630v1 Announce Type: new Abstract: Intraoperative ultrasound (ioUS) is a versatile, cost-effective modality in brain tumour surgery, but its interpretation is difficult: acquisition plane

AgentDS Technical Report: Benchmarking the Future of Human-AI Collaboration in Domain-Specific Data Science

arXiv:2603.19005v2 Announce Type: replace-cross Abstract: Data science plays a critical role in transforming complex data into actionable insights across numerous domains. Recent developments in large

Are LLMs Ready for Neural-integrated Mechanistic Modeling? A Benchmark and Agentic Framework

arXiv:2602.18008v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown promise in constructing mechanistic models from data. However, existing evaluations largely focus on s

Benchmark Dataset for Catalysis on 2D MXenes

arXiv:2606.00794v1 Announce Type: cross Abstract: Merging first-principles calculations with machine learning (ML), we aim to accelerate the exploration of catalytic behaviour in novel materials. We f