Search: “benchmark”

Beyond Rigid: Benchmarking Non-Rigid Video Editing

arXiv:2601.18340v2 Announce Type: replace Abstract: As video generation models are increasingly expected to manipulate physical dynamics, there is a growing need to move evaluation beyond appearance f

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Do Text Edits Generalize to Visual Generation? Benchmarking Cross-Modal Knowledge Editing in UMMs

DGX agent

arXiv:2606.00477v1 Announce Type: new Abstract: Unified multimodal models (UMMs) have emerged as a promising paradigm for general-purpose multimodal intelligence. As they are deployed in real-world ap

model-releasesarxiv-cs-cl

2 Jun 2026

DrugClaw and DrugAudit: A Primary-Source-Grounded Agent and Authority-Aware Benchmark for Drug-Information Question Answering

arXiv:2606.01434v1 Announce Type: new Abstract: Drug-information question answering is a high-stakes setting where hallucinated facts can mislead clinical decision-making and the provenance of each ci

Early Prediction of Liver Cirrhosis Up to Two Years in Advance: A Machine Learning Study Benchmarking Against the FIB-4 and APRI Scores

arXiv:2601.00175v2 Announce Type: replace Abstract: Objective: Develop and evaluate machine learning (ML) models for predicting incident liver cirrhosis (LC) one and two years prior to diagnosis using

HAIM: Human-AI Music Datasets for AI Music Production Tracking Benchmark

arXiv:2606.01686v1 Announce Type: cross Abstract: As generative platforms such as Suno and Udio reach human-grade audio quality, the scope of AI's utility has expanded across the entire music producti

'I Strongly Suspect This Website Is a Scam': Benchmarking PII Leakage and Detection without Defense in Autonomous Web Agents

arXiv:2606.00497v1 Announce Type: cross Abstract: Deceptive web content, widely instantiated across the internet and commonly known as extit{social-engineering attacks}, manipulates autonomous web age

IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages

arXiv:2606.01260v1 Announce Type: cross Abstract: Despite being home to more than 1300 ethnic groups and 700 indigenous languages, bias in Large Language Models has not been fully studied in Indonesia

ProductWebGen: Benchmarking Multimodal Product Webpage Generation

arXiv:2606.01022v1 Announce Type: cross Abstract: Crafting a product display webpage from a source product image, along with layout and visual content instructions, holds significant practical value f

RenoBench: A Citation Parsing Benchmark

arXiv:2603.25640v2 Announce Type: replace-cross Abstract: Accurate parsing of citations is necessary for machine-readable scholarly infrastructure. But, despite sustained interest in this problem, exi

SuperMemory-VQA: An Egocentric Visual Question-Answering Benchmark for Long-Horizon Memory

arXiv:2606.00825v1 Announce Type: new Abstract: AI glasses present a compelling platform for AI agents to serve as personalized memory assistants. To be genuinely useful, such systems must move beyond

TrustLDM: Benchmarking Trustworthiness in Language Diffusion Models

arXiv:2606.00023v1 Announce Type: cross Abstract: The rapid development of Language Diffusion Models (LDMs) challenges the dominant position of auto-regressive competitors in language processing. Howe

Truthful AI Advisors: A Pre-Specified Benchmark for Large Language Model Honesty Under Preference Misalignment

arXiv:2606.01456v1 Announce Type: cross Abstract: Large language models are increasingly deployed as advisors whose objective is not aligned with the user's: recommenders optimize for engagement, sale

TukaBench: A Culturally Grounded Jailbreak Benchmark for African Languages

arXiv:2606.01322v1 Announce Type: cross Abstract: Safety evaluation of Large Language Models (LLMs) remains heavily English-centric, leaving Low-Resource Languages (LRLs), particularly African ones, c

v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound

arXiv:2509.25773v3 Announce Type: replace-cross Abstract: AI models capable of comprehending humor hold real-world promise -- for example, enhancing engagement in human-machine interactions. To gauge

What to Format and How: A Benchmark and Workflow Approach for Document Formatting

arXiv:2606.01936v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have opened up new possibilities for automated document formatting. However, real-world formatting often

3DAE: Binaural Quality Assessment for Audio Novel View Synthesis with Spatial Maps and Benchmark

arXiv:2605.30469v1 Announce Type: cross Abstract: 3D audio and novel-view acoustic synthesis models are usually evaluated with global metrics.However, global metrics often hide where and why binaural

Benchmarking Uncertainty and its Disentanglement in multi-label Chest X-Ray Classification

arXiv:2508.04457v2 Announce Type: replace-cross Abstract: Reliable uncertainty quantification is crucial for trustworthy decision-making and the deployment of AI models in medical imaging. While prior

DeepImageSearch: Benchmarking Multimodal Agents for Context-Aware Image Retrieval in Visual Histories

arXiv:2602.10809v2 Announce Type: replace Abstract: Existing multimodal retrieval systems excel at semantic matching but implicitly assume that query-image relevance can be measured in isolation. This

Gait2Hip-60: A Unified Deep Learning Benchmark for Predicting Hip Muscle Forces and Joint Moments from Multi-Cadence Gait Kinematics

arXiv:2605.30374v1 Announce Type: new Abstract: Estimating hip muscle forces and joint moments during gait typically relies on musculoskeletal simulation, which is informative but time-consuming and d

KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

arXiv:2603.08721v2 Announce Type: replace-cross Abstract: New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels, a time-consu

MedFact: Benchmarking the Fact-Checking Capabilities of Large Language Models on Chinese Medical Texts

arXiv:2509.12440v3 Announce Type: replace-cross Abstract: Deploying Large Language Models (LLMs) in medical applications requires fact-checking capabilities to ensure patient safety and regulatory com

PInVerify: An Offline Embodied Benchmark for Active Instance Verification

arXiv:2605.30639v1 Announce Type: cross Abstract: Embodied agents have made strong progress in navigating to target objects, but reaching the goal vicinity does not guarantee that the agent has found

Translation Analytics for Freelancers II: Benchmarking Local LLMs for Confidential Translation Workflows

arXiv:2605.31452v1 Announce Type: new Abstract: Building on our previous work, this paper develops practical, low-barrier methods for freelance translators and smaller language service providers to ev

Benchmarking Large Vision-Language Models on CFMME: A Comprehensive Chinese Financial Multimodal Evaluation Dataset

arXiv:2605.29462v1 Announce Type: cross Abstract: The emergence of Large Vision-Language Models (LVLMs) has substantially expanded model capabilities beyond text-only understanding, enabling unified i

Benchmarking LLM-Assisted Blue Teaming via Standardized Threat Hunting

arXiv:2509.23571v3 Announce Type: replace-cross Abstract: As cyber threats continue to grow in scale and sophistication, blue team defenders increasingly require advanced tools to proactively detect a

Benchmarking Positional Encoding Strategies for Transformer-Based EEG Foundation Models

arXiv:2605.29754v1 Announce Type: new Abstract: Electroencephalography (EEG) is a widely used non-invasive technique for measuring brain activity in brain-computer interface (BCI) applications. Superv

Benchmarking Single-Factor Physical Video-to-Audio Generation

arXiv:2605.30339v1 Announce Type: new Abstract: Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical process

BenchTrace: A Benchmark for Testing Reflection Ability and Controlled Evolution in LLM Agents

arXiv:2605.29225v1 Announce Type: new Abstract: Self-evolving agents improve over time by reflecting on past failures, but existing evaluation is limited in two ways: it measures only task scores, lea

Building and Road Recognition in Dense Urban Informal Settlements: A Dataset and Benchmark

arXiv:2605.29856v1 Announce Type: new Abstract: As a widespread form of informal settlements, urban villages present significant challenges for sustainable urban development and governance. Precise ma

CLUBench: A Clustering Benchmark

arXiv:2605.29933v1 Announce Type: new Abstract: Clustering is a fundamental problem in data science with a long-standing research history, yielding numerous insightful algorithms. Despite this progres

Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

arXiv:2605.29054v1 Announce Type: cross Abstract: Coding agents increasingly act as codebase-scale collaborators that can assist with codebase conversion, but this progress has exposed a critical weak

Do Physics Foundation Models Learn Generalizable Physics? A Bias-Aware Benchmark Across Physical Regimes and Distribution Shifts

arXiv:2605.29283v1 Announce Type: cross Abstract: Recent physics foundation models claim general spatiotemporal forecasting ability, yet their evaluations often collapse performance into a single aver

GEO-Bench: Benchmarking Ranking Manipulation in Generative Engine Optimization

arXiv:2605.29107v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly rank products, documents, and recommendations for user queries, which makes manipulating these rankings a gr

LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation

arXiv:2510.26412v3 Announce Type: replace-cross Abstract: Recent advances in text-to-video generation have achieved impressive performance on short clips, yet evaluating long-form generation under com

LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis

arXiv:2605.28876v1 Announce Type: cross Abstract: CI failure logs are large (median 5k lines, max 200k in this corpus) and noisy. Coding agents that try to debug them depend on an upstream tool to red

Notation Matters: A Benchmark Study of Token-Optimized Formats in Agentic AI Systems

arXiv:2605.29676v1 Announce Type: new Abstract: Large language models in Agentic AI systems consume tool schemas and execution results and emit tool invocations as structured data. The default languag

PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology

model-releasesarxiv-cs-ro

arXiv:2605.29710v1 Announce Type: new Abstract: Real-world evaluation of vision-language-action (VLA) policies still rests on binary success rate at a fixed timeout with N le 25 rollouts per condition

PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing

arXiv:2605.29815v1 Announce Type: new Abstract: The growing number of submitted papers has motivated the exploration of Large Language Models (LLMs) as a means to support and augment the peer review p

Safety

Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

arXiv:2605.29447v1 Announce Type: cross Abstract: While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge th

safetyarxiv-cs-cl

Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectories

arXiv:2605.29893v1 Announce Type: new Abstract: LLM-based agents have demonstrated strong capabilities in solving complex tasks through multi-step reasoning and tool use. However, existing evaluation

SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers

arXiv:2605.29059v1 Announce Type: cross Abstract: Smart contract decompilation aims to recover high-level source code from bytecode, but evaluating decompilers remains difficult because existing studi

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents

arXiv:2605.28108v1 Announce Type: new Abstract: A long-lived LLM agent, such as OpenClaw, earns its value by acting on a user's preferences and constraints across sessions, not just the current reques

Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

arXiv:2604.00913v2 Announce Type: replace-cross Abstract: 2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect error

Benchmarking Ultrasound Foundation Models for Fetal Plane Classification

arXiv:2605.27796v1 Announce Type: cross Abstract: Ultrasound is widely used in obstetric care due to its safety, accessibility, and real-time imaging. However, interpretation remains operator-dependen

ClinConsensus: A Physician-Calibrated Benchmark for Evaluating Clinical Rubric Coverage in Chinese Medical LLMs

arXiv:2603.02097v5 Announce Type: replace Abstract: Open-ended medical LLM evaluation remains weakly grounded in physician-calibrated coverage of clinically relevant response criteria, especially in l

DisasterBench: Benchmarking LLM Planning under Typed Tool Interface Constraints

arXiv:2605.27957v1 Announce Type: new Abstract: Disasters cause severe societal impacts, demanding rapid coordination of heterogeneous AI tools, from satellite analysis to flood prediction and damage

Applications

FinBoardBench: Benchmarking Dynamic Wealth Management and Strategic Financial Reasoning of LLMs via Board Game Simulations

arXiv:2605.27896v1 Announce Type: new Abstract: Recently, large language models (LLMs) have achieved superior performance in static financial reasoning and simple dynamic trading tasks. However, exist

applicationsarxiv-cs-cl

High Performance, Low Reliability: Uncertainty Benchmarking for Tabular Foundation Models

arXiv:2605.28554v1 Announce Type: new Abstract: Recent Tabular Foundation Models (TFMs) have demonstrated state-of-the-art predictive performance, often surpassing Gradient-Boosted Decision Trees (GBD