Search: “benchmark”

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

arXiv:2607.04033v1 Announce Type: cross Abstract: Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, an

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

DGX agent

arXiv:2607.02897v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy conc

model-releasesarxiv-cs-ai

7 Jul 2026

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

arXiv:2510.17801v2 Announce Type: replace-cross Abstract: Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems

Towards Open-World Referring Expression Comprehension: A Benchmark with Training-free Multi-task Consistency Checker

arXiv:2605.25706v2 Announce Type: replace Abstract: Referring expression comprehension (REC) aims to localize a target object within an image based on a given expression. Although recent advances in v

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking

arXiv:2410.15135v5 Announce Type: replace Abstract: With the surge of online misinformation, Large Language Models (LLMs) and Reasoning Large Language Models (RLMs) serving as Automatic Fact-Checking

URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

arXiv:2607.04688v1 Announce Type: cross Abstract: Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Re

XPlainVerse: A Million-Scale Benchmark for Explainable Deepfake Detection

arXiv:2607.03562v1 Announce Type: new Abstract: As deepfake detection models increasingly produce natural language explanations, their reasoning often remains weakly grounded in visual artifacts, limi

Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing

arXiv:2607.01728v1 Announce Type: cross Abstract: Visual regression testing (VRT) is a standard quality assurance step in modern software release pipelines. On every change, it re-renders user interfa

3 Jul 2026

Human Capital, Not Model Benchmarks, Predicts Hybrid Intelligence in Forecasting

arXiv:2607.02467v1 Announce Type: cross Abstract: Whether pairing people with AI helps or hurts is usually reported as a single average effect. Using a real-money prediction market (Polymarket) as an

3 Jul 2026

AV-SyncBench: Decoupled Benchmarking of Temporal and Semantic Audio-Visual Synchronization

arXiv:2607.00726v1 Announce Type: new Abstract: Audio-visual feature extraction is a fundamental component of multimodal understanding and generation tasks. However, existing evaluation protocols for

EgoSafetyBench: A Diagnostic Egocentric Video Benchmark for Evaluating Embodied VLMs as Runtime Safety Guards

arXiv:2607.00218v1 Announce Type: cross Abstract: Vision-language models (VLMs) are now proposed as runtime safety guards for embodied agents in homes and factories. A deployable guard must catch genu

EmbodimentSemantic: A Spatial Scene-Graph Dataset and Benchmark for Vision-Language Models on Embodied Manipulation Trajectories

model-releasesarxiv-cs-ro

arXiv:2607.00020v1 Announce Type: new Abstract: Spatial grounding remains a key limitation of vision-language-action (VLA) systems for robotic manipulation. While current models can recognize objects

LongVQUBench: Benchmarking Long-Term Video Quality Understanding of Vision-Language Models

arXiv:2607.01086v1 Announce Type: cross Abstract: The evaluation of long-term video quality understanding remains an open challenge for large vision-language models (LVLMs). Existing video quality ben

MemSyco-Bench: Benchmarking Sycophancy in Agent Memory

arXiv:2607.01071v1 Announce Type: cross Abstract: Memory has emerged as a cornerstone of modern LLM-based agents, supporting their evolution from single-turn assistants to long-term collaborators. How

MoHallBench: A Benchmark for Motion Hallucination in Video Large Language Models

arXiv:2607.01117v1 Announce Type: new Abstract: Video Large Language Models (VideoLLMs) have shown strong progress in video understanding, yet they still suffer from hallucinations that are inconsiste

MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

arXiv:2607.00724v1 Announce Type: new Abstract: Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that lang

PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents

arXiv:2607.00436v1 Announce Type: new Abstract: Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more

Arena-T2I Hard: Benchmarking and Improving Faithfulness with Dependency-Aware Checklist

arXiv:2606.31711v1 Announce Type: new Abstract: Faithfulness -- how precisely a generated image aligns with its prompt -- is increasingly central to the real-world utility of text-to-image (T2I) model

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

arXiv:2606.22723v2 Announce Type: replace Abstract: Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended,

Bridging Scientific Heritage: An Arabic--Russian Parallel Corpus and LLM Benchmark for Sustainable Knowledge Transfer

arXiv:2606.30943v1 Announce Type: new Abstract: Russian and Arabic are among the major languages of scientific communication. Language barriers impede the exchange of research results between these co

ClawArena-Team: Benchmarking Subagent Orchestration and Dynamic Workflows in Language-Model Agents

arXiv:2606.31174v1 Announce Type: new Abstract: Production large language-model (LLM) agents are increasingly deployed not as lone problem-solvers but as managers: a main model creates specialized sub

CooperScene: Multi-Modal Cooperative Autonomy Benchmark with C-V2X Communication Characterization

arXiv:2606.31219v1 Announce Type: new Abstract: Cellular vehicle-to-everything (C-V2X) enables cooperative perception, prediction, and planning beyond the field of view of individual agents. However,

M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding

arXiv:2601.08758v4 Announce Type: replace-cross Abstract: Chain-of-Thought (CoT) reasoning has proven effective in enhancing large language models by encouraging step-by-step intermediate reasoning, a

OopsieVerse: A Safety Benchmark with Damage-Aware Simulation for Robot Manipulation

model-releasesarxiv-cs-ro

arXiv:2606.31993v1 Announce Type: new Abstract: While robotic manipulation capabilities have advanced rapidly, physical safety remains a major barrier to deploying household robots: task success is in

PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks

arXiv:2606.31154v1 Announce Type: cross Abstract: Creating and editing slides is a rich, multimodal activity that is ubiquitous in professional and educational settings, making it an ideal testbed for

Qualified Educational Capacity Planning under Heterogeneous Student Support Needs: A Synthetic Benchmark and Decision-Support Framework

arXiv:2606.30650v1 Announce Type: cross Abstract: Educational support services often face a qualified-capacity problem: staff time is scarce, qualifications decay, new support needs can appear before

AerialMetric: Benchmarking and Adapting UAV Monocular Metric Depth Estimation in the Real World

arXiv:2606.29716v1 Announce Type: new Abstract: This paper addresses the problem of monocular metric depth estimation in aerial UAV imagery. Although recent data-driven methods have achieved remarkabl

Benchmark AUC Is Not Deployable Reliability: A Cross-Dataset Audit of Off-the-Shelf Features for Surveillance Video Anomaly Detection

arXiv:2606.29506v1 Announce Type: new Abstract: Automated 'suspicious behavior' flagging is a headline promise of AI surveillance, and the field reports high frame-level ROC-AUC on standard video anom

Agents

Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio

arXiv:2606.17041v4 Announce Type: replace Abstract: Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregati

agentsarxiv-cs-cl

Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models

arXiv:2606.28406v1 Announce Type: cross Abstract: Text-to-image and multimodal generative models are increasingly used to produce scientific figures such as mechanism diagrams, experimental-design sch

CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models

arXiv:2501.14940v4 Announce Type: replace-cross Abstract: Aligning large language models (LLMs) with human values is essential for their safe deployment and widespread adoption. Current LLM safety ben

How Far Can You Get Without a GPU? A Systematic Benchmark of Lightweight Hallucination Detection Across Question Answering, Dialogue, and Summarisation

arXiv:2606.29809v1 Announce Type: cross Abstract: Hallucination detection has become a pressing requirement for trustworthy AI deployment at scale. The most accurate detection methods depend on GPU-in

KrishokChat: A Citation-Grounded Dataset and Benchmark for Bengali Agricultural Advisory

model-releasesarxiv-cs-lg

arXiv:2606.29243v1 Announce Type: new Abstract: We present KrishokChat, the first citation-grounded Bengali agricultural instruction-tuning dataset for crop advisory in low-resource settings. We estab

Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

arXiv:2606.28925v1 Announce Type: cross Abstract: Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while o

Research

Preference-ASR: A Preference-Aware Test Set for Benchmarking ASR in the Era of Speech LLMs

arXiv:2606.29534v1 Announce Type: new Abstract: Popular ASR test sets adopt inconsistent conventions for numbers, disfluencies, entities, and casing, while standard normalizers erase the format distin

researcharxiv-cs-cl

REPAIR-Bench: A Benchmark for Robot Error Perception And Interaction Recovery

model-releasesarxiv-cs-ro

arXiv:2606.29937v1 Announce Type: new Abstract: Understanding how users perceive and respond to robot failures is essential for building robust and trustworthy robot systems. Prior work, however, (i)

RoAd-RL: A Unified Library and Benchmark for Robust Adversarial Reinforcement Learning

arXiv:2606.29867v1 Announce Type: cross Abstract: Deep Reinforcement Learning (DRL) has achieved significant success in robotics and autonomous systems, yet remains vulnerable to adversarial perturbat

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

arXiv:2606.29520v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software

Research

Semantic Correspondence: Unified Benchmarking and a Strong Baseline

arXiv:2505.18060v4 Announce Type: replace Abstract: Establishing semantic correspondence is a challenging task in computer vision, aiming to match keypoints with the same semantic information across d

researcharxiv-cs-cv

SHOVIR: A Benchmark for Evaluating Vision Shortcut Learning in Radiology Report Generation

arXiv:2606.30201v1 Announce Type: cross Abstract: Current evaluation protocols for Vision-Language Models (VLMs) in Radiology Report Generation (RRG) rely on report-level metrics that measure lexical

SWITCH: Benchmarking Modeling and Handling of Tangible Interfaces in Long-horizon Embodied Scenarios

arXiv:2511.17649v4 Announce Type: replace-cross Abstract: Tangible control interfaces (TCIs), such as appliance panels, remotes, elevators, and embedded GUIs, are a fundamental component of everyday h

The Human Creativity Benchmark

arXiv:2606.30561v1 Announce Type: new Abstract: Modern AI evaluation frameworks treat evaluator disagreement as noise to be resolved. In creative domains, professional disagreement reflects genuine di

VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On

arXiv:2603.11734v2 Announce Type: replace Abstract: As virtual try-on (VTON) continues to advance, a growing number of real-world scenarios have emerged, pushing beyond the ability of the existing spe

When Medical Safety Alignment Fails: A Benchmark for Evaluating LLMs on High-Risk Medical Queries

arXiv:2606.28332v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for medical and health-related questions, yet their safety in high-risk medical scenarios remains p

CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

arXiv:2602.20094v2 Announce Type: replace Abstract: As large language models (LLMs) witness increasing deployment in complex, high-stakes decision-making scenarios, it becomes imperative to ground the

Mosaic: A Benchmark Suite for Differentiable Physics Solvers

model-releasesarxiv-cs-lg

arXiv:2606.27895v1 Announce Type: cross Abstract: Differentiable partial differential equation (PDE) solvers underpin solver-in-the-loop ML training, gradient-based optimal control, and inverse proble

NormAct: A Benchmark for Hidden Social Norm Compliance in Embodied Planning

arXiv:2606.27826v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly deployed as embodied planners in egocentric environments, where task success requires not only

SpatialUAV: Benchmarking Spatial Intelligence for Low-Altitude UAV Perception, Collaboration, and Motion

arXiv:2606.27876v1 Announce Type: cross Abstract: Spatial intelligence is essential for low-altitude unmanned aerial vehicle (UAV) perception, collaboration, and navigation. However, existing UAV benc