Search: “benchmark”

Benchmarking Vision-Language-Action Models on SO-101: Failure and Recovery Analysis

arXiv:2606.08881v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have demonstrated strong generalization in robotic manipulation, yet existing evaluations are primarily conducted

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese

DGX agent

arXiv:2606.07853v1 Announce Type: cross Abstract: Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted i

model-releasesarxiv-cs-ai

9 Jun 2026

ContextShift: A Controlled Benchmark for Context Dependence in Object Detection

arXiv:2606.09495v1 Announce Type: new Abstract: Modern object detectors achieve strong performance on standard benchmarks, yet their robustness to contextual variation remains insufficiently understoo

Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops

arXiv:2606.08960v1 Announce Type: cross Abstract: Agent benchmarks score submissions with outcome verifiers that are typically hand-written and brittle, leaving them open to reward hacking. We audit 1

OmniGameArena: A Unified UE5 Benchmark for VLM Game Agents with Improvement Dynamics

arXiv:2606.09826v1 Announce Type: cross Abstract: Vision-language model (VLM) agents are increasingly deployed in interactive game environments. Yet game benchmarks for VLM agents typically report a s

This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great…

model-releaseskarpathy--x

This is a super exciting release - Claude Fable 5 is the same underlying model as Mythos but with added safeguards. The benchmarks are great and it's SOTA on everything by a margin but I'll add that *

MMAE: A Massive Multitask Audio Editing Benchmark

arXiv:2606.07229v1 Announce Type: cross Abstract: We introduce MMAE, a Massive Multitask Audio Editing benchmark, serving as the first comprehensive evaluation testbed designed for general-purpose ins

MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models

arXiv:2606.06696v1 Announce Type: cross Abstract: Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling

REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference

arXiv:2606.07141v1 Announce Type: cross Abstract: Language models trained for clinical disease inference are trained on patient data, which may include sensitive and private information, and data owne

ScenicRules: An Autonomous Driving Benchmark with Multi-Objective Specifications and Abstract Scenarios

arXiv:2602.16073v2 Announce Type: replace-cross Abstract: Developing autonomous driving systems for complex traffic environments requires balancing multiple objectives, such as avoiding collisions, ob

Seeing a number of benchmarks showing Opus is the best model for long-running work. Five tips for running Opus autonomously for hours/days: …

model-releasesboris-cherny--x

Seeing a number of benchmarks showing Opus is the best model for long-running work. Five tips for running Opus autonomously for hours/days: 1. Use auto mode for permissions, so Claude doesn’t ask for

SWE-Explore: Benchmarking How Coding Agents Explore Repositories

arXiv:2606.07297v1 Announce Type: cross Abstract: Repository-level coding benchmarks such as SWE-bench have driven a rapid surge in the capabilities of coding agents. Yet they usually treat coding tas

When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents

arXiv:2606.05806v1 Announce Type: new Abstract: Existing benchmarks evaluate Tool-Integrated Reasoning (TIR) in LLMs on idealized ''happy paths'', largely overlooking real-world tool failures. We intr

6 Jun 2026

Arena AI Agentic User Benchmark Ranking

Arena AI's agentic benchmark ranks AI models on how well they orchestrate tools for real-world agentic tasks, based on signals like tool reliability, task completion, and steerability. The leaderboard

model-releasesr-chatgpt

Benchmarking Open-Source Layout Detection Models for Data Snapshot Extraction from Institutional Documents

arXiv:2606.06242v1 Announce Type: new Abstract: Institutional documents contain substantial amounts of operational and analytical information embedded within figures and tables. Current approaches for

Research

Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering

arXiv:2510.05709v2 Announce Type: replace-cross Abstract: LLM benchmarking metrics often misstate performance and uncertainty as they rely on two assumptions that frequently do not hold in practice: (

researcharxiv-cs-cl

MCBench: A Multicontext Safety Assessment Benchmark for Omni Large Language Models

arXiv:2606.05177v1 Announce Type: new Abstract: Existing multimodal safety benchmarks focus solely on visual inputs and cannot assess Omni Large Language Models (LLMs) that process vision, audio, and

TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework

arXiv:2606.05570v1 Announce Type: new Abstract: Repository-level coding benchmarks face a trade-off between task difficulty and evaluation reliability: tasks that challenge frontier models often invol

UltraVR: A Diagnostic Ultra-Resolution Image-VQA Benchmark for Evidence-Grounded Reasoning

arXiv:2606.05576v1 Announce Type: new Abstract: Vision-language models (VLMs) excel on visual question answering and multimodal reasoning benchmarks. Yet their capability on ultra-resolution images -

100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?

arXiv:2505.19293v2 Announce Type: replace-cross Abstract: Long-context capability is considered one of the most important abilities of LLMs, as a truly long context-capable LLM enables users to effort

AUDDT: A Unified Benchmark Toolkit for Audio and Speech Deepfake Detectors

arXiv:2509.21597v2 Announce Type: replace-cross Abstract: With the prevalence of artificial intelligence (AI)-generated content, such as audio deepfakes, a large body of recent work has focused on dev

Tutorials

Measuring What Matters: Synthetic Benchmarks for Concept Bottleneck Models

arXiv:2606.04326v1 Announce Type: cross Abstract: Concept bottleneck models predict outcomes from high-level concepts detected in inputs. Although concepts provide a simple way to reap benefits from i

tutorialsarxiv-cs-ai

NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models

arXiv:2606.04773v1 Announce Type: cross Abstract: Reliable evaluation of human motion understanding is fundamental to advancing embodied AI, robotics, and animation. However, existing benchmarks suffe

UniCAD: A Unified Benchmark and Universal Model for Multi-Modal Multi-Task CAD

arXiv:2606.05058v1 Announce Type: cross Abstract: Computer-Aided Design (CAD) underpins modern engineering and manufacturing by enabling the creation of precise, editable 3D models. However, CAD resea

A Benchmark for Semi-supervised Multi-modal Crowd Counting

arXiv:2606.03646v1 Announce Type: new Abstract: This paper constructs the first benchmark on semi-supervised multi-modal crowd counting. To lay the foundation for this unexplored task, we first formul

Research

Anomalies in Multivariate Time Series Benchmarks Are Mostly Univariate

arXiv:2606.02670v1 Announce Type: cross Abstract: Many recent multivariate time series anomaly detection (MT-SAD) models incorporate cross-channel modeling, under the implicit assumption that the stru

researcharxiv-cs-ai

COD10K-C: Benchmarking Robustness of Camouflaged Object Detection Under Natural Image Corruptions

arXiv:2606.02603v1 Announce Type: new Abstract: Camouflaged object detection has improved substantially, but most standard benchmarks evaluate models only on clean images. This is not realistic becaus

EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

arXiv:2606.03363v1 Announce Type: new Abstract: Text-to-SQL enables natural language access to databases, and recent LLMs have substantially advanced its capabilities. Existing benchmarks such as Spid

IdiomX A Multilingual Benchmark for Idiom Understanding, Retrieval, and Interpretation

arXiv:2606.02584v1 Announce Type: cross Abstract: Idiomatic expressions remain a persistent challenge for natural language processing because their meanings are often non-compositional, context-depend

scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation

arXiv:2606.03906v1 Announce Type: new Abstract: Simultaneous measurement of multiple omics modalities in single cells enables researchers to gain a more comprehensive understanding of cellular states

Sources: Benchmark raised 2B across two new funds, including a 1.25B fund for late-stage bets, its first growth fund after decades focusing on new startups (Kate Clark/Wall Street Journal)

Kate Clark / Wall Street Journal: Sources: Benchmark raised 2B across two new funds, including a 1.25B fund for late-stage bets, its first growth fund after decades focusing on new startups — After a

model-releasestechmeme

ADRA-Bank: A Modular Benchmark for Academic Deep Research Agents

arXiv:2512.00986v3 Announce Type: replace Abstract: A surge in academic publications calls for automated deep research (DR) systems, but accurately evaluating them is still an open problem. First, exi

Applications

AI benchmarks are breaking. Trace analysis is what comes next.

Models got smart enough to cheat their benchmarks, and outcome-only scores stopped measuring what we thought they measured. The fix, full trace analysis, is the same methodology production AI teams ha

applicationsarize-ai

Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages

arXiv:2606.00154v1 Announce Type: cross Abstract: Recent advancements in multimodal large language models (MLLMs) have achieved remarkable progress in multimodal reasoning and code generation, catalyz

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

arXiv:2606.00871v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing

CARTE: A Benchmark for Mapping Language Model Knowledge Across France

arXiv:2606.01995v1 Announce Type: new Abstract: We introduce CARTE 1 (Culturally Anchored Regional-Territorial Evaluation), a multiplechoice benchmark for evaluating the ability of large language mode

Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialogue

arXiv:2606.01223v1 Announce Type: cross Abstract: Despite substantial progress in long-context modeling, existing benchmarks remain confined to factual memory for explicit recall, failing to measure t

CV-Arena: An Open Benchmark for Instructional Computer Vision Problem Solving with Human-AI Collaborative Preferences

arXiv:2606.00931v1 Announce Type: cross Abstract: Instruction-guided image editing is becoming a general interface for visual work, yet existing benchmarks still focus largely on narrow appearance edi

FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

arXiv:2604.03893v2 Announce Type: replace Abstract: Current multimodal benchmarks for scientific reasoning primarily evaluate local information extraction -- models recognize symbols and values and th

LLM-WikiRace Benchmark: How Far Can LLMs Plan over Real-World Knowledge Graphs?

arXiv:2602.16902v4 Announce Type: replace Abstract: We introduce LLM-Wikirace, a benchmark for evaluating planning, reasoning, and world knowledge in large language models (LLMs). In LLM-Wikirace, mod

MBench: A Comprehensive Benchmark on Memory Capability for Video World Models

arXiv:2606.00793v1 Announce Type: new Abstract: Recent advancements in video-based world models have demonstrated an unprecedented ability to synthesize high-fidelity visual sequences. However, a fund

MedGym:A Unified Continuous-Time Benchmark for Dynamic Medical Treatment Reinforcement Learning

model-releasesarxiv-cs-lg

arXiv:2606.01028v1 Announce Type: new Abstract: Medical treatment recommendation poses several challenges to reinforcement learning (RL): patient physiology evolves in continuous time, measurements an

OmniEEG-Bench: A Standardized Evaluation Benchmark for EEG Foundation Models

model-releasesarxiv-cs-lg

arXiv:2606.00815v1 Announce Type: new Abstract: Electroencephalography (EEG) supports a variety of brain-computer interface (BCI) tasks ranging from brain-state monitoring to human-LLM interactions. E

RoboTrustBench: Benchmarking the Trustworthiness of Video World Models for Robotic Manipulation

arXiv:2606.01600v1 Announce Type: cross Abstract: Video world models are increasingly used in robotic manipulation, yet existing benchmarks mostly evaluate them under valid, feasible, and safe instruc

SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation

arXiv:2606.01481v1 Announce Type: new Abstract: With the rapid advancements in text-to-image diffusion models, generative video models (T2V models) like Sora can now produce short synthetic videos fro

TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks

model-releasesarxiv-cs-lg

arXiv:2606.02384v1 Announce Type: new Abstract: Progress in tabular machine learning has largely focused on increasingly sophisticated model architectures. At the same time, feature engineering remain

Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory

arXiv:2605.31086v1 Announce Type: new Abstract: In existing memory benchmarks for Large Language Models (LLMs), the evaluated dialogue sessions often lack long-term semantic consistency, and the under

1 Jun 2026

EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

arXiv:2605.30637v1 Announce Type: new Abstract: Clinical decision-making (CDM) is central to real-world clinical workflows, where clinicians infer diagnoses, select treatments, or anticipate future he