Search: “benchmark”

EgoCoT-Bench: Benchmarking Grounded and Verifiable Operation-Centric Chain of Thought Reasoning for MLLMs

arXiv:2605.19559v1 Announce Type: cross Abstract: The rapid development of Multimodal Large Language Models (MLLMs) has led to growing interest in egocentric video understanding, specifically the abil

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

DGX agent

arXiv:2511.14159v2 Announce Type: replace Abstract: Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-wo

model-releasesarxiv-cs-cv

20 May 2026

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

arXiv:2605.19382v1 Announce Type: new Abstract: Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluat

RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents

model-releasesarxiv-cs-ro

arXiv:2605.19328v1 Announce Type: cross Abstract: Recent advances in Vision-Language Models (VLMs) facilitate a new class of embodied AI systems, where these models are integrated into physical platfo

Synthetic Data Generation for Brain-Computer Interfaces: Overview, Benchmarking, and Future Directions

arXiv:2603.12296v2 Announce Type: replace-cross Abstract: Deep learning has achieved transformative performance across diverse domains, largely driven by large-scale and high-quality training data. In

The World Won't Stay Still: Programmable Evolution for Agent Benchmarks

arXiv:2603.05910v2 Announce Type: replace Abstract: LLM-powered tool-calling agents fulfill user requests by interacting with environments, querying data, and invoking tools in a multi-turn process. Y

Trajectory Planning and Control near the Limits: an Open Experimental Benchmark on the RoboRacer Platform

model-releasesarxiv-cs-ro

arXiv:2605.19881v1 Announce Type: new Abstract: We present a modular framework to benchmark new and existing methods for trajectory planning and control in high-acceleration maneuvers that push autono

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

arXiv:2605.16819v1 Announce Type: cross Abstract: GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial

Benchmarking Mythos-Linked Bug Rediscovery

arXiv:2605.17416v1 Announce Type: cross Abstract: Anthropic's April 2026 Mythos materials combine benchmark claims with concrete bug-finding stories across OpenBSD, FreeBSD, Linux, FFmpeg, and browser

BoLT: A Benchmark to Democratize Black-box Optimization Research for Expensive LLM Tasks

arXiv:2605.17000v1 Announce Type: cross Abstract: Optimization of LLM training and inference configurations, such as hyperparameters, data mixtures, and prompts, is critical to performance, but it is

Can LLMs Think Like Consumers? Benchmarking Crowd-Level Reaction Reconstruction with ConsumerSimBench

arXiv:2605.17079v1 Announce Type: cross Abstract: LLMs are increasingly used as ``digital consumers'' to simulate public opinion, pre-test marketing decisions, and anticipate audience response. Howeve

DeTrack: A Benchmark and Altitude-Aware Dual World Model for Drone-embodied Tracking

arXiv:2605.17451v1 Announce Type: new Abstract: Aerial object tracking has broad applications in public safety, emergency rescue, wildlife monitoring, and related fields. However, existing aerial trac

DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

arXiv:2601.13839v2 Announce Type: replace Abstract: Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage asse

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

arXiv:2605.17554v1 Announce Type: new Abstract: Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand. They are being de

EvilGenie: A Reward Hacking Benchmark

model-releasesarxiv-cs-lg

arXiv:2511.21654v2 Announce Type: replace Abstract: We introduce EvilGenie, a benchmark for reward hacking in programming settings. We source problems from LiveCodeBench and create an environment in w

FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

arXiv:2510.08886v3 Announce Type: replace Abstract: Going beyond simple text processing, financial auditing requires detecting semantic, structural, and numerical inconsistencies across large-scale di

FinTagging: Benchmarking LLMs for Extracting and Structuring Financial Information

arXiv:2505.20650v5 Announce Type: replace-cross Abstract: Accurate interpretation of numerical data in financial reports is critical for markets and regulators. Although XBRL (eXtensible Business Repo

LEAF: A Living Benchmark for Event-Augmented Forecasting

arXiv:2605.16358v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, se

PluRule: A Benchmark for Moderating Pluralistic Communities on Social Media

arXiv:2605.17187v1 Announce Type: cross Abstract: Social media are shifting towards pluralism -- community-governed platforms where groups define their own norms. What violates rules in one community

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

arXiv:2605.18693v1 Announce Type: new Abstract: As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether t

StyleText: A Large-Scale Dataset and Benchmark for Stylized Scene Text Inpainting

arXiv:2605.17309v1 Announce Type: cross Abstract: We present StyleText, a large-scale dataset and benchmark for localized scene-text inpainting with style preservation. StyleText contains 28,518 image

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

arXiv:2602.21265v2 Announce Type: replace Abstract: We introduce ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. ToolMAT

UniER: A Unified Benchmark for Item-level and Path-level Exercise Recommendation

arXiv:2605.16750v1 Announce Type: cross Abstract: Personalized exercise recommendation dynamically aligns pedagogical resources with individual knowledge mastery, which is crucial for satisfying stude

WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform

arXiv:2605.17912v1 Announce Type: cross Abstract: World models have emerged as a central paradigm for embodied intelligence, enabling agents to predict action-conditioned future and reason about envir

YOLO-NAS-Bench: A Surrogate Benchmark with Self-Evolving Predictors for YOLO Architecture Search

arXiv:2603.09405v2 Announce Type: replace Abstract: Neural Architecture Search (NAS) for object detection is severely bottlenecked by high evaluation cost, as fully training each candidate YOLO archit

MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

arXiv:2605.15589v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in the mental health domain, yet it remains unclear how well they capture related biomedical knowledg

18 May 2026

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

arXiv:2605.15222v1 Announce Type: cross Abstract: Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-crit

18 May 2026

A Benchmark for Early-stage Parkinson's Disease Detection from Speech

arXiv:2605.14066v1 Announce Type: cross Abstract: Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare

Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows

arXiv:2605.14322v1 Announce Type: new Abstract: Language agents are increasingly deployed in complex professional workflows, with tutoring emerging as a particularly high-stakes capability that remain

Research

Do Composed Image Retrieval Benchmarks Require Multimodal Composition?

arXiv:2605.14787v1 Announce Type: cross Abstract: Composed Image Retrieval (CIR) is a multimodal retrieval task where a query consists of a reference image and a textual modification, and the goal is

researcharxiv-cs-cl

GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

arXiv:2605.14498v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems t

PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media

arXiv:2605.14534v1 Announce Type: cross Abstract: Evaluating object removal in images and videos remains challenging because the task is inherently one-to-many, yet existing metrics frequently disagre

SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding

arXiv:2510.13016v3 Announce Type: replace Abstract: A truly capable AI system must do more than detect objects or recognize activities in isolation. It must form unified, grounded representations of w

VLRS-Bench: A Vision-Language Reasoning Benchmark for Remote Sensing

arXiv:2602.07045v2 Announce Type: replace-cross Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have enabled complex reasoning. However, existing remote sensing (RS) benchmar

A Benchmark for Multi-Party Negotiation Games from Real Negotiation Data

arXiv:2603.14066v2 Announce Type: replace-cross Abstract: Many real-world multi-party negotiations unfold as sequences of binding, action-level commitments rather than a single final outcome, yet this

14 May 2026

BARISTA: A Multi-Task Egocentric Benchmark for Compositional Visual Understanding

arXiv:2605.12074v1 Announce Type: new Abstract: Scene understanding is central to general physical intelligence, and video is a primary modality for capturing both state and temporal dynamics of a sce

Covering Human Action Space for Computer Use: Data Synthesis and Benchmark

arXiv:2605.12501v1 Announce Type: new Abstract: Computer-use agents (CUAs) automate on-screen work, as illustrated by GPT-5.4 and Claude. Yet their reliability on complex, low-frequency interactions i

CTFusion: A CTF-based Benchmark for LLM Agent Evaluation

model-releasesarxiv-cs-lg

arXiv:2605.11504v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have enabled agentic systems for complex, multi-step tasks; cybersecurity is emerging as a prominent app

MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models

arXiv:2501.02955v2 Announce Type: replace Abstract: In recent years, vision language models (VLMs) have made significant advancements in video understanding. However, a crucial capability - fine-grain

Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation

arXiv:2605.11307v1 Announce Type: new Abstract: Image-to-code generation tests whether a vision-language model (VLM) can recover the structure of an image enough to express it as executable code. Exis

Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values

arXiv:2605.10365v1 Announce Type: new Abstract: Autonomous agents have rapidly matured as task executors and seen widespread deployment via harnesses such as OpenClaw. Safety concerns have rightly dra

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling

arXiv:2604.08178v2 Announce Type: replace Abstract: In classical Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs) serve as the fundamental signal provider for model alignment. As

Benchmarking Safety Risks of Knowledge-Intensive Reasoning under Malicious Knowledge Editing

arXiv:2605.10146v1 Announce Type: new Abstract: Large language models (LLMs) increasingly rely on knowledge editing to support knowledge-intensive reasoning, but this flexibility also introduces criti

Beyond the All-in-One Agent: Benchmarking Role-Specialized Multi-Agent Collaboration in Enterprise Workflows

model-releasesarxiv-cs-lg

arXiv:2605.08761v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly expected to operate in enterprise environments, where work is distributed across specialized roles,

DeepTumorVQA: A Hierarchical 3D CT Benchmark for Stage-Wise Evaluation of Medical VLMs and Tool-Augmented Agents

arXiv:2605.09679v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) and AI agents have made significant progress in learning to analyze and reason about clinical images. However, e

DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding

arXiv:2605.08888v1 Announce Type: new Abstract: Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation

GAMBIT: A Three-Mode Benchmark for Adversarial Robustness in Multi-Agent LLM Collectives

arXiv:2605.09027v1 Announce Type: new Abstract: In multi-agent systems (MAS), a single deceptive agent can nullify all gains of an agentic AI collective and evade deployed defenses. However, existing

GraphBench: Next-generation graph learning benchmarking

arXiv:2512.04475v5 Announce Type: replace-cross Abstract: Machine learning on graphs has made substantial progress across domains such as molecular property prediction and chip design. Yet benchmarkin