Search: “benchmark”

HighTide: An Agent-Curated Open-Source VLSI Benchmark Suite

arXiv:2606.04126v1 Announce Type: cross Abstract: We introduce HighTide, an evolving AI-assisted benchmark suite. Specifically, the contributions are: (i) a diverse open-source suite spanning multiple

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Impostor: An Agent-Curated Benchmark for Realistic AIGC Manipulation Localization

DGX agent

arXiv:2606.04545v1 Announce Type: new Abstract: Recent advances in generative image editing have improved the realism and controllability of localized image manipulation, raising new challenges for im

model-releasesarxiv-cs-cv

4 Jun 2026

It's TIME: Towards the Next Generation of Time Series Forecasting Benchmarks

arXiv:2602.12147v4 Announce Type: replace Abstract: Time series foundation models (TSFMs) are revolutionizing the forecasting landscape from specific dataset modeling to generalizable task evaluation.

LCSHBench: A Multilingual, Consensus-Grounded Benchmark for Library of Congress Subject Heading Assignment

arXiv:2606.04382v1 Announce Type: cross Abstract: Automated subject cataloging assigns controlledvocabulary headings to bibliographic records, but LCSH has no standard public benchmark. We introduce L

RIDE: An Open Dataset and Benchmark for Train Delay Prediction

arXiv:2606.05070v1 Announce Type: new Abstract: Train delay prediction is an important problem for both passengers and railway operators, yet progress in the field remains difficult to assess due to t

VAMPS: Visual-Assisted Mathematical Problem Solving Benchmark

arXiv:2606.04244v1 Announce Type: new Abstract: Multimodal large language models are increasingly capable of complex reasoning, yet their performance often degrades when they must externalize a proble

Benchmarking Visual State Tracking in Multimodal Video Understanding

arXiv:2606.03920v1 Announce Type: new Abstract: Understanding a video requires more than recognizing isolated moments, as humans continuously track entities, states, and events over time. This capacit

BigFinanceBench: A Workflow-Grounded Benchmark for Financial-Research Agents

arXiv:2606.03829v1 Announce Type: new Abstract: Financial-research answers are decision-relevant only when another analyst can audit how they were produced: which source was chosen, which period and a

ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

arXiv:2606.03157v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

arXiv:2606.03410v1 Announce Type: new Abstract: Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through d

Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning

arXiv:2606.03918v1 Announce Type: new Abstract: AI agents can increasingly handle the mechanical tasks of financial analysis: retrieving documents, calculating formulas, updating spreadsheets. The har

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

arXiv:2606.02642v1 Announce Type: cross Abstract: Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing be

VistaHop: Benchmarking Multi-hop Visual Reasoning for Visual DeepSearch

arXiv:2606.03273v1 Announce Type: cross Abstract: Visual DeepSearch requires multimodal large reasoning model (MLRM) agents to answer complex visual queries by repeatedly inspecting image regions, gro

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

arXiv:2606.01992v1 Announce Type: cross Abstract: Industrial anomaly detection has historically been a unimodal task. Recent multimodal vision-language models have produced systems that admit textual

Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation

arXiv:2512.16310v3 Announce Type: replace-cross Abstract: LLM-based agents increasingly use multiple external tools to complete complex tasks. We study Tools Orchestration Privacy Risk (TOP-R): an age

Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

arXiv:2606.00925v1 Announce Type: cross Abstract: Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supp

CityTrajBench: A Unified Benchmark for City-Scale Vehicle Trajectory Generation

arXiv:2606.02287v1 Announce Type: cross Abstract: Urban trajectory generation is a fundamental task for transportation simulation, urban planning, and mobility analytics. However, systematic compariso

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

arXiv:2606.01393v1 Announce Type: cross Abstract: Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Opt

Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking

arXiv:2602.03912v4 Announce Type: replace Abstract: This paper investigates the performance of Echo State Networks (ESNs) for univariate forecasting of monthly and quarterly time series from the M4 Fo

HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers

arXiv:2606.01132v1 Announce Type: new Abstract: Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchma

LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

arXiv:2512.07436v3 Announce Type: replace Abstract: Recent advances in large reasoning models LRMs have enabled agentic search systems to perform complex multi-step reasoning across multiple sources.

MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

arXiv:2606.02470v1 Announce Type: new Abstract: The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and too

MGRegBench: A Novel Benchmark Dataset with Anatomical Landmarks for Mammography Image Registration

arXiv:2512.17605v2 Announce Type: replace-cross Abstract: Robust mammography registration is essential for clinically relevant applications like tracking disease progression in breast tissue. However,

Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

arXiv:2606.00616v1 Announce Type: cross Abstract: Recent Vision-Language Models (VLMs) struggle with grounded reasoning, temporal consistency, and context aware planning in videos. We introduce pause-

Agents

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

arXiv:2602.12984v2 Announce Type: replace Abstract: Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely ov

agentsarxiv-cs-cl

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

arXiv:2606.01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural la

VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio

arXiv:2512.10120v2 Announce Type: replace-cross Abstract: General-purpose audio representations aim to map acoustically variable instances of the same event to nearby points, resolving content identit

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

arXiv:2602.03554v2 Announce Type: replace-cross Abstract: Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evalu

FAM-Bench: A Multimodal Benchmark for Condition-Aware Food-as-Medicine Reasoning

arXiv:2605.31410v1 Announce Type: new Abstract: Food-as-Medicine requires models to reason beyond what a dish is or what nutrition it contains: they must decide whether a concrete food choice is appro

GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing

model-releasesarxiv-cs-cl

arXiv:2509.14221v3 Announce Type: replace-cross Abstract: Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integr

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

arXiv:2602.02220v2 Announce Type: replace Abstract: Language-conditioned goal navigation (LGN) requires agents to locate user-specified targets without step-by-step guidance. However, existing benchma

Applications

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

arXiv:2605.30393v1 Announce Type: cross Abstract: Public numeric benchmarks appear in pretraining, so an evaluation that conditions on a date may be measuring memorized recall rather than out-of-sampl

applicationsarxiv-cs-ai

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

arXiv:2605.31572v1 Announce Type: new Abstract: Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations

Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark

arXiv:2605.29400v1 Announce Type: new Abstract: We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Acti

AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials

arXiv:2510.04704v4 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown promising potential in scientific research, enabling tasks ranging from knowledge retrieval to propert

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

arXiv:2605.30188v1 Announce Type: cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibr

DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

arXiv:2605.29339v1 Announce Type: new Abstract: With the rapid advancement of multimodal large language models (MLLMs), models have demonstrated increasingly powerful multimodal capabilities. However,

DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark

arXiv:2605.30027v1 Announce Type: new Abstract: Multimodal documents contain diverse elements, such as tables, figures, and layouts, which can complicate retrieval tasks. While current approaches typi

K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance

arXiv:2605.29523v1 Announce Type: new Abstract: Large Language Models (LLMs) have advanced financial automation through Retrieval-Augmented Generation (RAG), yet hallucinations remain a critical barri

LUMINA: A Multi-Vendor Mammography Benchmark with Energy Harmonization Protocol

arXiv:2603.14644v3 Announce Type: replace-cross Abstract: Publicly available full-field digital mammography (FFDM) datasets remain limited in size, clinical annotations, and vendor diversity, hinderin

MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing

arXiv:2605.22100v2 Announce Type: replace Abstract: Document parsing converts visually rich documents into machine-readable structured representations, forming a crucial foundation for information sys

ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation

arXiv:2605.29579v1 Announce Type: new Abstract: While multimodal large language models (MLLMs) have achieved rapid progress in vision-language understanding, they remain prone to multimodal hallucinat

Toward Ethical Facial Age Estimation: A Generalized Zero-Shot Benchmark Without Training on Children's Data

arXiv:2605.29230v1 Announce Type: cross Abstract: Age estimation from facial images typically relies on training data that includes images of minors, a practice that raises serious ethical, legal, and

What drives performance in molecular MPNNs? An operator-level factorial benchmark

arXiv:2605.30195v1 Announce Type: cross Abstract: Message-passing neural networks (MPNNs) are widely used for molecular property prediction, but their deployment as monolithic architectures makes it d

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

arXiv:2605.28508v1 Announce Type: new Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape us

Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems

arXiv:2605.27492v1 Announce Type: cross Abstract: LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

arXiv:2605.28183v1 Announce Type: cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER d

EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

arXiv:2605.27820v1 Announce Type: new Abstract: As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop