Search: “benchmark”

What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead

arXiv:2609.10962v1 Announce Type: cross Abstract: Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, pop

12 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

From Evaluation to Enhancement: Benchmarking and Improving Think-with-Video Reasoning for Video Generative Models

DGX agent

arXiv:2609.11242v1 Announce Type: new Abstract: Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--

model-releasesarxiv-cs-cv

11 Sept 2026

Leveraging Avatar Fingerprinting: A Multi-Generator Photorealistic Talking-Head Public Database and Benchmark

arXiv:2603.26934v2 Announce Type: replace Abstract: Recent advances in photorealistic avatar generation have enabled highly realistic talking-head avatars, raising security concerns regarding identity

11 Sept 2026

LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

arXiv:2609.09754v1 Announce Type: new Abstract: As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning e

11 Sept 2026

AgroVisNet: A lightweight Convolutional Network and the BD-PlantDX Expert-Validated Benchmark for Radish, Potato and Pointed Gourd Disease Classification

arXiv:2609.10469v1 Announce Type: new Abstract: Automated plant disease diagnosis is increasingly deployed on farmer-held devices in regions where agronomic expertise is scarce and network connectivit

Benchmarking Hybrid Deep Research Across Database Querying and Web Search

arXiv:2609.09410v1 Announce Type: new Abstract: While autonomous agents have made significant strides in 'deep research' by iteratively navigating the open web to synthesize information, real-world pr

Beyond Coherence: Benchmarking Professional Editing-Technique Execution in Multi-Shot Audio-Video Generation

arXiv:2609.08275v1 Announce Type: new Abstract: Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute ed

Beyond Final Decisions: A Process-Centric Benchmark for Transparent AI-Assisted Peer Review

arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of ge

CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records

arXiv:2609.07022v1 Announce Type: cross Abstract: Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cros

FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?

arXiv:2609.07603v1 Announce Type: new Abstract: Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agen

NormViz: A Benchmark and Framework for Grounding Multimodal Reasoning in Global Cultures

arXiv:2609.06831v1 Announce Type: new Abstract: AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI

SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs

arXiv:2609.06149v1 Announce Type: cross Abstract: Script-based malware remains a prevalent attack technique. These scripts often contain indicators of compromise (IOCs) that provide actionable threat

TTGBench: Benchmarking Topological Evolution and Semantic Drift in Text-attributed Temporal Graphs

arXiv:2609.08226v1 Announce Type: new Abstract: Temporal graph learning models the evolution of dynamic systems, where both structural interactions and semantic states change over time. However, exist

Benchmarking Storage Systems for Machine Learning Workloads Using NIO Bench

model-releasesarxiv-cs-lg

arXiv:2609.05418v1 Announce Type: cross Abstract: Machine learning training workloads place unique demands on storage systems, yet most existing benchmarks focus on computational throughput rather tha

Emo-DVS: A Multimodal Benchmark for Privacy-Aware Emotion Recognition with Event Cameras

arXiv:2609.06928v1 Announce Type: new Abstract: Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventiona

How Long Until Your Robot Ignores You? A Safety Benchmark for LLM Orchestrators in Human-Humanoid Collaboration

model-releasesarxiv-cs-ro

arXiv:2609.07288v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly employed to orchestrate robot behavior through natural-language interfaces, yet no benchmark exists to eva

MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials

model-releasesarxiv-cs-lg

arXiv:2609.08399v1 Announce Type: new Abstract: Universal machine-learning interatomic potentials (u-MLIPs) aim to generalize across diverse configurations. Benchmarks enable reproducible evaluation b

ONE CYLinder: A Benchmark for Graph-Based Surrogate Modeling of Unsteady Bluff-Body Flows

model-releasesarxiv-cs-lg

arXiv:2609.08947v1 Announce Type: cross Abstract: Graph-based surrogate models offer a promising route to accelerate computational fluid dynamics (CFD) simulations on unstructured meshes. However, the

Unexplored flaws in multiple-choice VQA make benchmarking unreliable

arXiv:2511.22341v2 Announce Type: replace Abstract: Previous works identify sensitivity to option order as a key issue in multiple-choice VQA (MC-VQA) evaluation and propose protocols to mitigate this

A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark

arXiv:2609.04641v1 Announce Type: new Abstract: Natural-language-to-SQL systems have ad- vanced rapidly on academic benchmarks, yet production enterprise schemas exhibit graph- like, semi-structured,

7 Sept 2026

Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human ref…

Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human reference scores 82.2%. Here is what the task actually is. The

model-releasesdair-ai--x

7 Sept 2026

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

arXiv:2609.05405v1 Announce Type: new Abstract: Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whethe

7 Sept 2026

Enhancing Financial Question Answering: A Novel Benchmark Dataset of Banks' financial statements

arXiv:2609.03654v1 Announce Type: cross Abstract: The comparative analysis of banks' financial statements poses significant challenges for automated question answering systems due to their complexity,

GPS-Bench: A Governance Policy Benchmark for Automating Policy Analysis

arXiv:2609.03553v1 Announce Type: new Abstract: Policy analysis requires more than predicting whether a proposal will pass: it requires identifying who will be affected, how those actors respond, and

HalluPeer: A Taxonomy-driven Benchmark for Detecting Hallucinations in Scientific Peer Reviews

arXiv:2609.03580v1 Announce Type: new Abstract: The growing scale of academic peer review has motivated the use of Large Language Models (LLMs) as review assistants, yet LLMs can generate fluent but u

I benchmarked 21 Qwen3.8 27B variants on 16GB VRAM

model-releasesr-localllama

After Qwen3.8 27B came out, I decided to benchmark the models that could fit in my GPU (RTX 5080) on my actual code (C code), the results were not completely unexpected but some quants were definitely

LexIssue: Benchmarking Legal Issue Identification in Chinese Civil Litigation

arXiv:2609.02954v1 Announce Type: new Abstract: Identifying the issues disputed between litigating parties is a crucial component of real-world litigation. However, legal issues remain comparatively u

LongCounsel-8: A Benchmark Suite for Longitudinal Depression Tracking from Multi-Session Counseling Dialogues

arXiv:2609.03507v1 Announce Type: cross Abstract: Tracking depression from multi-session counseling dialogues requires estimating both current symptom severity and how it changes across sessions. Yet

MetaStructAtlas: A Grounded 3D Vision-Language Dataset and Benchmark for Functional and Structural Reasoning in Whole-Body PET/CT

arXiv:2609.03690v1 Announce Type: new Abstract: The joint interpretation of metabolic function and anatomical structure is essential for clinical diagnosis in whole-body PET/CT. Although recent advanc

Persistent Identity Preservation in Generative Image Models: A Benchmark and Evaluation System

arXiv:2609.04151v1 Announce Type: new Abstract: Generative image models can now produce high-quality images, follow complex instructions, and support precise edits, but they still struggle to preserve

Hardware

RobustSeiz: An Open-Source Framework for Benchmarking the Robustness of EEG Seizure Detection Models

arXiv:2609.04007v1 Announce Type: new Abstract: Despite strong performance on held-out electroencephalography (EEG) data, seizure detectors may fail under real-world acquisition variability, artifacts

hardwarearxiv-cs-lg

Urban Boundaries, Social Barriers: A Benchmark and Vision-Centric Framework for Mapping Gated Communities and Equity Implications

arXiv:2609.03804v1 Announce Type: new Abstract: Communities are fundamental spatial units that shape urban form and social life. Whether a residential compound is spatially open or enclosed affects mo

VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models

arXiv:2601.19956v2 Announce Type: replace-cross Abstract: As Speech Language Models (SLMs) transition from personal devices to shared, multi-user environments such as smart homes, a new challenge emer

Benchmarking RAW and RGB Restoration in Image Signal Processors

arXiv:2609.02831v1 Announce Type: new Abstract: Modern cameras transform RAW sensor measurements into sRGB images through an image signal processor (ISP). We benchmark two placements for blind restora

FDARxBench: Benchmarking Regulatory and Clinical Reasoning on FDA Generic Drug Assessment

arXiv:2603.19539v2 Announce Type: replace-cross Abstract: We introduce an expert curated, real-world benchmark for evaluating document-grounded question-answering (QA) motivated by generic drug assess

HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

arXiv:2507.21815v2 Announce Type: replace Abstract: Millions of individuals' well-being are challenged by the harms of substance use. Harm reduction as a public health strategy provides non-judgementa

MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Models

arXiv:2609.01772v1 Announce Type: new Abstract: Meme understanding goes beyond recognizing visual content or literal text; it requires implicit cultural knowledge and pragmatic inference that most vis

A Benchmark for Vehicle Attribute Classification in Cross-Domain Surveillance Scenarios

arXiv:2609.01584v1 Announce Type: new Abstract: Vehicle attribute analysis is a key component of Intelligent Transportation Systems (ITS), supporting applications such as vehicle identification, traff

2 Sept 2026

Efficient SWE Agent Benchmarking via Trajectory-Aware Evaluation

arXiv:2609.01603v1 Announce Type: cross Abstract: Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and t

2 Sept 2026

Fi-ImageNet-1k: An OOD Benchmark From the Inside of the ImageNet-1k Validation Set

arXiv:2609.01027v1 Announce Type: new Abstract: Out-of-distribution (OOD) detection predicts whether a test image belongs to none of the predefined classes. To evaluate this task, benchmarks need imag

2 Sept 2026

EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents

arXiv:2601.16690v2 Announce Type: replace Abstract: We introduce EMemBench, a programmatic benchmark generator for evaluating long-term episodic memory of agents through interactive games. Rather than

HeTGB: A Comprehensive Benchmark for Heterophilic Text-Attributed Graphs

arXiv:2503.04822v2 Announce Type: replace-cross Abstract: Graph neural networks (GNNs) have demonstrated success in modeling relational data primarily under the assumption of homophily. However, many

Kaitchup posted Qwen3.8 27B Benchmarks for quants from Q4 to Q1

model-releasesr-localllama

Kaitchup just posted results of his benchmarks for Qwen3.8 27B for quants from different labs, Q4 to Q1, . All the details are hidden behind the paywall, but high level result is visible and looks lik

MLCommons Releases New MLPerf Storage v3.0 Benchmark Results

MLCommons announced the release of MLPerf Storage v3.0 benchmark results, expanding the suite to cover the full spectrum of AI storage workloads, including an S3 object‑storage access layer in additio

model-releasesmlcommons

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However,

OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

arXiv:2603.23938v2 Announce Type: replace Abstract: Most testbeds for omni-modal models assess multimodal understanding via textual outputs, leaving it unclear whether these models can properly speak

Open-Set Cattle Muzzle Identification: A Leakage-Controlled Benchmark and Evaluation Protocol

arXiv:2608.28663v1 Announce Type: new Abstract: Reliable individual cattle identification supports disease surveillance, vaccination records, breeding management, and livestock insurance. Although the

Qdrant Builds a 10-Billion-Document Vector Search Benchmark Dataset on Vultr

Qdrant has released **Qdrant‑Fineweb‑10B**, a public benchmark dataset comprising dense and sparse embeddings for 10 billion documents from the FineWeb corpus and exact results for 120,000 queries to

model-releasesvultr