Search: “benchmark”

OpenDocRouter is here! https://www.opendocrouter.ai/ Some time ago, I realized LlamaIndex was really good at benchmarking and applying nearly every model to document parsing (see: ParseBench). So, why

agentsjerry-liu--x

7 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences

DGX agent

arXiv:2610.08691v1 Announce Type: new Abstract: Large language model agents are accelerating scientific automation, yet verified executions rarely become persistent program-level improvements, and exi

model-releasesarxiv-cs-ai

7 Oct 2026

ST-Bench: A Spatial-Temporal Benchmark for Multi-Agent System Generation on Scientific Research Tasks

arXiv:2610.07763v1 Announce Type: new Abstract: The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where exec

7 Oct 2026

[Benchmark] Running Local LLMs on Orange Pi 5 Plus (RK3588, 16GB): Ollama Tok/s, NPU Offloading, Core Pinning & Thermals

model-releasesr-localllama

Disclosure: This unit was provided free of charge by Orange Pi for testing. No editorial review, no preconditions, no script. All data, bottlenecks, and thermal behavior are reported directly from har

decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x th…

model-releasesgeorgi-gerganov--x

decision models + local ai = ❤️ tried Laya (421M) on llama.cpp in a little apple-picking benchmark, it's almost as good as Qwen 4B at 12x the speed. Media Decision models in llama.cpp are now availabl

EMG-FM-Bench: A Comprehensive Benchmark for Foundation Model Transfer and Adaptation on Electromyography

arXiv:2610.06450v1 Announce Type: new Abstract: Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream phys

FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models

arXiv:2610.05790v1 Announce Type: new Abstract: Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecolog

MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models

arXiv:2510.13276v2 Announce Type: replace Abstract: The rapid advancement of long-context vision language models (LCVLMs) has led to a significant expansion of their context windows. However, an exten

Multi-Crop Leaf Disease Recognition: A Unified Benchmark and Cross-Region Study

arXiv:2610.04456v1 Announce Type: new Abstract: Deep learning models for crop leaf disease recognition routinely report near-perfect accuracy yet are typically trained and evaluated on a single datase

One-Cycle Fault Classification and Faulted-Line Identification on the PROTECT-90 Dataset: An Initial Application Benchmark

arXiv:2610.04155v1 Announce Type: cross Abstract: Open electromagnetic-transient datasets are beginning to make reproducible learning-based protection studies possible, but the practical use of these

WILLIE: A Unified Framework and Benchmark for Wound Classification, Segmentation, and Localization

arXiv:2610.05341v1 Announce Type: new Abstract: Chronic wound management affects over 8.2 million patients in the United States and imposes substantial clinical and economic burden. Clinical wound ass

Beyond Trained Models: Compiling GNNs for a Sound Explainer Benchmark

arXiv:2610.03526v1 Announce Type: cross Abstract: Explainers for Graph Neural Networks (GNNs) are commonly evaluated by their plausibility, i.e., how well their explanations recover a predefined groun

Safety

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

arXiv:2610.02460v1 Announce Type: new Abstract: Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fide

safetyarxiv-cs-cl

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

arXiv:2610.03020v1 Announce Type: new Abstract: Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history ev

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

model-releasesarxiv-cs-cl

arXiv:2610.02986v1 Announce Type: new Abstract: Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without acces

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

arXiv:2610.02982v1 Announce Type: new Abstract: Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large la

ReviewBench: An open benchmark for AI code review

model-releasesgithub-ai-blog

We’re launching ReviewBench, a benchmark for code review agents built on representative GitHub pull requests, multi-source ground truth, calibrated evaluation, and production-aligned metrics. The post

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

arXiv:2610.02599v1 Announce Type: new Abstract: Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug an

TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text

arXiv:2610.02256v1 Announce Type: new Abstract: Electricity price forecasting (EPF) supports scheduling, bidding, and risk management in electricity markets, yet existing benchmarks focus mainly on nu

Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling

arXiv:2610.01488v1 Announce Type: cross Abstract: Backchannel prediction has been studied almost entirely in dyadic conversation. We introduce a multi-party benchmark based on the AMI corpus, comprisi

Research

Ontology-Grounded, Reasoner-Verified Benchmarks for Evaluating LLM Reasoning in Scientific AI

arXiv:2610.00682v1 Announce Type: new Abstract: Large language models (LLMs) increasingly underpin scientific AI applications that reason over structured knowledge, from biomedical question answering

researcharxiv-cs-ai

ScholarCatalyst: A Benchmark for Retrieving Papers That Inspire New Research

arXiv:2610.02202v1 Announce Type: new Abstract: What makes great scientists great? Even as AI systems start to make progress on open problems, scientists remain far ahead of them at sensing which prio

VisionQ: VLM-as-a-Judge Taxonomy, Dataset and Benchmark for Qualitative Analysis in Computer Vision

arXiv:2610.00666v1 Announce Type: cross Abstract: Qualitative comparison figures are central evidence in computer vision papers, and vision-language models (VLMs) are increasingly used to judge them.

In a Streaming World, Should You Stand Still? A Comprehensive Benchmark of Anomaly Detection in Streams

arXiv:2609.39215v1 Announce Type: cross Abstract: Time series anomaly detection (TSAD) is increasingly deployed in streaming settings, where data arrive sequentially and may exhibit non-stationarity.

Measuring Structure in Graph Benchmark Datasets Using Graph Invariants

arXiv:2605.06462v2 Announce Type: replace Abstract: Progress in graph learning is hindered by benchmark practices that conflate the contributions of node features and graph structure, making it hard t

OSWorld-Science: A Benchmark of Computer Use Agents for Learning and Using Scientific Software

arXiv:2609.39903v1 Announce Type: new Abstract: Scientific software presents a demanding test for computer-using agents based on visual language models (VLMs): completing a research workflow requires

RealWorldShop: Benchmarking and Improving Conversational Shopping Agents in Real-World E-commerce

arXiv:2609.38974v1 Announce Type: new Abstract: Large language models are reshaping ecommerce from static recommenders into interactive shopping assistants, yet real-world shopping requires session-le

SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models

arXiv:2609.39847v1 Announce Type: cross Abstract: Audio language models (ALMs) are increasingly used for audio deepfake detection (ADD), yet existing benchmarks assess their verdicts or rationale plau

SHIFT-Truck: A High-Fidelity Aerodynamics Dataset and Benchmark for Pickup Trucks

arXiv:2609.38638v1 Announce Type: cross Abstract: Pickup trucks account for 14% of new light-duty vehicles produced in the United States, yet are among the least aerodynamic. Their open cargo bed adds

AerialDojo-200K: A Large-Scale Benchmark Suite for Open-World Aerial Object-Goal Search

arXiv:2609.36066v1 Announce Type: cross Abstract: Open-world aerial object-goal search is a foundational yet challenging task, requiring aerial agents to autonomously explore large-scale, unstructured

ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge

arXiv:2602.09839v2 Announce Type: replace Abstract: Existing multimodal retrieval benchmarks largely emphasize semantic matching on daily-life images and offer limited diagnostics of professional know

Beyond Legibility: Benchmarking Visual Text Rendering and In-Place Editing in Unified Video Generation

arXiv:2609.36598v1 Announce Type: new Abstract: A video can exhibit convincing motion and photorealism yet fail immediately when visual text collapses. Unlike generic scene content, visual text is unf

CruxBench: A Benchmark of Information Discovery

arXiv:2609.35879v1 Announce Type: cross Abstract: Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many comp

CypherTurn: A Multi-Turn Benchmark for Conversational Text-to-Cypher Evaluation and the Autonomy Divergence

model-releasesarxiv-cs-cl

arXiv:2609.36987v1 Announce Type: new Abstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the m

ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving

arXiv:2609.37871v1 Announce Type: cross Abstract: Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDri

HEAR: Real Voices, Real Bias: A Large-Scale Human-Recorded, Demographically Diverse Benchmark for Audio Language Models

arXiv:2609.35952v1 Announce Type: cross Abstract: We introduce HEAR (Human-recorded Evaluation of Audio-LLM bias by Real speakers), a large-scale, ecologically valid benchmark comprising 87k real huma

Recovering the View: Benchmarking Physical Active Vision for Occlusion Recovery in Robotic Manipulation

model-releasesarxiv-cs-ro

arXiv:2609.37292v1 Announce Type: new Abstract: Physical active vision allows robots to change their viewpoint when task-relevant observations become unreliable, yet existing manipulation benchmarks p

WitnessGym: Benchmarking Coding Agents on the Construction of Bug Witnesses

arXiv:2609.36635v1 Announce Type: cross Abstract: Bug validation asks a coding agent to produce an executable witness for a reported bug. The witness combines a concrete input with a testing harness a

Active Causal Discovery Benchmark: Evaluating LLM Agents Under Budgeted Interventions

arXiv:2609.31675v1 Announce Type: new Abstract: We introduce the Active Causal Discovery Benchmark (ACDB), an SCM-grounded environment for evaluating whether LLM agents recover causal graph structure

BOCCHI: A More Realistic and Challenging Benchmark for Local Motion Blur Detection with MSDCT-UNet

arXiv:2607.10427v2 Announce Type: replace Abstract: Local motion blur detection requires pixel-level localization of blurred regions. Existing benchmarks let models rely on gradient shortcuts that fai

DynGraphAgentBench: A Benchmark for Agentic Lifecycle Control in Dynamic Graph Anomaly Detection

arXiv:2609.33980v1 Announce Type: new Abstract: Dynamic graph anomaly detection requires repeated decisions as graph structure and class prevalence drift, yet detector benchmarks usually score a fixed

JRDB-AVR: An Active Visual Reasoning Benchmark for Embodied Agents in Real-World Environments

arXiv:2609.35032v1 Announce Type: cross Abstract: In complex embodied visual reasoning scenarios, an agent often has only a limited field of view, and the evidence needed to answer a question may be d

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

arXiv:2609.33893v1 Announce Type: new Abstract: Large audio-language models (LALMs) produce fluent responses about audio but often hallucinate by making plausible yet ungrounded claims. Existing audio

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

arXiv:2609.32810v1 Announce Type: cross Abstract: Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet bench

QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

arXiv:2609.35581v1 Announce Type: cross Abstract: We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debu

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

arXiv:2609.34384v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevan

VastMAT: A Large-Scale Multi-Category Benchmark for Multi-Animal Tracking

arXiv:2609.34390v1 Announce Type: new Abstract: Multi-animal tracking (MAT) supports the study of animal movement, behavior, and group interactions. However, general multi-object tracking (MOT) benchm

A Benchmark Framework for Screening Automation in Systematic Reviews