Search: “benchmark”

From Triage to Impact: A Multi-Hop Reasoning Benchmark for Explainable Persian Hate Speech

arXiv:2610.20370v1 Announce Type: new Abstract: Hate speech has become increasingly prevalent on social media, raising concerns about its detection, interpretation, and moderation. Persian, a relative

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Large language model-generated patterns for epidemiological modeling: a multi-benchmark evaluation framework

DGX agent

arXiv:2610.15994v1 Announce Type: new Abstract: Representing individual-level behavioral heterogeneity is challenging in epidemiological modeling, particularly when empirical behavioral data are spars

model-releasesarxiv-cs-ai

15 Oct 2026

LintEval: Benchmarking LLMs on Linter-Guided File Refactoring

arXiv:2610.20027v1 Announce Type: cross Abstract: Linters are widely used in code maintenance and real-time IDE editing to surface lint alerts related to performance, style, error-prone, and other pat

MedHarnessBench: A Unified Benchmark for Healthcare Agent Evaluation

arXiv:2610.15714v1 Announce Type: new Abstract: Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions und

NEPSEE: A Double-Annotated Benchmark for Nepali Sentiment Analysis

arXiv:2610.19190v1 Announce Type: new Abstract: Many sentiment datasets use one annotator for each item, so the test labels show the conventions of that person. We build a Nepali benchmark from two an

PhysCodeGym: Benchmarking Coding Agents for Physically Verifiable 4D Reconstruction

arXiv:2610.17288v1 Announce Type: new Abstract: Building physical simulations from video requires coding agents to translate observations into executable models and refine them through tool use. We in

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

arXiv:2412.12877v3 Announce Type: replace Abstract: Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the

When Non-Adversarial Noise Hurts Learning: Benchmarking Skill Self-Evolution in Agents

arXiv:2610.19910v1 Announce Type: new Abstract: Large language model agents increasingly improve by distilling experience into reusable skills, but errors in that experience can persist in learned rul

A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding

arXiv:2610.15666v1 Announce Type: new Abstract: Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interac

Can Synthetic UAV Benchmarks Be Trusted? Real-Synthetic Validation for UAV Model Selection

arXiv:2610.14647v1 Announce Type: new Abstract: Synthetic UAV imagery can expand pre-deployment test evidence when task-specific Real data are scarce, but visual plausibility or semantic label alignme

EmoGenie: Benchmarking and Evaluating Expressed and Evoked Emotion in T2I Generation with MLLMs

arXiv:2610.18455v1 Announce Type: new Abstract: Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with

LADDER: A Long-Horizon Robot Manipulation Benchmark

arXiv:2610.18053v1 Announce Type: cross Abstract: General-purpose robot manipulation policies learn short-horizon skills effectively, but degrade rapidly on long-horizon tasks. Existing benchmarks can

MFVSR: A Multi-Style and Fine-Grained Benchmark for Video Subtitle Removal

arXiv:2610.18461v1 Announce Type: new Abstract: Video subtitle removal (VSR) is an important video restoration task, yet existing evaluation remains limited by low-resolution datasets, fragmented test

Observation Quality in DSM-Assisted Aerial Localization: A Benchmark and Base-Anchored Uncertainty Calibration

arXiv:2610.18564v1 Announce Type: new Abstract: Aerial map matching provides position observations for navigation, but a position error alone does not determine how an observation should be weighted.

PhySimCode: A Benchmark and Evaluation Method for Physics Video to Code Generation

arXiv:2610.15722v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physic

VL-DocIR: A Benchmark for Vision-Based Long Document Information Retrieval

arXiv:2610.17057v1 Announce Type: cross Abstract: Vision-based document retrieval has improved rapidly, but current evaluation settings still emphasize short documents and single relevant pages. This

BLiMPPI: Benchmark of Linguistic Minimal Pairs for Polarity Items

arXiv:2610.13607v1 Announce Type: new Abstract: Negative polarity items (NPIs), such as 'anymore' and 'ever', are grammatical only in select environments, predominantly in the context of negation. Bec

GraphDecisionBench: Benchmarking System One Models on Graph Tasks

arXiv:2610.06354v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly explored for graph understanding and decision-making, while System One models such as Jev select direc

SeoulMMOD: A Large-Scale Multimodal Origin-Destination Flow Benchmark

arXiv:2610.13730v1 Announce Type: new Abstract: Origin-destination (OD) flow forecasting supports urban mobility decisions by predicting the volume and patterns of human movement across a city. Most e

SWE-DeceptionBench: Benchmarking Deceptive Behaviors of LLMs Across the Software Development Lifecycle

arXiv:2610.13626v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used throughout the software development lifecycle. This growing reliance on LLMs raises concerns about

Demonstrating Arena 5.0: A Photorealistic ROS2 Simulation Framework for Developing and Benchmarking Social Navigation

model-releasesarxiv-cs-ro

arXiv:2610.11220v1 Announce Type: new Abstract: Building upon the foundations laid by our previous work, this paper introduces Arena 5.0, the fifth iteration of our framework for robotics social navig

9 Oct 2026

OpenProblemBench: Benchmarking AI on Open Problems in the Foundational Theoretical Sciences

arXiv:2610.11118v1 Announce Type: new Abstract: The next frontier for artificial general intelligence is tackling unresolved scientific problems, calling for benchmarks that assess progress beyond est

9 Oct 2026

Pumpire: Unified Benchmark for Metric Distance Estimation

arXiv:2610.12423v1 Announce Type: new Abstract: We present Pumpire, a unified benchmark for evaluating metric point-pair distance estimation capability of both image- and video-level 3D foundation mod

9 Oct 2026

AirGroundVLN: A Large-Scale Benchmark for Goal-Oriented Air-Ground Collaborative Vision-and-Language Navigation

model-releasesarxiv-cs-ro

arXiv:2610.10421v1 Announce Type: new Abstract: Goal-oriented Vision-and-Language Navigation (VLN) requires agents to locate and reach targets described in natural language without prescribed routes.

Are We Really Benchmarking Forecasting Models? The Impact of Preprocessing on Time Series Performance

arXiv:2610.09096v1 Announce Type: new Abstract: While established literature underscores the pivotal role of preprocessing in forecasting accuracy, this stage remains largely overlooked in current res

Latent Watermarks under Generative Editing: A Benchmark and Analysis of Detection Survival

arXiv:2610.09702v1 Announce Type: new Abstract: Ordinary prompt-based editing can cause latent watermark detection to fail without explicitly targeting the watermark. We benchmark eight watermark meth

OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework

model-releasesarxiv-cs-ro

arXiv:2610.10384v1 Announce Type: new Abstract: Tactile feedback provides embodied agents with physical information beyond visual observations, enabling more reliable interaction with the real world.

Pre-training, Reasoning, Benchmarking: X-ray Report Generation on CheXpert Plus Dataset

arXiv:2610.08813v1 Announce Type: new Abstract: X-ray image-based Radiology Report Generation (RRG) constitutes a critical research direction within medical artificial intelligence, with great potenti

VCR-Bench: A Modular Open-Source Benchmark for Video Classification Robustness

arXiv:2610.08936v1 Announce Type: new Abstract: Robustness of image classification has several benchmarks, but their video counterparts are absent. In video classification temporal dimension introduce

Event Cameras for Melt-Pool Monitoring in Additive Manufacturing: A Benchmark and a Cross-Machine Transfer Analysis

arXiv:2610.06973v1 Announce Type: new Abstract: Melt-pool monitoring is central to qualifying metal additive manufacturing (AM), yet no public event-camera benchmark exists for this domain. Event came

FactorBench: A Portfolio-Aware Benchmark for Automated Factor Mining

arXiv:2610.06947v1 Announce Type: cross Abstract: Factor mining seeks to discover signals from financial data that predict future asset returns and guide portfolio construction. Automated factor minin

Language Unalignability: Why Some Concepts Resist Cross-Cultural Benchmark Evaluation

arXiv:2610.08303v1 Announce Type: new Abstract: Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures

ScienceClaw: Benchmarking Continual Self-Evolution of AI-for-Science Agents Across the Natural and Social Sciences

arXiv:2610.08691v1 Announce Type: new Abstract: Large language model agents are accelerating scientific automation, yet verified executions rarely become persistent program-level improvements, and exi

ST-Bench: A Spatial-Temporal Benchmark for Multi-Agent System Generation on Scientific Research Tasks

arXiv:2610.07763v1 Announce Type: new Abstract: The rapid progress of LLM-based multi-agent systems (MAS) has shown that they largely outperform single agents on coding, math, and QA tasks, where exec

EMG-FM-Bench: A Comprehensive Benchmark for Foundation Model Transfer and Adaptation on Electromyography

arXiv:2610.06450v1 Announce Type: new Abstract: Foundation models (FMs) are increasingly being developed for general time series and physiological signals, yet their transferability to downstream phys

FairRSFM: A Biome-Aware Benchmark and Debiasing Framework for Remote Sensing Foundation Models

arXiv:2610.05790v1 Announce Type: new Abstract: Remote sensing foundation models (RSFMs) are commonly evaluated using aggregate metrics, which can hide systematic performance disparities across ecolog

MMLongCite: A Benchmark for Evaluating Faithfulness of Long-Context Vision-Language Models

arXiv:2510.13276v2 Announce Type: replace Abstract: The rapid advancement of long-context vision language models (LCVLMs) has led to a significant expansion of their context windows. However, an exten

Multi-Crop Leaf Disease Recognition: A Unified Benchmark and Cross-Region Study

arXiv:2610.04456v1 Announce Type: new Abstract: Deep learning models for crop leaf disease recognition routinely report near-perfect accuracy yet are typically trained and evaluated on a single datase

One-Cycle Fault Classification and Faulted-Line Identification on the PROTECT-90 Dataset: An Initial Application Benchmark

arXiv:2610.04155v1 Announce Type: cross Abstract: Open electromagnetic-transient datasets are beginning to make reproducible learning-based protection studies possible, but the practical use of these

WILLIE: A Unified Framework and Benchmark for Wound Classification, Segmentation, and Localization

arXiv:2610.05341v1 Announce Type: new Abstract: Chronic wound management affects over 8.2 million patients in the United States and imposes substantial clinical and economic burden. Clinical wound ass

Beyond Trained Models: Compiling GNNs for a Sound Explainer Benchmark

arXiv:2610.03526v1 Announce Type: cross Abstract: Explainers for Graph Neural Networks (GNNs) are commonly evaluated by their plausibility, i.e., how well their explanations recover a predefined groun

Safety

CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking

arXiv:2610.02460v1 Announce Type: new Abstract: Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fide

safetyarxiv-cs-cl

DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users

arXiv:2610.03020v1 Announce Type: new Abstract: Long-term agents must remember not only what is true about a user, but also how a particular agent should work with that user as their shared history ev

OLMo-Detect: A Multi-Stage, Confounder-Controlled Benchmark for Membership Inference on Large Language Models

arXiv:2610.02986v1 Announce Type: new Abstract: Membership inference on large language models (LLMs) aims to determine whether a given text sample was included in an LLM's training data, without acces

PLCWorld: Benchmarking LLM-Generated PLC Programs in Closed-Loop Plant Simulation

arXiv:2610.02982v1 Announce Type: new Abstract: Programmable logic controllers (PLCs) coordinate industrial equipment by reading sensor inputs and issuing control commands. Evaluating whether large la

TasteBench: Multimodal Benchmark for Sensory Prediction, from Molecules to Sustainable Foods

arXiv:2610.02599v1 Announce Type: new Abstract: Sustainable protein discovery lacks the fast computational proxies, analogous to molecular docking or density functional theory, that accelerate drug an

TRACE: A Reproducible Benchmark for Electricity Price Forecasting with Official Operational Text

arXiv:2610.02256v1 Announce Type: new Abstract: Electricity price forecasting (EPF) supports scheduling, bidding, and risk management in electricity markets, yet existing benchmarks focus mainly on nu

Multi-Party Backchannel Prediction: a Diagnosis, a Benchmark, and a Ceiling

arXiv:2610.01488v1 Announce Type: cross Abstract: Backchannel prediction has been studied almost entirely in dyadic conversation. We introduce a multi-party benchmark based on the AMI corpus, comprisi