Search: “benchmark”

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

arXiv:2610.00523v1 Announce Type: new Abstract: Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy region

2 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing

DGX agent

arXiv:2610.00713v1 Announce Type: new Abstract: When a graph neural network (GNN) explainer produces an unexpected attribution on a molecule, the attribution alone cannot reveal whether the explainer

model-releasesarxiv-cs-lg

2 Oct 2026

A 3GPP-Compliant Benchmark Dataset for RIS-Aided Beyond 5G Networks

arXiv:2609.39058v1 Announce Type: cross Abstract: Reconfigurable Intelligent Surfaces (RIS) are emerging as a key technology for programmable wireless environments in the beyond the fifth generation (

AneumoBench: A Source-Linked Benchmark for Synthetic-Geometry Transfer in Aneurysm CFD

arXiv:2505.14717v2 Announce Type: replace-cross Abstract: Scientific machine learning uses simulation data to train surrogate models for fast physical-field prediction across geometries. Local shape e

Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation

arXiv:2609.38886v1 Announce Type: new Abstract: Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However,

Safety

Benchmarking EMlog Calibration for Autonomous Surface Vehicles

arXiv:2609.39203v1 Announce Type: new Abstract: Accurate velocity measurement is a fundamental requirement for autonomous surface and underwater vehicles. Commonly, velocity is provided by a Doppler v

safetyarxiv-cs-ro

MADBench: Benchmarking the Security of Multi-Agent Debate

arXiv:2609.39146v1 Announce Type: new Abstract: Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same

Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning

arXiv:2609.39437v1 Announce Type: new Abstract: Large language models (LLMs) encode substantial latent geographic knowledge, yet they reason poorly over space and are unreliable when queried from coor

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular int

Applications

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants

arXiv:2609.37559v1 Announce Type: new Abstract: To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use. Yet existing stre

applicationsarxiv-cs-cv

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

arXiv:2609.36777v1 Announce Type: new Abstract: Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely cont

CoSec: Benchmarking Agent Security in Communities

arXiv:2609.34790v2 Announce Type: replace-cross Abstract: LLM agents operate in persistent collaborative environments involving multiple users, communities, memories, files, and tools. Community bound

CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning

arXiv:2609.35823v1 Announce Type: new Abstract: Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes.

Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages

arXiv:2609.36970v1 Announce Type: new Abstract: The Rashomon effect describes the existence of multiple near-optimal models that achieve comparable performance while offering fundamentally different e

Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL

arXiv:2609.38065v1 Announce Type: cross Abstract: Training agents to follow arbitrary instructions is an important goal of multi-task reinforcement learning (RL). Linear temporal logic (LTL) provides

VesselBench-800K: A Large-scale Perception Benchmark for Multimodal Vessel Detection, Counting, and Density Estimation

arXiv:2609.37003v1 Announce Type: new Abstract: Vessel perception from space is crucial for a wide range of maritime applications, from traffic monitoring to environmental protection. However, most ex

3dgs-sc: a controlled static screen-content benchmark for 3d gaussian splatting

arXiv:2609.31756v1 Announce Type: new Abstract: Does high image fidelity imply readable screen content in 3D Gaussian Splatting (3DGS)? We introduce 3DGS-SC, a controlled static screen-content dataset

AmbiModBench: Benchmarking Gene Perturbation Prediction Beyond Shared Responses

arXiv:2609.32527v1 Announce Type: cross Abstract: Predicting cellular responses to genetic perturbations helps prioritize experiments in single-cell genomics, where exhaustive measurement is infeasibl

Safety

AnesTRACE: Benchmarking Intraoperative Anesthesia from Multimodal Perception to Multi-step Decision-Making

arXiv:2609.32740v1 Announce Type: new Abstract: Intraoperative anesthesia requires systems to interpret evolving multimodal evidence, recommend timely management, and revise decisions as patient state

safetyarxiv-cs-cv

ARCH-B: Architectural Representation, Comprehension and Hierarchy Benchmark

arXiv:2609.34047v1 Announce Type: new Abstract: Multimodal models increasingly interpret visual environments, but their ability to recognize the same building across photographs, floor plans, elevatio

Beyond Geometry: Benchmarking and Consistency Reasoning for 3D Logical Anomaly Detection

arXiv:2609.34143v1 Announce Type: new Abstract: Existing 3D industrial anomaly detection mainly targets local geometric deviations. In contrast, many industrial anomalies violate object-level design o

ConflictVLA-Bench: Benchmarking Behavioral Responses of Vision-Language-Action Models to Premise Conflicts

arXiv:2609.31792v1 Announce Type: cross Abstract: While Vision-Language-Action (VLA) models perform strongly on manipulation tasks, their responses to invalid task premises remain underexplored. Exist

CoSE-E: A Benchmark for Code-switched Speech Evaluation in Enterprise Settings

arXiv:2609.35645v1 Announce Type: cross Abstract: Code-switching (CS), a seamless alternation between languages within a single utterance, remains a critical challenge in automatic speech recognition

Cross-Dataset Generalization of Bangladeshi Rice Leaf Disease Classifiers: Benchmark, Diagnosis, and Mitigation

arXiv:2609.31709v1 Announce Type: new Abstract: Cross-dataset transfer in rice leaf disease classification remains a significant challenge, with models trained on one image collection performing subst

Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining

arXiv:2609.31870v1 Announce Type: new Abstract: Consistently profitable trading is difficult because equity markets are noisy, non-stationary, and only partially predictable from historical data. We b

eval-unlearn: Benchmarking unlearning in Text-to-Image Diffusion Models

arXiv:2609.35269v1 Announce Type: cross Abstract: The rising number of concept unlearning techniques for text-to-image (T2I) diffusion models has produced a fragmented evaluation landscape. Methods ar

Generative AI-Based Data Augmentation for Oral Lesion Classification: The PhotoMOCI Dataset and Benchmark

arXiv:2609.35226v1 Announce Type: new Abstract: Early detection of oral cancer via photographic imaging presents a promising avenue for large-scale oral cavity screening. However, the development of r

Jev in Medicine: A Benchmark Evaluation. Preliminary Results

arXiv:2609.34024v1 Announce Type: cross Abstract: Jev is a non-generative 'System One' model that assigns probabilities to predefined answer options and cannot answer outside them. Its accuracy and ca

Large Language Models for Automated Cross-Domain Machine Learning Task Type Identification: A Benchmark Dataset and Evaluation

arXiv:2609.35335v1 Announce Type: new Abstract: Machine learning task type identification is essential for constructing valid ML pipelines, yet in practice it is typically specified manually. We inves

MemTransfer: Benchmarking Memory Beyond Matched Experience in Embodied Decision-Making

arXiv:2609.32313v1 Announce Type: new Abstract: Memory lets an embodied agent reuse past experience, yet retaining useful information does not ensure that the agent can apply it when conditions change

NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations

arXiv:2607.10490v2 Announce Type: replace-cross Abstract: Tool-using large language model (LLM) agents are attractive for network operations, but tickets, alerts, logs, runbooks, and ChatOps messages

One-Step Is Optimal: Unconditional Rectified Flows are Noise2Noise Denoisers, and Multi-Step Integration Provably Hurts---A Benchmark and Task-Based Detectability Study on Low-Dose CT

arXiv:2609.31670v1 Announce Type: new Abstract: Iterative and generative denoisers are increasingly used under the assumption that multi-step refinement outperforms a single regression pass. We show t

OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models

arXiv:2609.35052v1 Announce Type: new Abstract: Video world models must preserve the visual state of the world over time, but existing evaluation protocols often rely on generated histories, video ref

ORAV: Benchmarking Audio-Video Generation from Multimodal Contexts

arXiv:2609.34843v1 Announce Type: new Abstract: Audio-video generation using heterogeneous multimodal references has emerged as a new challenge, requiring both compositional control over generation an

SV2V-RSim: A Comprehensive Benchmark for Self-Selective V2V Cooperative Perception with Near-Realistic Data

arXiv:2609.32863v1 Announce Type: new Abstract: Vehicle-to-Vehicle (V2V) cooperative perception enhances autonomous driving by enabling vehicles to share information beyond their direct line of sight.

THEIA: A Multimodal Dataset and Benchmark for Vision-Language Analysis of Layout

arXiv:2609.35035v1 Announce Type: new Abstract: The integration of artificial intelligence into computer-aided design frameworks has sparked a shift in the design of analog integrated circuits (ICs),

Traceable Human-to-Humanoid Sign Language Benchmarking

arXiv:2609.33354v1 Announce Type: new Abstract: Sign data collection is costly, and teleoperation scales poorly, motivating reuse of large video corpora. Humanoid signing requires converting video-der

When Does Geometric View Synthesis Help Wine Label Retrieval? A Public One-Shot Benchmark Across Self-Supervised and Vision-Language Backbones

arXiv:2609.33359v1 Announce Type: new Abstract: Geometric view synthesis can expand a single wine-label photograph into a training set, but its value with pretrained image encoders is unclear. We stud

Research

Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

arXiv:2609.33180v1 Announce Type: cross Abstract: As recursive self-improvement (RSI) rapidly advances, reliable evaluation becomes critical for guiding adaptive search. RSI typically relies on finite

researcharxiv-cs-lg

A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory

arXiv:2609.30813v1 Announce Type: new Abstract: Evaluating claim admission in shared agent memory is challenging because repeated claims may be mistaken for independent evidence. An agent may copy or

28 Sept 2026

Artificial Societies Benchmark: A Validation Framework for Synthetic Research

arXiv:2609.30030v1 Announce Type: new Abstract: A synthetic survey can reproduce the average answer while misrepresenting how people differ, how their answers relate to one another, or how they respon

Benchmarking Arabic--Russian Machine Translation: A Comparison of Fine-tuned NMT and Few-shot LLMs under Rich Morphology and Low Lexical Overlap

arXiv:2609.29559v1 Announce Type: new Abstract: Arabic-Russian machine translation (MT) remains under-explored due to the rich morphology of Arabic and low lexical overlap between the two languages. W

Benchmarking Argumentative Behaviour of LLMs: A Study of Defences Against Character Attacks

arXiv:2609.28673v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed as argumentative agents in persuasive dialogues, necessitating rigorous evaluation of their debat

Clinical Intent Extraction: A FHIR-Aligned Representation and the CIRCA Benchmark

arXiv:2609.29479v1 Announce Type: new Abstract: Prospective clinical actions, the follow-ups, orders, referrals, and instructions that deter-mine what happens to a patient next, are annotated today in

Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NER

arXiv:2609.29101v1 Announce Type: new Abstract: Medical Named Entity Recognition (NER) for low-resource languages remains a challenging task due to high linguistic variability and a scarcity of domain

Benchmarking Active Spot Selection for Cost-Efficient Spatial Transcriptomics

arXiv:2609.27208v1 Announce Type: new Abstract: Spatial transcriptomics (ST) measures gene expression in tissue context, but dense capture grids can be costly and may repeatedly sample morphologically

24 Sept 2026

Benchmarking Hyperspectral Foundation Models for Hyperspectral Unmixing

arXiv:2609.28283v1 Announce Type: new Abstract: Several foundation models dedicated to hyperspectral images have recently been made available. These models are trained on large unlabeled datasets and

24 Sept 2026

Can LLMs Catch a Rigged Backtest? A Clean-Control Calibration Benchmark

arXiv:2609.28090v1 Announce Type: cross Abstract: Backtest auditing is a calibration problem: high flaw recall is not useful when the model falsely flags matched clean strategies. We build a 96-item p