Search: “benchmark”

In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation

arXiv:2610.13411v1 Announce Type: cross Abstract: Traditional system identification typically involves two steps: initially, training data is processed to estimate a model of the system of interest; t

12 Oct 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization

DGX agent

arXiv:2610.12183v1 Announce Type: cross Abstract: Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited. Recent large la

model-releasesarxiv-cs-ai

9 Oct 2026

Agent4RE: A Self-Refining Multi-agent Framework for End-to-End Software Requirements Engineering and Benchmarking

arXiv:2610.10628v1 Announce Type: cross Abstract: Existing LLM-based approaches for software Requirements Engineering (RE) typically rely on basic prompting strategies or rudimentary agent collaborati

Beyond Imitation: A Framework and Benchmark for LLM-Assisted Peer Review

arXiv:2610.11087v1 Announce Type: new Abstract: The rapid growth of scientific publishing has strained peer review, particularly in machine learning, raising concerns about declining review quality an

Conversational Task Disambiguation over Tabular Data: Leakage-Aware Formulation, Benchmark Suite, and Training

arXiv:2610.10740v1 Announce Type: cross Abstract: Conversational task disambiguation over tabular data uses dialogue to resolve missing information about a user's intended task before producing a solu

How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defense

arXiv:2610.11005v1 Announce Type: new Abstract: Safety-aligned language models often refuse a harmful request stated directly but answer the same request inside a role-play or narrative wrapper. We me

LIVIN: Benchmarking Spatial and Embodied Intelligence in Digital Twins of Lived-In Homes

arXiv:2610.12069v1 Announce Type: new Abstract: Realistic household simulation must capture not only diverse environments but also the lived-in object arrangements and spatial constraints that shape r

READ-Bench: Benchmarking Historical Instance Retrieval for Time-Series Diagnosis

arXiv:2609.32123v2 Announce Type: replace Abstract: Time-series diagnostic systems rarely rely on retrieving relevant historical cases, and when they do, retrieval is evaluated only indirectly through

RideBench: A Large-Scale Exogenous-Aware Benchmark for Ride-Hailing Time Series Forecasting

arXiv:2610.11164v1 Announce Type: new Abstract: We release Ride-Hailing, a large-scale ride-hailing time series dataset synthesized from DiDi's marketplace data across 200 spatial areas. Ride-Hailing

Seeing Through the Glare: A Multi-Source Benchmark and Ocular-Adaptive Pixel MeanFlow for Eyeglass Reflection Removal

arXiv:2610.10703v1 Announce Type: new Abstract: Eyeglass reflection removal is important across smartphone imaging, video conferencing, and other face-centric visual applications. The task is challeng

TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models

arXiv:2610.11392v1 Announce Type: new Abstract: System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generati

Benchmarking Behavioral Steerability in Behavior Foundation Models

arXiv:2610.10198v1 Announce Type: new Abstract: Behavior Foundation Models (BFMs) are emerging as a paradigm for translating human intentions into executable humanoid behaviors. As these models evolve

CARES: A Controlled Synthetic Benchmark of Speaker Reactions to Sound

arXiv:2610.10208v1 Announce Type: cross Abstract: Automatic audio scene description turns a recording into a text account of a situation. One difficulty is deciding which elements of the audio should

Do Vision Models Learn Physical Constraints or Rendering Shortcuts? A Counterfactual Benchmark for Grounded Physical Consistency

arXiv:2610.09205v1 Announce Type: new Abstract: Modern image editing models can satisfy a text instruction while breaking the physics of the edited scene. A new object may cast no shadow, a mirror may

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

model-releasesarxiv-cs-cl

arXiv:2610.09671v1 Announce Type: new Abstract: Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decisio

RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment

arXiv:2610.09294v1 Announce Type: cross Abstract: Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move i

UltraText Bench: A Comprehensive Bilingual Benchmark for Evaluating Visual Text Rendering in Image Generation

arXiv:2610.09823v1 Announce Type: new Abstract: Dense visual text requires image generators to reproduce long strings across multiple regions with correct placement and legibility. As short-string ren

Research

HygieneRoboBench: Benchmarking Hygiene-Aware Planning for Household Robots

arXiv:2610.08642v1 Announce Type: cross Abstract: Contact with contaminated objects can spread hazards through a household robot's grippers, tools, and shared surfaces, while new contacts can make an

researcharxiv-cs-ai

Label-Efficient Deep Learning for ECG Delineation: A Multi-Dataset Benchmark against Widely Used Delineation Tools

arXiv:2610.07885v1 Announce Type: cross Abstract: Electrocardiogram (ECG) delineation, the identification of waveform boundaries, is a foundational step that translates raw ECG signals into clinically

Neutrosophic Ensemble Classification for Uncertainty-Aware Bearing Fault Detection: Evidence from Laboratory and Variable-Speed Industrial Benchmarks

arXiv:2610.06880v1 Announce Type: cross Abstract: Machine learning classifiers for bearing fault detection produce scalar confidence scores that conflate confident errors with genuinely ambiguous pred

TabiBERT: A Large-Scale ModernBERT Foundation Model and A Unified Benchmark for Turkish

model-releasesarxiv-cs-cl

arXiv:2512.23065v4 Announce Type: replace Abstract: The introduction of BERT established encoder-only transformer models as a foundational paradigm in natural language processing. Encoder-only models

Benchmarking Generative Trajectory Models for Active-Inference Control

arXiv:2610.05692v1 Announce Type: new Abstract: Learning from trajectory demonstrations offers a route to active-inference control of complex systems whose dynamics are difficult to model explicitly.

How Execution Assumptions Change Short-Horizon Sharpe Rankings: Evidence from a Synthetic Trading Benchmark

arXiv:2610.05077v1 Announce Type: new Abstract: Backtests of LLM trading agents often assume that every order fills at the closing price. We ask whether this choice changes only reported returns or al

WasserMan: Benchmark for Underwater Manipulation Policy Learning

arXiv:2610.04536v1 Announce Type: new Abstract: Underwater manipulation couples visual decisions, contact forces and a thruster-controlled floating base. We introduce WasserMan, to our knowledge the f

Watermarks and Fingerprints as Soft Bindings for Content Provenance: An Open-Licence Benchmark for Images, Audio and Video

arXiv:2610.04151v1 Announce Type: new Abstract: Content-provenance standards such as C2PA let a platform recover a stripped manifest through a soft binding: an invisible watermark read from the conten

A benchmark for LLMs playing Civilization V. GLM-5.3 is ahead of Opus-5.5, and Qwen-3.8-27B holds up surprisingly well.

model-releasesr-localllama

A while ago, I posted here getting OSS-120B and GLM-4.6 playing full games of Civilization V. Since then, models have moved pretty far, and we wanted a better understanding about models' capabilities

DexJoCo-X: Benchmarking Action Representations for Multi-Hand Dexterous Manipulation

arXiv:2610.03278v1 Announce Type: new Abstract: As dexterous hands proliferate, collecting data and training policies separately for every morphology becomes increasingly impractical. Scalable cross-e

EditHero: A Benchmark for Long-Horizon Part-Level 3D Editing and Vibe Modeling

arXiv:2610.02298v1 Announce Type: cross Abstract: 3D editing methods are usually tested on a single edit, yet an asset is built through a long sequence of revisions, each of which must implement the r

Applications

Time Series Forecasting Benchmarks Need Scenario-Grounded Stress Testing

arXiv:2610.02608v1 Announce Type: new Abstract: Time series forecasting (TSF) increasingly drives decisions in transportation, energy, finance, healthcare, and infrastructure, yet current evaluation r

applicationsarxiv-cs-ai

Does Continual Imitation Learning Remain Grounded? A Language-Perturbed Benchmark for Robotic Task Retention

arXiv:2610.00542v1 Announce Type: new Abstract: Continual imitation learning evaluates whether a robot can learn new knowledge without forgetting previously learned skills. However, retaining task per

2 Oct 2026

SimplexUQ: An Evaluation Framework and Benchmark for Conformal Uncertainty on Simplex-Valued Predictions

arXiv:2610.00523v1 Announce Type: new Abstract: Conformal prediction guarantees marginal coverage, but a single calibration threshold can still spread that coverage unevenly, over-covering easy region

2 Oct 2026

WOMBAT: Whitebox Oracle for Molecular Benchmarking and Attribution Testing

arXiv:2610.00713v1 Announce Type: new Abstract: When a graph neural network (GNN) explainer produces an unexpected attribution on a molecule, the attribution alone cannot reveal whether the explainer

2 Oct 2026

A 3GPP-Compliant Benchmark Dataset for RIS-Aided Beyond 5G Networks

arXiv:2609.39058v1 Announce Type: cross Abstract: Reconfigurable Intelligent Surfaces (RIS) are emerging as a key technology for programmable wireless environments in the beyond the fifth generation (

AneumoBench: A Source-Linked Benchmark for Synthetic-Geometry Transfer in Aneurysm CFD

arXiv:2505.14717v2 Announce Type: replace-cross Abstract: Scientific machine learning uses simulation data to train surrogate models for fast physical-field prediction across geometries. Local shape e

Benchmarking and Enhancing Skill-Level Memory for Partially Observable Robotic Manipulation

arXiv:2609.38886v1 Announce Type: new Abstract: Recent advances in robot learning have enabled manipulation policies to perform increasingly diverse tasks and generalize across environments. However,

Safety

Benchmarking EMlog Calibration for Autonomous Surface Vehicles

arXiv:2609.39203v1 Announce Type: new Abstract: Accurate velocity measurement is a fundamental requirement for autonomous surface and underwater vehicles. Commonly, velocity is provided by a Doppler v

safetyarxiv-cs-ro

Local Ai

DDR4/PCIe4 vs DDR5/PCIe5 for LLMs- I benchmarked them for pre-training. What are your thoughts?

Hello everyone. I've recently ran some experiments comparing DDR4/PCIe4 and DDR5/PCIe5 for AI workstations on a pre-training run, and would like to hear yours thoughts. First of all, and as a summary

local-air-localllama

MADBench: Benchmarking the Security of Multi-Agent Debate

arXiv:2609.39146v1 Announce Type: new Abstract: Multi-agent debate (MAD) can improve large language model (LLM) reasoning by allowing multiple agents to exchange and critique their answers to the same

Network-based Spatial Context Retrieval for Open-weight LLMs: A Faithfulness Benchmark for Grounded Geographic Reasoning

arXiv:2609.39437v1 Announce Type: new Abstract: Large language models (LLMs) encode substantial latent geographic knowledge, yet they reason poorly over space and are unreliable when queried from coor

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

arXiv:2609.38867v1 Announce Type: new Abstract: Large language model (LLM) computer-use agents are typically evaluated with clean written instructions, despite speech being an increasingly popular int

Applications

APM-Bench: Benchmarking Cross-session Persistent Memory for Egocentric Streaming Video Assistants

arXiv:2609.37559v1 Announce Type: new Abstract: To serve as real-world personal assistants, streaming video models need persistent memory that retains past experiences for later use. Yet existing stre

applicationsarxiv-cs-cv

Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes

arXiv:2609.36777v1 Announce Type: new Abstract: Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely cont

CoSec: Benchmarking Agent Security in Communities

arXiv:2609.34790v2 Announce Type: replace-cross Abstract: LLM agents operate in persistent collaborative environments involving multiple users, communities, memories, files, and tools. Community bound

CoVLM-Bench: A Real-World Benchmark for Cooperative Driving Question Answering and Planning

arXiv:2609.35823v1 Announce Type: new Abstract: Vision-language models (VLMs) have made substantial progress in autonomous driving, but their success has primarily been studied in ego-centric scenes.

Equally Good, Yet Different: Benchmarking Rashomon sets in AutoML packages

arXiv:2609.36970v1 Announce Type: new Abstract: The Rashomon effect describes the existence of multiple near-optimal models that achieve comparable performance while offering fundamentally different e

Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RL

arXiv:2609.38065v1 Announce Type: cross Abstract: Training agents to follow arbitrary instructions is an important goal of multi-task reinforcement learning (RL). Linear temporal logic (LTL) provides

VesselBench-800K: A Large-scale Perception Benchmark for Multimodal Vessel Detection, Counting, and Density Estimation

arXiv:2609.37003v1 Announce Type: new Abstract: Vessel perception from space is crucial for a wide range of maritime applications, from traffic monitoring to environmental protection. However, most ex

3dgs-sc: a controlled static screen-content benchmark for 3d gaussian splatting

arXiv:2609.31756v1 Announce Type: new Abstract: Does high image fidelity imply readable screen content in 3D Gaussian Splatting (3DGS)? We introduce 3DGS-SC, a controlled static screen-content dataset