Search: “benchmark”

Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmark

arXiv:2609.22584v1 Announce Type: new Abstract: As industrial processes grow in complexity, traditional Proportional-Integral-Derivative (PID) controllers are often insufficient for handling their non

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark

DGX agent

arXiv:2609.22222v1 Announce Type: cross Abstract: Large language models can generate executable data-analysis code, but successful execution is not equivalent to a valid official-statistics result. Th

model-releasesarxiv-cs-cl

22 Sept 2026

ECP-Bench: Benchmarking and Learning Entertainment Content Promotion with Foundation Models

arXiv:2609.22150v1 Announce Type: cross Abstract: Content promotion spans a broad set of skills, from understanding content to forecasting its market reception. However, LLMs' ability to support such

ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift

arXiv:2609.23659v1 Announce Type: new Abstract: Ethereum decentralized finance (DeFi) provides a public, time-stamped record of transaction-level event streams, but the same public symbols can create

LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions

arXiv:2609.24172v1 Announce Type: new Abstract: Legends are fundamental to chart understanding, as reliable interpretation requires correctly binding legend entries to corresponding visual marks. Whil

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

arXiv:2609.24161v1 Announce Type: cross Abstract: As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet under

MIS-Bench: Benchmarking Multimodal LLMs for Psychotherapeutic Interpersonal Skills Assessment

arXiv:2609.22778v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require expert

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

arXiv:2609.22285v1 Announce Type: new Abstract: Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the mode

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

arXiv:2609.22216v1 Announce Type: cross Abstract: Quantization enables deployment of large language models on resource-constrained clinical edge devices, but its effect on clinical accuracy and safety

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

arXiv:2609.22452v1 Announce Type: new Abstract: Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient inf

A benchmark dataset and baseline methods for four-dimensional STEM diffraction patterns

arXiv:2609.21593v1 Announce Type: new Abstract: Four-dimensional scanning transmission electron microscopy (4D-STEM) records a two-dimensional diffraction pattern at each electron-probe position, yiel

Benchmarking World Models for Continual Learning on Compositional Tasks

arXiv:2609.22055v1 Announce Type: new Abstract: A desirable property of a world model is the ability to learn continually across tasks, adapting to new environments without forgetting what the agent h

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

arXiv:2609.21722v1 Announce Type: cross Abstract: Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply roote

GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development

arXiv:2609.21293v1 Announce Type: new Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessaril

mu^2-Bench: A Multilingual Machine Unlearning Benchmark

arXiv:2609.20945v1 Announce Type: new Abstract: Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indi

Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

arXiv:2609.21392v1 Announce Type: new Abstract: Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

arXiv:2609.22069v1 Announce Type: new Abstract: Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omn

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

arXiv:2609.21465v1 Announce Type: cross Abstract: We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly

PhysioBench: A Unified Benchmark for Physiological Signal Question Answering

arXiv:2609.20836v1 Announce Type: new Abstract: Physiological signals support diverse clinical and monitoring tasks, yet existing physiological signal foundation models typically require task-specific

Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA

arXiv:2602.00279v2 Announce Type: replace Abstract: Reliable uncertainty quantification (UQ) is essential for safe deployment of large language models (LLMs) in scientific question answering, where lo

Benchmarking LLM Compliance with China AI Generated Content Regulations

arXiv:2609.19989v1 Announce Type: new Abstract: The widespread adoption of LLMs has led to escalating content compliance risks. Prior works have contributed to addressing these risks in the English co

Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understanding

arXiv:2609.19767v1 Announce Type: new Abstract: True machine intelligence requires transcending passive pixel registration to master top-down functional reasoning over absent information via visual ne

From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization

arXiv:2609.19630v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into vehicle voice assistants. But linking natural-language requests to vehicle functions creat

Multi-hour llama.cpp optimization experiments on Qwen MoE models, patches, benchmarks, and reproduction guides

model-releasesr-localllama

I had Astra run a multi-hour investigation into speeding up local MoE inference while keeping the model weights and quantization unchanged. The experiments covered Qwen3.6-35B-A3B and Qwen3.8 Flash-Ne

SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

arXiv:2609.20584v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows

SlugTrails: An Egocentric Benchmark for Floor Plan Localization in Large Buildings

arXiv:2609.19876v1 Announce Type: new Abstract: Floor-plan-based indoor visual localization enables infrastructure-free positioning, but most methods are developed and evaluated in small residential e

A3P5 NEMESIS Integrated Rover Design for Environmental Reconnaissance and Robotic Sampling with Reproducible Mobility Analysis and an External Data Machine Learning Calibration Benchmark

arXiv:2609.18245v1 Announce Type: new Abstract: A3P5 NEMESIS is a four-wheel rover intended to combine remote inspection, environmental observation and lightweight manipulation within one serviceable

DynoFluxBench: Benchmarking Kinodynamic Space-Time Planners in Dynamic Environments

arXiv:2609.18549v1 Announce Type: new Abstract: Robots that leave structured, static environments must plan motions that are kinodynamically feasible and safe among moving obstacles. However, there ar

ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts

arXiv:2609.18844v1 Announce Type: new Abstract: Multimodal coding agents are expected to turn visual inputs into usable artifacts, and they act through a harness, the layer of tools, context managemen

TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

arXiv:2609.18748v1 Announce Type: cross Abstract: Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based t

Safety

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

arXiv:2609.18585v1 Announce Type: cross Abstract: Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essenti

safetyarxiv-cs-lg

Research

Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

arXiv:2609.16023v1 Announce Type: new Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale

researcharxiv-cs-cl

16 Sept 2026

SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP

arXiv:2509.07801v5 Announce Type: replace Abstract: Structured information extraction from scientific literature is crucial for capturing core concepts and emerging trends in specialized fields. While

16 Sept 2026

The Neverwhere Visual Parkour Benchmark Suite

arXiv:2609.16443v1 Announce Type: cross Abstract: State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world pe

16 Sept 2026

A derivative-fidelity failure mode in physics-informed neural networks: strengthened benchmark evidence from function-value training

arXiv:2609.13171v1 Announce Type: new Abstract: Physics-informed neural networks (PINNs) use automatic differentiation to impose differential-equation residuals, but good agreement in function values

Accuracy Is Not Service: A Decision-Aware Benchmark for Intermittent-Demand Forecasting

arXiv:2609.13840v1 Announce Type: new Abstract: A contract-logistics spare-parts operator is paid on order-level service: an order counts only if every requested line is fulfilled, yet forecasters are

Decentralized Multi-Robot Task Allocation Under Degraded Communication: A Benchmark of Performance, Reliability, and Computation

arXiv:2609.13711v1 Announce Type: new Abstract: Selecting a decentralized Multi-Robot Task Allocation (MRTA) method for embedded deployment on autonomous platforms requires considering more than route

GroundBench: A Factorized, Counterfactual Benchmark for Locating VLM Affordance Failures

arXiv:2609.13308v1 Announce Type: new Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language mo

InterSocialBench: Benchmarking Human and LLM Preferences for Companion-Robot Social Behavior

arXiv:2609.15455v1 Announce Type: new Abstract: Companion robots face everyday situations in which several feasible behaviors may be appropriate, yet different people prefer different responses. We in

Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity

arXiv:2603.29396v2 Announce Type: replace Abstract: Standard evaluations of Large language models (LLMs) focus on task performance, offering limited insight into whether correct behavior reflects appr

MoveBench: A Benchmark for Global-Scale Wildlife Movement Forecasting

arXiv:2609.15780v1 Announce Type: new Abstract: Understanding and predicting wildlife movement is critical for ecology and conservation. While trajectory forecasting has advanced for human and vehicle

Agents

NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities

arXiv:2609.15695v1 Announce Type: new Abstract: Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains ch

agentsarxiv-cs-ai

Research

Time Series Analysis in Frequency Domain: A Survey of Open Challenges, Opportunities and Benchmarks

arXiv:2504.07099v5 Announce Type: replace-cross Abstract: Frequency-domain analysis has emerged as a powerful paradigm for time series analysis, offering unique advantages over traditional time-domain

researcharxiv-cs-lg

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

arXiv:2609.15939v1 Announce Type: cross Abstract: Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detec

What Makes a 3D Scene Editable? A Factorized Benchmark of Fidelity, Locality, Consistency, and Preservation

arXiv:2609.14899v1 Announce Type: new Abstract: Neural 3D scene editing is often evaluated by semantic alignment alone, although a convincing result may alter unrelated content or become inconsistent

Agents

Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks

arXiv:2609.11018v1 Announce Type: new Abstract: The term agent in artificial intelligence lacks a standard definition, complicating the evaluation, comparison, and reproducibility of AI agent research

agentsarxiv-cs-ai

12 Sept 2026

SynCo: Synthetic Community-Aware Attributed Graph Generator for Graph Neural Network Benchmarking

arXiv:2609.10742v1 Announce Type: cross Abstract: Graph Neural Networks (GNNs) are powerful models for handling attributed graphs in tasks such as classification, link prediction, and community detect

11 Sept 2026

Development and Validation of a Physics-Guided Machine Learning Extrapolation Framework Using a Classical Transient Diffusion Benchmark

arXiv:2609.09912v1 Announce Type: new Abstract: Machine learning models used in engineering are typically trained within limited operating ranges, yet reliable predictions are often required beyond th