Search: “benchmark”

arXiv:2609.28236v1 Announce Type: new Abstract: Long-horizon embodied interaction requires agents to retain and continually update information about the environment as they observe, act, and encounter

safetyarxiv-cs-cv

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

EvEMTBench: An Open Benchmark for Machine Learning in Power System Protection

DGX agent

arXiv:2609.28149v1 Announce Type: cross Abstract: Studies of machine-learning-based power system protection are difficult to compare because task definitions, measurement access, data partitions, metr

model-releasesarxiv-cs-lg

24 Sept 2026

MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos

arXiv:2609.28437v1 Announce Type: new Abstract: Online information is increasingly consumed in video format. Much of this comes in the form of *raw video*: continuous footage taken on a cell phone, wi

SafeTutors: Benchmarking Pedagogical Safety in AI Tutoring Systems

arXiv:2603.17373v2 Announce Type: replace Abstract: Large language models are rapidly being deployed as AI tutors, yet current evaluation paradigms assess problem-solving accuracy and generic safety i

WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents

arXiv:2510.01354v2 Announce Type: replace-cross Abstract: Multiple prompt injection attacks have been proposed against web agents. At the same time, various methods have been developed to detect gener

DUMA-Bench: A Dual-Control Multi-Agent Benchmark for Evaluating LLM Agent Security

arXiv:2609.24662v1 Announce Type: new Abstract: LLM-based agents increasingly operate in environments where they interact with users, tools, and external systems. Yet most security evaluations assume

Label-Efficient Learning for Ground-Based Sky-Image Classification: A Benchmark of Transfer Learning, Active Learning, and Pseudo-Labeling on GCD

arXiv:2609.26631v1 Announce Type: new Abstract: Accurate ground-based cloud classification is important for atmospheric monitoring, solar-energy forecasting, aviation weather assessment, and climate o

MatcherCompass: A Deployment-Aware Benchmark to Guide Image Matcher Selection in the Wild

arXiv:2609.25688v1 Announce Type: new Abstract: Field robots operating across time of day and sensing modalities require accurate image correspondences within onboard time and resource budgets. Howeve

Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development

arXiv:2609.25396v1 Announce Type: new Abstract: Parallel coding agents can produce patches that work alone but fail when merged. This happens when one agent changes an interface or rule that another a

Peerify: Benchmarking Peer-Review Claim Verification

arXiv:2609.25046v1 Announce Type: new Abstract: Peer review plays a central role in scholarly publishing, yet verifying whether reviewer claims are supported by manuscript evidence remains a largely m

Trains but Doesn't Learn: A Post-Training Delivery Benchmark for LLM Agents as Forward-Deployed Engineers

arXiv:2609.25237v1 Announce Type: cross Abstract: Post-training is becoming a service (PTaaS): a customer hands an operator data and a goal, and a forward-deployed engineer (FDE) returns a fine-tuned,

What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code

arXiv:2609.12708v2 Announce Type: replace-cross Abstract: AI coding assistants are becoming co-authors of production software, yet their evaluation centers on functional correctness, leaving open whet

WidgetVA: A Widget-Centric Framework and Benchmark for Agentic Visual Analytics

arXiv:2609.24094v1 Announce Type: cross Abstract: Visual analytics (VA) enables sensemaking through interactive visualization, but effective analysis often requires experts to translate high-level int

An Empirical Survey and Benchmark of Learned Distance Indexes for Road Networks

arXiv:2602.04068v2 Announce Type: replace Abstract: The calculation of shortest-path distances in road networks is a core operation in navigation systems, location-based services, and spatial analytic

ARCGym: Benchmarking Deep Reinforcement Learning in Autonomous Robotic Colonoscopy

arXiv:2609.22803v1 Announce Type: new Abstract: Simulations for learning-based autonomous colonoscopic navigation focus mainly on fully actuated capsule robots, failing to capture the contact-rich nav

Augmenting PID Control with Deep Reinforcement Learning: A Hybrid Approach to the Industrial Benchmark

arXiv:2609.22584v1 Announce Type: new Abstract: As industrial processes grow in complexity, traditional Proportional-Integral-Derivative (PID) controllers are often insufficient for handling their non

Can Coding Agents Reproduce Official Statistics? Metadata, Retry Budget and the Limits of Execution Feedback in a Controlled Eurostat Benchmark

arXiv:2609.22222v1 Announce Type: cross Abstract: Large language models can generate executable data-analysis code, but successful execution is not equivalent to a valid official-statistics result. Th

ECP-Bench: Benchmarking and Learning Entertainment Content Promotion with Foundation Models

arXiv:2609.22150v1 Announce Type: cross Abstract: Content promotion spans a broad set of skills, from understanding content to forecasting its market reception. However, LLMs' ability to support such

ETH-TraceBench: A Large-Scale Event-Stream Benchmark for Ethereum DeFi under Temporal, Protocol, and Contract Shift

arXiv:2609.23659v1 Announce Type: new Abstract: Ethereum decentralized finance (DeFi) provides a public, time-stamped record of transaction-level event streams, but the same public symbols can create

LegendBench: A Diagnostic Benchmark for Legend Understanding with Counterfactual Interventions

arXiv:2609.24172v1 Announce Type: new Abstract: Legends are fundamental to chart understanding, as reliable interpretation requires correctly binding legend entries to corresponding visual marks. Whil

MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents

arXiv:2609.24161v1 Announce Type: cross Abstract: As LLM agents increasingly interact with external tools through standardized protocols such as MCP, tool-interface design becomes a critical yet under

MIS-Bench: Benchmarking Multimodal LLMs for Psychotherapeutic Interpersonal Skills Assessment

arXiv:2609.22778v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are increasingly used as evaluators, yet their reliability in professional assessment tasks that require expert

ORDER: A Fictitious-World Benchmark for Domain-Adaptive Embodied AI

arXiv:2609.22285v1 Announce Type: new Abstract: Adapting language models to new domains via continual pre-training raises a basic evaluation problem: if the training corpus overlaps with what the mode

The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families

arXiv:2609.22216v1 Announce Type: cross Abstract: Quantization enables deployment of large language models on resource-constrained clinical edge devices, but its effect on clinical accuracy and safety

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

arXiv:2609.22452v1 Announce Type: new Abstract: Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient inf

A benchmark dataset and baseline methods for four-dimensional STEM diffraction patterns

arXiv:2609.21593v1 Announce Type: new Abstract: Four-dimensional scanning transmission electron microscopy (4D-STEM) records a two-dimensional diffraction pattern at each electron-probe position, yiel

Benchmarking World Models for Continual Learning on Compositional Tasks

arXiv:2609.22055v1 Announce Type: new Abstract: A desirable property of a world model is the ability to learn continually across tasks, adapting to new environments without forgetting what the agent h

CIBuzzBench: A Benchmark for Cross-Lingual Understanding of Chinese Internet Buzzwords

arXiv:2609.21722v1 Announce Type: cross Abstract: Chinese social media has generated a vast and continually evolving lexicon of internet buzzwords whose meanings are often non-literal and deeply roote

GameASG-Bench: Benchmarking Autonomous Software Generation for Game Development

arXiv:2609.21293v1 Announce Type: new Abstract: Autonomous software generation (ASG) aims to turn human requirements into executable applications, but delivering these applications does not necessaril

mu^2-Bench: A Multilingual Machine Unlearning Benchmark

arXiv:2609.20945v1 Announce Type: new Abstract: Undesired information such as harmful content and private data propagates through Multilingual Large Language Models (LLMs) via direct training and indi

Omni Demand Understanding: A Benchmark for Contextual User-Intent Inference in Multimodal Interaction

arXiv:2609.21392v1 Announce Type: new Abstract: Natural audio-visual interaction is emerging as an important interface for AI assistants, allowing users to communicate through speech and vision rather

OmniVBench: A Benchmark and Large-Scale Dataset for Omni Reference-to-Video Generation

arXiv:2609.22069v1 Announce Type: new Abstract: Reference-to-video (R2V) generation is evolving toward increasingly general and versatile reference control, giving rise to the emerging paradigm of omn

OmniVChat: Synthesizing, Benchmarking, and Training for Native Audio-Visual Dialogue

arXiv:2609.21465v1 Announce Type: cross Abstract: We define OmniVChat (Omni Video Chat) as the task of native audio-visual dialogue between a user and an omni model. In OmniVChat, omni models directly

PhysioBench: A Unified Benchmark for Physiological Signal Question Answering

arXiv:2609.20836v1 Announce Type: new Abstract: Physiological signals support diverse clinical and monitoring tasks, yet existing physiological signal foundation models typically require task-specific

Semantic Calibration Prevails Where Token Confidence Fails: Benchmarking Long-Form Scientific QA

arXiv:2602.00279v2 Announce Type: replace Abstract: Reliable uncertainty quantification (UQ) is essential for safe deployment of large language models (LLMs) in scientific question answering, where lo

Benchmarking LLM Compliance with China AI Generated Content Regulations

arXiv:2609.19989v1 Announce Type: new Abstract: The widespread adoption of LLMs has led to escalating content compliance risks. Prior works have contributed to addressing these risks in the English co

Benchmarking MLLMs via Cognitive Expected Scene Graph for Safety-Critical Visual Negation Understanding

arXiv:2609.19767v1 Announce Type: new Abstract: True machine intelligence requires transcending passive pixel registration to master top-down functional reasoning over absent information via visual ne

From Intent to Action: Benchmarking LLM Safety in Vehicle Voice Command Authorization

arXiv:2609.19630v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into vehicle voice assistants. But linking natural-language requests to vehicle functions creat

SAFARI: An Industrial Benchmark for LLM-Assisted Hazard Analysis and Risk Assessment

arXiv:2609.20584v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly considered for safety-critical engineering, yet their reliability in regulated functional-safety workflows

SlugTrails: An Egocentric Benchmark for Floor Plan Localization in Large Buildings

arXiv:2609.19876v1 Announce Type: new Abstract: Floor-plan-based indoor visual localization enables infrastructure-free positioning, but most methods are developed and evaluated in small residential e

A3P5 NEMESIS Integrated Rover Design for Environmental Reconnaissance and Robotic Sampling with Reproducible Mobility Analysis and an External Data Machine Learning Calibration Benchmark

arXiv:2609.18245v1 Announce Type: new Abstract: A3P5 NEMESIS is a four-wheel rover intended to combine remote inspection, environmental observation and lightweight manipulation within one serviceable

DynoFluxBench: Benchmarking Kinodynamic Space-Time Planners in Dynamic Environments

arXiv:2609.18549v1 Announce Type: new Abstract: Robots that leave structured, static environments must plan motions that are kinodynamically feasible and safe among moving obstacles. However, there ar

ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts

arXiv:2609.18844v1 Announce Type: new Abstract: Multimodal coding agents are expected to turn visual inputs into usable artifacts, and they act through a harness, the layer of tools, context managemen

TeleAntiFraud 2.0: A Refreshable, Profile-Grounded, and Audio-Based Benchmark for Telecom Fraud Detection

arXiv:2609.18748v1 Announce Type: cross Abstract: Telecom fraud scripts evolve rapidly and are often designed to resemble routine service conversations, creating two key requirements for audio-based t

Safety

TTM-Bench: A Framework for Text-to-Music System Performance Benchmarking

arXiv:2609.18585v1 Announce Type: cross Abstract: Text-to-music (TTM) systems are increasingly used to generate musical audio from natural-language descriptions. Robust evaluation is therefore essenti

safetyarxiv-cs-lg

Research

Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

arXiv:2609.16023v1 Announce Type: new Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale

researcharxiv-cs-cl

16 Sept 2026

SciNLP: A Domain-Specific Benchmark for Full-Text Scientific Entity and Relation Extraction in NLP

arXiv:2509.07801v5 Announce Type: replace Abstract: Structured information extraction from scientific literature is crucial for capturing core concepts and emerging trends in specialized fields. While

16 Sept 2026

The Neverwhere Visual Parkour Benchmark Suite

arXiv:2609.16443v1 Announce Type: cross Abstract: State-of-the-art visual locomotion controllers are increasingly capable at handling complex visual environments, making evaluating their real-world pe