Search: “benchmark”

ZeroGAR: Benchmarking the Adversarial Robustness of Zero-Shot Graph Models

arXiv:2609.33314v1 Announce Type: new Abstract: Zero-shot graph models (ZGMs), which learn transferable knowledge from source graphs and directly apply to unseen target graphs without any adaptation,

29 Sept 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

A Benchmarking Framework for Context-aware XR Interfaces

DGX agent

arXiv:2609.30466v1 Announce Type: cross Abstract: Everyday Extended Reality (XR) systems aim to provide context-aware access to the right functionalities at the right time and place, with minimal manu

researcharxiv-cs-ai

28 Sept 2026

Research

Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model

arXiv:2609.30454v1 Announce Type: new Abstract: Non-generative 'System-1' models return structured probabilistic decisions in a single forward pass, without autoregressive decoding, at a small fractio

researcharxiv-cs-lg

Benchmarking the Connectomes of Caenorhabditis elegans within the Reservoir Computing Framework

arXiv:2609.30508v1 Announce Type: new Abstract: The aim of this work is to examine the connectomes of Caenorhabditis elegans through a computational lens using the reservoir computing framework. Conne

JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

arXiv:2609.31142v1 Announce Type: cross Abstract: Models trained with reinforcement learning for calibrated decisions (RLCD), such as Jev, answer a typed question about an input, the state, with a pro

RefRef: A Dataset and Benchmark for Reconstructing Refractive and Reflective Objects

arXiv:2505.05848v3 Announce Type: replace Abstract: Modern 3D reconstruction and novel view synthesis approaches have demonstrated strong performance on scenes with opaque, non-refractive objects. How

UltraG-Bench: A Multi-task Benchmark for assessing Large Vision-Language Models on Pixel-level Evidence Grounding in Ultrasound

arXiv:2609.30928v1 Announce Type: cross Abstract: Ultrasound is one of the most widely used medical imaging modalities, and recent large vision-language models(VLMs) have shown increasing capabilities

Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languages

arXiv:2609.29798v1 Announce Type: cross Abstract: This paper presents an end-to-end study of automatic speech recognition (ASR) for adolescent health communication in three Ghanaian languages (Twi, Da

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

arXiv:2609.28826v1 Announce Type: new Abstract: Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evalua

Delay-of-Gratification as a Multi-Agent Survival Micro-benchmark for Long-Horizon LLMs: Social Exposure, Personas, and Tool Use Budgets

arXiv:2609.29509v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as multi-turn agents that must sustain goals, use tools, and adapt to other agents over extended

EnSiTa - A Trilingual Multi-Domain Parallel Dataset and Benchmark for Domain-Specific Machine Translation

arXiv:2609.29511v1 Announce Type: new Abstract: Machine Translation (MT) for low-resource languages remains far behind that of high-resource languages, and the gap is widest in specialised domains, wh

GeoRefer-Bench: A Benchmark from Referring Pixels to Verifiable Geospatial Reasoning

arXiv:2609.29541v1 Announce Type: cross Abstract: Referring segmentation in overhead imagery is inherently relational: a query may ask for the buildings north of the road or the pond closest to a resi

IDRBench: Benchmarking the Interactive Capabilities of Deep Research Agents

arXiv:2601.06676v3 Announce Type: replace-cross Abstract: Large Language Model (LLM)-based deep research agents perform multi-step reasoning, web exploration, and long-form report generation. In these

Limited Structural Reliability in Public Educational Prediction Benchmarks: A Four-Dimension Audit of Seven Datasets

arXiv:2609.29625v1 Announce Type: new Abstract: Across seven public educational prediction datasets, three passed all four pre-modeling reliability checks; the remaining four either failed group-aware

OptiSAR-Net++: A Large-Scale Benchmark and Transformer-Free Framework for Cross-Domain Remote Sensing Visual Grounding

arXiv:2603.24876v2 Announce Type: replace Abstract: Remote sensing visual grounding (RSVG) aims to localize specific targets in remote sensing images using natural language expressions. However, exist

PFArena: Benchmarking Language Models for Protein Modification

arXiv:2609.28921v1 Announce Type: new Abstract: Protein modification requires navigating an immense sequence space, yet wet-lab validation remains low-throughput and costly. Although computational par

STRAND: Benchmarking and Improving Object-Centric Spatio-Temporal Monitoring in Video Large Language Models

arXiv:2609.29607v1 Announce Type: cross Abstract: While multimodal large language models (MLLMs) have advanced video understanding, they remain highly prone to hallucinations in dynamic scenes. We arg

A Systematic Benchmark of Explainable Methods for Temporal Attribution in Sequential Recommendation Systems

arXiv:2609.27201v1 Announce Type: new Abstract: Sequential RecSys are central to modern personalization, exploiting user's historical interaction sequences to drive next-step decisions. Deep learning

Beyond Balanced Accuracy: A Resolution and Parity-Controlled Benchmark for Vision-Language and Vision-Only Defect Assessment in UAV Power-Line Inspection

arXiv:2609.27457v1 Announce Type: new Abstract: Vision-language models (VLMs) are often reported to outperform task-specific vision backbones for unmanned aerial vehicle (UAV) power-line defect assess

Does Graph Structure Earn Its Place in Microservice Root-Cause Analysis? A Controlled Study on RCAEval, and What the Benchmark Was Really Measuring

arXiv:2609.27069v1 Announce Type: cross Abstract: Graph neural networks dominate recent work on microservice root-cause analysis, yet recent results question whether the graph contributes. Those resul

Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach

arXiv:2609.27205v1 Announce Type: cross Abstract: Text-to-speech systems increasingly process user-generated text (UGT) such as ppl and imo, whose pronunciation must be inferred from the canonical rat

Agents

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

arXiv:2609.27490v1 Announce Type: new Abstract: AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding,

agentsarxiv-cs-ai

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

arXiv:2609.25192v1 Announce Type: new Abstract: Financial search is a highly demanding task for LLM agents, requiring not only a correct final answer but also temporally valid information retrieval, a

FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

arXiv:2609.24002v1 Announce Type: new Abstract: Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specifie

FireWorldBench: Benchmarking Complex Physical World Intelligence through Coupled-Field Fire Dynamics

arXiv:2609.23064v1 Announce Type: new Abstract: Understanding the physical world requires more than object recognition, scene description, and short-term visual prediction, as real-world physical syst

MCPGen: Benchmarking LLMs on Executable MCPWorkflow Development

arXiv:2609.23925v1 Announce Type: cross Abstract: We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, a

PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI

arXiv:2609.23695v1 Announce Type: new Abstract: Recent advances in Physical AI have accelerated the use of foundation models in autonomous systems such as unmanned aerial vehicles (UAVs), which must p

SpecialEduBench: Benchmarking Vision-Language Models on Knowledge, Skill, and Attitude in Language Intervention for Autistic Children

arXiv:2609.26090v1 Announce Type: new Abstract: Language is the target of most early intervention for autistic children. Because the goal and the method change from child to child, the work falls to a

TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents

arXiv:2609.23363v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have led to the emergence of coding agents capable of performing complex engineering tasks, including re

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

arXiv:2609.24677v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone canno

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

arXiv:2511.18921v2 Announce Type: replace Abstract: Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously act

CCTU: A Benchmark for Tool Use under Complex Constraints

arXiv:2603.15309v2 Announce Type: replace Abstract: Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LL

CytoCrowd: A Multi-Annotator Benchmark Dataset for Cytology Image Analysis

arXiv:2602.06674v2 Announce Type: replace Abstract: High-quality annotated datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most datasets

H2RBench: A Real-to-Sim Benchmark for Evaluating Human-to-Robot Transfer

model-releasesarxiv-cs-ro

arXiv:2609.24778v1 Announce Type: new Abstract: Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing diff

MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents

arXiv:2609.24259v1 Announce Type: new Abstract: The effectiveness of agent memory ultimately depends on whether the underlying LLM gives each memory in context an appropriate degree of influence over

MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairs

arXiv:2609.23336v1 Announce Type: new Abstract: Localizing and describing fine-grained differences between near-identical images is a critical yet underexplored capability for multimodal large languag

Mind the Gaps: A Curated Benchmark for Form Field Detection

arXiv:2609.23679v1 Announce Type: new Abstract: Form Field Detection (FFD) is a fundamental component of document understanding systems, enabling applications ranging from large-scale industrial digit

Omni2Web: Benchmarking Audiovisual Website Development

arXiv:2609.23417v1 Announce Type: cross Abstract: Screen-recorded web editing requests contain weak deictic expressions such as ``this'' and ``there,'' whose referents depend on speech, cursor traject

Rethinking Class Imbalance for Single-Cell Foundation Models: A Systematic Benchmark Across Architectures and Long-Tail Loss Functions

arXiv:2609.23325v1 Announce Type: new Abstract: Single-cell foundation models (scGPT, scBERT, Geneformer) achieve cell-type classification accuracy up to 97.5% in our experiments, yet this aggregate a

When Is Availability-Aware Training Worth It? A Benchmark and Empirical Study of Interruption-Resilient Optimization Under Predictable Compute Schedules

arXiv:2609.22087v1 Announce Type: cross Abstract: Training under non-stationary but predictable compute availability (satellites under eclipse, duty-cycled edge devices, power-capped datacenters) is o

Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluation

arXiv:2509.22754v2 Announce Type: replace-cross Abstract: Autonomous driving remains a highly active research domain that seeks to enable vehicles to perceive dynamic environments, predict the future

Chinese Competitive Debating Dataset and Benchmark

arXiv:2609.21637v1 Announce Type: cross Abstract: Debate adjudication requires tracking how arguments develop through interaction, yet existing datasets rarely combine fine-grained debate transcripts

DPed-VLN: A Benchmark for Socially Compliant Vision-and-Language Navigation in Dynamic Pedestrian Environments

model-releasesarxiv-cs-ro

arXiv:2609.21504v1 Announce Type: new Abstract: Vision-and-language navigation (VLN) has advanced rapidly in static indoor environments, but robots operating in human-populated spaces must ground lang

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

arXiv:2609.20850v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easil

OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

arXiv:2609.21527v1 Announce Type: new Abstract: Graph-enhanced multi-agent systems (G-MAS) coordinate large language model agents through communication graphs and role assignments, which determine how

PlaceReasoner-Beta: Reasoning-Driven Macro Placement and Benchmarking

arXiv:2609.21263v1 Announce Type: new Abstract: Automated macro placement remains a fundamental challenge in VLSI physical design. Despite decades of research, existing approaches predominantly optimi

PRISM-BN: A Controlled Corpus and Benchmark for Text-to-Parameterized Bayesian Network Extraction

arXiv:2609.21673v1 Announce Type: new Abstract: Probabilistic Graphical Models (PGMs), especially Bayesian Networks (BNs), expose directed structure and probabilistic parameters, making them natural s

VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration

arXiv:2609.21521v1 Announce Type: cross Abstract: While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understandi