Search: “benchmark”

Phonemizing User-Generated Text: A Benchmark, Taxonomy, and Compositional Approach

arXiv:2609.27205v1 Announce Type: cross Abstract: Text-to-speech systems increasingly process user-generated text (UGT) such as ppl and imo, whose pronunciation must be inferred from the canonical rat

24 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents

DGX agent

arXiv:2609.27490v1 Announce Type: new Abstract: AI research agents need reliable knowledge of how their experiments change outcomes. We introduce WhatWorkedBench to measure experimental understanding,

agentsarxiv-cs-ai

24 Sept 2026

FinFIRST: Benchmarking Search Agents for Financial Information Retrieval, Sourcing and Traceability

arXiv:2609.25192v1 Announce Type: new Abstract: Financial search is a highly demanding task for LLM agents, requiring not only a correct final answer but also temporally valid information retrieval, a

FinInteract: Benchmarking Clarification and Intent Integration in Ambiguous Financial Question Answering

arXiv:2609.24002v1 Announce Type: new Abstract: Large language model agents increasingly answer financial questions by searching regulatory filings. Such questions are often deceptively under-specifie

FireWorldBench: Benchmarking Complex Physical World Intelligence through Coupled-Field Fire Dynamics

arXiv:2609.23064v1 Announce Type: new Abstract: Understanding the physical world requires more than object recognition, scene description, and short-term visual prediction, as real-world physical syst

MCPGen: Benchmarking LLMs on Executable MCPWorkflow Development

arXiv:2609.23925v1 Announce Type: cross Abstract: We study whether LLMs can produce executable workflow artifacts that remain consistent across graph structure, tool implementation, schema bindings, a

PhysAI-Bench: A Benchmark for LLM-Based Agentic Decision-Making in Autonomous UAV-Centric Physical AI

arXiv:2609.23695v1 Announce Type: new Abstract: Recent advances in Physical AI have accelerated the use of foundation models in autonomous systems such as unmanned aerial vehicles (UAVs), which must p

SpecialEduBench: Benchmarking Vision-Language Models on Knowledge, Skill, and Attitude in Language Intervention for Autistic Children

arXiv:2609.26090v1 Announce Type: new Abstract: Language is the target of most early intervention for autistic children. Because the goal and the method change from child to child, the work falls to a

TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents

arXiv:2609.23363v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have led to the emergence of coding agents capable of performing complex engineering tasks, including re

TimeLitmus: A Diagnostic Benchmark for Cross-Modal Understanding and Explanation Faithfulness in Event-Conditioned Time-Series Prediction

arXiv:2609.24677v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to make predictions from numerical time-series histories and textual events. Yet accuracy alone canno

BackdoorVLM: A Benchmark for Backdoor Attacks and Defenses on Vision-Language Models

arXiv:2511.18921v2 Announce Type: replace Abstract: Backdoor attacks undermine the reliability and trustworthiness of machine learning systems by injecting hidden behaviors that can be maliciously act

CCTU: A Benchmark for Tool Use under Complex Constraints

arXiv:2603.15309v2 Announce Type: replace Abstract: Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LL

CytoCrowd: A Multi-Annotator Benchmark Dataset for Cytology Image Analysis

arXiv:2602.06674v2 Announce Type: replace Abstract: High-quality annotated datasets are crucial for advancing machine learning in medical image analysis. However, a critical gap exists: most datasets

H2RBench: A Real-to-Sim Benchmark for Evaluating Human-to-Robot Transfer

arXiv:2609.24778v1 Announce Type: new Abstract: Learning robot manipulation policies from human video demonstrations constitutes a promising avenue for scalable robot learning. However, comparing diff

MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents

arXiv:2609.24259v1 Announce Type: new Abstract: The effectiveness of agent memory ultimately depends on whether the underlying LLM gives each memory in context an appropriate degree of influence over

MinCU: A Fine-Grained Benchmark for Grounded Minimal-Change Understanding in Image Pairs

arXiv:2609.23336v1 Announce Type: new Abstract: Localizing and describing fine-grained differences between near-identical images is a critical yet underexplored capability for multimodal large languag

Mind the Gaps: A Curated Benchmark for Form Field Detection

arXiv:2609.23679v1 Announce Type: new Abstract: Form Field Detection (FFD) is a fundamental component of document understanding systems, enabling applications ranging from large-scale industrial digit

Omni2Web: Benchmarking Audiovisual Website Development

arXiv:2609.23417v1 Announce Type: cross Abstract: Screen-recorded web editing requests contain weak deictic expressions such as ``this'' and ``there,'' whose referents depend on speech, cursor traject

Rethinking Class Imbalance for Single-Cell Foundation Models: A Systematic Benchmark Across Architectures and Long-Tail Loss Functions

arXiv:2609.23325v1 Announce Type: new Abstract: Single-cell foundation models (scGPT, scBERT, Geneformer) achieve cell-type classification accuracy up to 97.5% in our experiments, yet this aggregate a

When Is Availability-Aware Training Worth It? A Benchmark and Empirical Study of Interruption-Resilient Optimization Under Predictable Compute Schedules

arXiv:2609.22087v1 Announce Type: cross Abstract: Training under non-stationary but predictable compute availability (satellites under eclipse, duty-cycled edge devices, power-capped datacenters) is o

Benchmarking Autonomous Driving Planners Across Leaderboards: A Unified CARLA-Based Evaluation

arXiv:2509.22754v2 Announce Type: replace-cross Abstract: Autonomous driving remains a highly active research domain that seeks to enable vehicles to perceive dynamic environments, predict the future

Chinese Competitive Debating Dataset and Benchmark

arXiv:2609.21637v1 Announce Type: cross Abstract: Debate adjudication requires tracking how arguments develop through interaction, yet existing datasets rarely combine fine-grained debate transcripts

DPed-VLN: A Benchmark for Socially Compliant Vision-and-Language Navigation in Dynamic Pedestrian Environments

arXiv:2609.21504v1 Announce Type: new Abstract: Vision-and-language navigation (VLN) has advanced rapidly in static indoor environments, but robots operating in human-populated spaces must ground lang

MME-Safety: A Fine-grained Benchmark for Safety Evaluation of MLLMs

arXiv:2609.20850v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) show remarkable advancements, their cross-modal capabilities introduce complex vulnerabilities that easil

OpenMAS-GCom. A Diagnostic Benchmark for Graph-enhanced Multi-Agent Systems

arXiv:2609.21527v1 Announce Type: new Abstract: Graph-enhanced multi-agent systems (G-MAS) coordinate large language model agents through communication graphs and role assignments, which determine how

PlaceReasoner-Beta: Reasoning-Driven Macro Placement and Benchmarking

arXiv:2609.21263v1 Announce Type: new Abstract: Automated macro placement remains a fundamental challenge in VLSI physical design. Despite decades of research, existing approaches predominantly optimi

PRISM-BN: A Controlled Corpus and Benchmark for Text-to-Parameterized Bayesian Network Extraction

arXiv:2609.21673v1 Announce Type: new Abstract: Probabilistic Graphical Models (PGMs), especially Bayesian Networks (BNs), expose directed structure and probabilistic parameters, making them natural s

VidOmni-Bench: A Benchmark for Fine-Grained Video Understanding via Spatio-Temporal Event Verification across Complexity and Duration

arXiv:2609.21521v1 Announce Type: cross Abstract: While Video Large Language Models (Video-LLMs) have recently demonstrated strong performance, reliably evaluating their fine-grained video understandi

BioPhys-Bridge: A Benchmark for Interdisciplinary Scientific Reasoning in Physics-Grounded Biological Research

arXiv:2609.19180v1 Announce Type: new Abstract: Language models face unique challenges in analyzing interdisciplinary scientific research literature. In biophysics research, faithful answers require g

18 Sept 2026

BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software

arXiv:2509.25248v2 Announce Type: replace-cross Abstract: Automatically compiling open-source software (OSS) projects is a vital, labor-intensive, and complex task, which makes it a good challenge for

18 Sept 2026

Benchmarking Visual-Inertial Odometry in Subterranean Environments Under Sensor Degradation, Miscalibration, and Dynamic Occlusion

arXiv:2609.18628v1 Announce Type: new Abstract: Visual-inertial odometry (VIO) is a core capability for autonomous operation in GPS-denied subterranean environments, yet its reliability can degrade sh

17 Sept 2026

iMINDBench: iEEG Multi-Institution Neural Decoding Benchmark

arXiv:2609.18104v1 Announce Type: new Abstract: Intracranial electroencephalography (iEEG) is widely used to record electrical activity directly from electrodes inside the human brain, making it an at

17 Sept 2026

BRAVE-6D: Benchmark for Robotic Active Vision in 6DOF Pose Estimation

arXiv:2609.17106v1 Announce Type: new Abstract: Detecting and grasping small objects remains a significant challenge in robotics. Active vision, where the robot moves closer to the object, is an intui

16 Sept 2026

What Breaks Local Watermarks? A Robustness Benchmark for Local Invisible Image Watermarking

arXiv:2609.16832v1 Announce Type: cross Abstract: Local image watermarking embeds an invisible signal into selected image regions rather than spreading it across the entire image, enabling payload rec

16 Sept 2026

Bangla Sentence Function Classification: Corpus Development, Model Benchmarking, and Interpretability

arXiv:2609.13869v1 Announce Type: cross Abstract: Automatic sentence function identification is important for many downstream natural language processing (NLP) applications such as dialogue systems, t

Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration

arXiv:2609.15669v1 Announce Type: cross Abstract: Multimodal image registration is a key component of many clinical workflows, yet it remains challenging because corresponding anatomical structures of

Benchmarking Optimizers to Solve Inverse Problems with Differentiable Physics Simulators

arXiv:2609.13819v1 Announce Type: new Abstract: Solving inverse problems with differentiable physics simulators holds the potential to revolutionize scientific discovery and engineering design, as it

EMCStereo: Attention-Enhanced Stereo Matching for Thin-Structure Depth Estimation with a Synthetic Tree-Branch Benchmark

arXiv:2609.13233v1 Announce Type: new Abstract: Thin structures such as tree branches are among the hardest cases for stereo matching: a branch is only a few pixels wide, the background is cluttered,

Research

GP-VMimesSMA: Benchmarking General-Purpose Vision Models and Specialized Architectures for 2D Medical Image Segmentation

arXiv:2603.13044v2 Announce Type: replace-cross Abstract: Medical image segmentation (MIS) is a fundamental component of computer-assisted diagnosis and clinical decision support. Over the past decade

researcharxiv-cs-ai

K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations

arXiv:2609.15855v1 Announce Type: cross Abstract: % !TEX root = ../main.tex People increasingly use large language models (LLMs) for mental health support, yet their safety in evolving, high-risk conv

SomBench: Benchmark Dataset for Advancing Machine Learning in Lunar Science

arXiv:2609.13277v1 Announce Type: new Abstract: Lunar orbital missions, such as Lunar Reconnaissance Orbiter, Kaguya/SELENE, Gravity Recovery and Interior Laboratory, and Lunar Prospector, among other

The MODA General Attribute Suite: A Four-Track Evaluation Benchmark for Fashion Attribute Extraction

arXiv:2609.13279v1 Announce Type: new Abstract: Fashion attribute extraction is evaluated inconsistently: results are reported as single aggregate numbers across image types that pose different proble

Toward Secure Code Generation: Bridging Correctness and Security via Task-Adaptive Vulnerability Modeling and Execution-Based Benchmarking

arXiv:2407.02395v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used for program synthesis, yet they often generate code that is functionally plausible but inse

Statistical Uncertainty Quantification for Aggregate Performance Metrics in Machine Learning Benchmarks

arXiv:2501.04234v2 Announce Type: replace-cross Abstract: Modern artificial intelligence is supported by machine learning models (e.g., foundation models) that are pretrained on a massive data corpus

14 Sept 2026

Agents

UrduFactCheck: An Agentic Fact-Checking Framework for Urdu with Evidence Boosting and Benchmarking

arXiv:2505.15063v3 Announce Type: replace Abstract: The rapid adoption of Large Language Models (LLMs) has raised important concerns about the factual reliability of their outputs, particularly in low

agentsarxiv-cs-cl

14 Sept 2026

Applications

Beyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions

arXiv:2609.11126v1 Announce Type: new Abstract: Verification and validation (V&V) of classification models is crucial to enable a wide range of sensor processing applications. Currently, the V&V proce

applicationsarxiv-cs-cv

11 Sept 2026

CMNIE: An Information Extraction Benchmark for Chinese Military News

arXiv:2609.10722v1 Announce Type: new Abstract: Structured extraction from Chinese military news supports intelligence analysis, decision-making, and knowledge base construction. However, existing res

11 Sept 2026

ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

arXiv:2609.10895v1 Announce Type: new Abstract: Reacting to sudden physical hazards (catching a slipping plate, dodging a falling knife) is both a meaningful test of embodied intelligence and a hard r