Search: “benchmark”

arXiv:2610.00648v1 Announce Type: new Abstract: AI coding agents are ubiquitous in engineering workflows amongst industry and academia. Yet, despite their use in app coding, relatively less attention

arXiv:2610.01819v1 Announce Type: new Abstract: Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism d

arXiv:2610.00314v1 Announce Type: new Abstract: Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while v

arXiv:2610.02205v1 Announce Type: new Abstract: Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However

arXiv:2610.00084v1 Announce Type: new Abstract: Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific A

arXiv:2609.38948v1 Announce Type: cross Abstract: Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present

arXiv:2609.39378v1 Announce Type: new Abstract: Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving

arXiv:2609.39420v1 Announce Type: new Abstract: Large language models are strong general-purpose code generators, but executable algorithmic trading remains a demanding specialization target: a model

arXiv:2607.10891v2 Announce Type: replace Abstract: Large language models (LLMs) are rapidly shifting toward agents that solve tasks through diverse interfaces, including web and graphical user interf

arXiv:2609.38269v1 Announce Type: cross Abstract: Coding agents are increasingly asked to build software rather than patch it, yet benchmarks for from-scratch repository construction are mostly limite

arXiv:2609.36556v1 Announce Type: new Abstract: Recent studies report that LLM-based multi-agent systems (MAS) fail at rates of 41%-87%, yet to our knowledge, no benchmark to date supports systematic

arXiv:2609.36686v1 Announce Type: new Abstract: Identifying the root cause of an anomaly among hundreds of sensors is critical for preventing safety incidents and costly downtime in complex monitored

arXiv:2609.28560v1 Announce Type: cross Abstract: Evaluating a stochastic large language model is costly: benchmark scores estimate expected performance from randomized rollouts, yet uniform repetitio

arXiv:2609.28610v1 Announce Type: new Abstract: Benchmarking is an increasingly critical part of research in machine learning and the domains where it is applied, including healthcare. Yet, despite th

arXiv:2609.27867v1 Announce Type: new Abstract: A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24

arXiv:2609.25352v1 Announce Type: cross Abstract: Evaluation of large language models (LLMs) for safety, security, and privacy (SSP) relies heavily on static benchmarks, which suffer from score satura

arXiv:2609.24971v1 Announce Type: new Abstract: Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are bui

arXiv:2603.02540v2 Announce Type: replace Abstract: Large language models (LLMs) display a unified 'general factor' of capability across 10 benchmarks (a finding confirmed by our factor analysis of 15

arXiv:2609.19607v1 Announce Type: cross Abstract: Coding-agent benchmarks are built for broad and comprehensive comparisons, not frequent development decisions. Individual runs vary, full suites are e

arXiv:2609.18270v1 Announce Type: new Abstract: Payment operations are a critical financial infrastructure, but the value of large language models in this domain remains unclear because payment rules

arXiv:2609.16610v1 Announce Type: new Abstract: Zero-shot waypoint navigation requires vision-language models to select, from the current first-person observation, a sequence of spatial actions that i

arXiv:2609.16919v1 Announce Type: new Abstract: This paper introduces NeuroSymbEAD, a large-scale neuro-symbolic caption dataset featuring an ego-centric knowledge graph (KG) of static and dynamic obj

arXiv:2609.13556v1 Announce Type: cross Abstract: Large Language Models (LLMs) have shown remarkable proficiency on general-purpose tasks, yet their performance often degrades in highly-specialized te

arXiv:2609.15571v1 Announce Type: new Abstract: As deep learning models become fundamental to modern healthcare, the 'Right to be Forgotten' mandated by privacy regulations like GDPR and HIPAA necessi

arXiv:2609.13283v1 Announce Type: cross Abstract: We present a multimodal foundation model for lunar remote sensing, pretrained from scratch on SomBench, a geographically partitioned corpus of nearly

arXiv:2609.14400v1 Announce Type: new Abstract: Agent benchmarks evaluate policy compliance but assume each policy determines a unique correct action. Natural-language policies can violate this assump

arXiv:2609.14118v1 Announce Type: cross Abstract: Online understanding of who is talking to the camera wearer is a key capability for egocentric social interaction. However, existing talk-to-me (TTM)

arXiv:2609.13005v1 Announce Type: new Abstract: Large language models (LLMs) have achieved strong performance on a wide range of natural language tasks, and recent benchmarks suggest that they are inc

arXiv:2609.10892v1 Announce Type: cross Abstract: When an indirect prompt injection succeeds against an LLM agent, the compromise is visible in the agent's own behavior: a benign prefix of tool calls,

arXiv:2609.11900v1 Announce Type: cross Abstract: Spatial reasoning depends not only on metric properties such as distance, angle, and shape, but also on topological relations that remain invariant un

arXiv:2609.11244v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) have achieved remarkable progress across diverse tasks, they suffer from hallucinations where generated o

arXiv:2609.11117v1 Announce Type: new Abstract: Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verificatio

arXiv:2609.08966v1 Announce Type: new Abstract: Language-model checkpoints are commonly selected by pretraining loss or benchmark scores, assuming that the highest-scoring checkpoint will remain the b

arXiv:2609.10016v1 Announce Type: cross Abstract: We introduce MetroLLM-Bench, a 955-case benchmark for testing language models as the policy layer of a transit kiosk. It covers six real metro systems

arXiv:2609.05762v1 Announce Type: new Abstract: UAV image collections contain spatially and temporally related frames, yet semantic-segmentation benchmarks commonly split them at image level. Such spl

arXiv:2609.06107v1 Announce Type: new Abstract: Data policies for reinforcement learning with verifiable rewards (RLVR) determine which rollouts are used, how strongly they are weighted, and which dom

arXiv:2609.08316v1 Announce Type: new Abstract: Multi-modal large language models (MLLMs) have demonstrated significant potential in image quality assessment (IQA) by bridging visual perception with d

arXiv:2609.08130v1 Announce Type: new Abstract: Driver alerting from dashcam video requires sequential decision-making under partial observability: a system must decide not only whether a scene is ris

arXiv:2609.07373v1 Announce Type: new Abstract: Modern Visual Place Recognition (VPR) methods excel on standard benchmarks yet remain brittle in feature-poor environments. By treating each query image

arXiv:2503.04666v2 Announce Type: replace Abstract: High-quality benchmarks are crucial for driving progress in machine learning research. However, despite the growing interest in video generation, th

arXiv:2609.05178v1 Announce Type: new Abstract: Vision-Language-Action (VLA) or World Action (WAM) models have recently demonstrated remarkable performance in robotic manipulation. On LIBERO, SOTA met

arXiv:2609.05381v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated on molecular property benchmarks, but accuracy cannot distinguish a model that predicts a proper

arXiv:2609.04550v1 Announce Type: new Abstract: Video-language benchmarks are usually constructed by the dataset authors without published reliability statistics, leaving the noise floor of the constr

arXiv:2609.03526v1 Announce Type: new Abstract: Multimodal language models achieve near-ceiling scores on food recognition benchmarks, yet it remains unclear whether this success reflects genuine cult

arXiv:2510.17460v2 Announce Type: replace Abstract: Idioms remain a persistent challenge in natural language processing due to their figurative and culturally grounded meanings, which distinguish them

arXiv:2609.03781v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used in multilingual settings, yet their safety is still evaluated primarily in English. This limits our