Search: “benchmark”

arXiv:2604.08362v1 Announce Type: new Abstract: The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain co

arXiv:2610.21109v1 Announce Type: new Abstract: When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place gr

arXiv:2610.22372v1 Announce Type: cross Abstract: Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic

arXiv:2610.19704v1 Announce Type: new Abstract: Large language models (LLMs) increasingly help users arrange appointments, plan journeys, and allocate limited resources. These tasks require models to

arXiv:2610.16191v1 Announce Type: new Abstract: A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark th

arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily e

arXiv:2610.19185v1 Announce Type: new Abstract: Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them.

arXiv:2610.16782v1 Announce Type: new Abstract: Industrial manipulation in flexible manufacturing, e.g., high-mix, low-volume (HMLV) electronics assembly, requires robot policies to adapt to changing

arXiv:2610.06170v2 Announce Type: replace Abstract: Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple

arXiv:2610.17728v1 Announce Type: new Abstract: Egocentric multimodal assistants see through the user's eyes and guide what the user does next, so their outputs directly shape physical actions, includ

arXiv:2606.20235v2 Announce Type: replace-cross Abstract: Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, i

arXiv:2610.19063v1 Announce Type: new Abstract: As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requir

arXiv:2610.18305v1 Announce Type: new Abstract: The world encompasses both physical environments and human actors. Predicting its evolution requires understanding both physical laws and how emotions s

arXiv:2610.17573v1 Announce Type: new Abstract: Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fine-

arXiv:2610.18844v1 Announce Type: new Abstract: Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently

arXiv:2610.13947v2 Announce Type: replace Abstract: In this paper, we study physically grounded articulated 3D asset generation---moving the goal from reproducing how objects look to capturing how the

arXiv:2601.17645v2 Announce Type: replace-cross Abstract: Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examin

arXiv:2610.07939v1 Announce Type: new Abstract: Due to rapid advances in Generative AI, commercial video generation tools can be used to produce fabricated surveillance footage that can fool both huma

arXiv:2610.08106v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) advance rapidly, while conventional benchmark development lags behind, delaying investigation of newly observed

arXiv:2610.08571v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) systems are vulnerable to prompt-injection attacks embedded in retrieved content. We introduce RAG-PIBench, a ben

arXiv:2610.04307v1 Announce Type: cross Abstract: Machine Learning (ML) methods promise a fast solution process for Optimal Power Flow (OPF). While inconsistent test cases, implementations, and evalua

arXiv:2407.18137v2 Announce Type: replace Abstract: Small object detection and tracking in videos remain critical yet underexplored challenges in computer vision, particularly for applications such as

arXiv:2610.03130v1 Announce Type: cross Abstract: Biological literature retrieval systems are often developed and evaluated using broad biomedical corpora and general-purpose search tasks. However, ma

arXiv:2602.03255v2 Announce Type: replace Abstract: Computer-use agents (CUAs) execute multi-stage tasks through tools, where an early unsafe decision can propagate to consequential actions. Evaluatin

arXiv:2610.02563v1 Announce Type: cross Abstract: We present OpenGameEval, a benchmark and evaluation framework for agentic game development inside Roblox Studio. It runs language models as agents in

arXiv:2610.00969v1 Announce Type: cross Abstract: Large language models (LLMs) have been increasingly used for financial document analysis, including earnings call transcripts (ECTs). Beyond generatin

arXiv:2610.00728v1 Announce Type: cross Abstract: Weather reanalysis products rely on computationally intensive numerical weather predictions followed by data assimilation that corrects the forecast t

arXiv:2505.17613v2 Announce Type: replace Abstract: Automatically evaluating multimodal generation presents a significant challenge, as automated metrics often struggle to align with human evaluation,

arXiv:2610.00817v1 Announce Type: cross Abstract: Join discovery aims to identify tables from large data repositories that can augment a query table with complementary information, enabling downstream

arXiv:2510.23191v2 Announce Type: replace Abstract: Predictive benchmarking, evaluating machine learning models based on predictive performance and competitive ranking, is central to machine learning

arXiv:2610.00118v1 Announce Type: new Abstract: Customer churn prediction on the IBM Telco Customer Churn benchmark (n = 7,043) routinely reports test accuracies above 95%, with the most cited publish

arXiv:2609.38652v1 Announce Type: new Abstract: Agentic AI systems increasingly rely on cloud-hosted large language models for planning, tool use, and iterative execution, raising concerns about API c

arXiv:2510.17932v5 Announce Type: replace-cross Abstract: We introduce Chart2Code, a new benchmark for evaluating the chart understanding and code generation capabilities of large multimodal models (L

arXiv:2609.38523v1 Announce Type: cross Abstract: Financial forecasting from earnings conference calls requires models to reason over complex corporate disclosures, market expectations, and subtle com

arXiv:2609.37053v1 Announce Type: new Abstract: Multimodal GUI agents have achieved impressive results on general software benchmarks, yet their ability to operate professional scientific software rem

arXiv:2609.37494v1 Announce Type: new Abstract: Multiple-choice benchmarks are cheap to grade and are running out of room, and the standard remedy, writing harder items, is slow and repeated for every

arXiv:2609.33261v1 Announce Type: new Abstract: Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical q

arXiv:2609.31651v1 Announce Type: new Abstract: Historical manuscripts remain largely absent from modern vision-language benchmarks, leaving open how well multimodal large language models (MLLMs) hand

arXiv:2509.23219v2 Announce Type: replace Abstract: Technical-domain benchmarks constructed from arXiv papers can overlap the same public text used in LLM pretraining. Auditing this risk at training-c

arXiv:2609.30483v1 Announce Type: cross Abstract: Audio language models state numbers for acoustic quantities, and neither human opinion nor a judge model says whether such a number is true of the sig

arXiv:2608.00008v2 Announce Type: replace Abstract: The local deployment of large language models (LLMs) is gaining traction due to privacy concerns and the desire for on-premise inference. However, t

arXiv:2609.30205v1 Announce Type: new Abstract: Large language model (LLM)-based clinical assistants are increasingly being integrated into electronic health record (EHR) systems, transforming how cli

arXiv:2609.27036v1 Announce Type: cross Abstract: Polymer property prediction lacks open, standardized benchmarks that enable rigorous comparison of machine-learning methods, with existing resources c

arXiv:2609.26872v1 Announce Type: new Abstract: Musculoskeletal (MSK) humanoids provide a physiologically grounded embodiment for studying full-body motor control, but their high-dimensional muscle ac

arXiv:2609.26490v1 Announce Type: new Abstract: This study introduces an interdisciplinary framework for benchmarking robots deployed in public environments, addressing the gap between traditional lab

arXiv:2609.26618v1 Announce Type: new Abstract: End-to-end (E2E) driving policies have progressed rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy withstands comp