Search: “benchmark”

arXiv:2601.11895v3 Announce Type: replace-cross Abstract: DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,8

arXiv:2605.14537v1 Announce Type: new Abstract: We introduce extsc{Cattle Trade, a multi-agent benchmark for evaluating large language models (LLMs) as agents in strategic reasoning under imperfect in

arXiv:2605.14167v1 Announce Type: new Abstract: Every AI benchmark operationalizes theoretical assumptions about the capability it claims to assess. When assumptions function as unexamined commitments

arXiv:2605.12361v1 Announce Type: new Abstract: Evaluating large language models (LLMs) in the biomedical domain requires benchmarks that can distinguish reasoning from pattern matching and remain dis

arXiv:2605.10448v1 Announce Type: new Abstract: Interactive agent benchmarks map an agent run to a binary outcome through outcome checks. When these checks rely on surface level signals or fail to cap

arXiv:2605.08197v1 Announce Type: cross Abstract: Most causal benchmarks for language models score local answers or graph structure. We introduce ReplaySCM, a 1,300 item benchmark for executable causa

arXiv:2603.24946v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on

arXiv:2605.02003v1 Announce Type: new Abstract: Machine Learning (ML) has transformed many scientific fields, yet key applications still lack standardized benchmarks. Raman spectroscopy, a widely used

arXiv:2605.02209v1 Announce Type: cross Abstract: Evaluating large language models across many benchmarks is expensive, yet many benchmarks are highly correlated. We formalize the selection of a small

arXiv:2604.24929v1 Announce Type: new Abstract: Agent benchmarks remain largely English-centric, while their multilingual versions are often built with machine translation (MT) and limited post-editin

DeepSeek released a new open-weights model with strong benchmark performance, though Mollick notes that benchmark results may not fully capture the capabilities of open models compared to closed syste

arXiv:2506.06211v2 Announce Type: replace-cross Abstract: Puzzlehunts are a genre of complex, multi-step puzzles lacking well-defined problem definitions. In contrast to conventional reasoning benchma

arXiv:2510.16822v3 Announce Type: replace-cross Abstract: Coral reefs are rapidly declining under anthropogenic pressures (e.g., climate change), creating an urgent need for scalable and automated mon

arXiv:2604.17778v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed in the telecommunications domain for critical tasks, relying heavily on Retrieval-Augmented Gener

arXiv:2604.15871v1 Announce Type: cross Abstract: The evaluation of visual editing models remains fragmented across methods and modalities. Existing benchmarks are often tailored to specific paradigms

arXiv:2505.09368v2 Announce Type: replace Abstract: Standard benchmarks for optical flow, scene flow, and stereo vision algorithms generally focus on model accuracy rather than robustness to image cor

Ocr benchmark We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is

arXiv:2602.02188v2 Announce Type: replace Abstract: While large language models (LLMs) have shown strong performance in math and logic reasoning, their ability to handle combinatorial optimization (CO

arXiv:2602.11354v2 Announce Type: replace Abstract: The literature has witnessed an emerging interest in AI agents for automated assessment of scientific papers. Existing benchmarks focus primarily on

The entire team put a lot of effort into this benchmark. Document parsing and OCR certainly isn't solved, but now it is a bit easier to measure 🦙 We’re open sourcing the first document OCR benchmark f

We’re open sourcing the first document OCR benchmark for the agentic era, ParseBench. Document parsing is the foundation of every AI agent that works with real-world files. ParseBench is a benchmark t

arXiv:2604.08457v1 Announce Type: new Abstract: Cooperative autonomous driving requires traffic scene understanding from both vehicle and infrastructure perspectives. While vision-language models (VLM

arXiv:2510.04641v3 Announce Type: replace Abstract: Large-scale web-scraped text corpora used to train general-purpose AI models often contain harmful demographic-targeted social biases, creating a re

arXiv:2604.02445v2 Announce Type: replace Abstract: Matrix Profile (MP) methods are an interpretable and scalable family of distance-based methods for time-series anomaly detection, but strong benchma

arXiv:2604.08211v1 Announce Type: new Abstract: Modern multimodal generators can now produce scientific figures at near-publishable quality, creating a new challenge for visual forensics and research

arXiv:2604.08362v1 Announce Type: new Abstract: The emergence of Large Language Models (LLMs) has illuminated the potential for a general-purpose user simulator. However, existing benchmarks remain co

Saritha Rai / Bloomberg: DeepSeek releases V4.1-Flash-1017, an update it says offers “stronger benchmark performance, faster task completion, and shorter outputs” than its predecessor — DeepSeek relea

arXiv:2610.21109v1 Announce Type: new Abstract: When humans collaborate with AI agents on long-horizon tasks, what happens to the agents' contributions? For solving long-horizon tasks, humans place gr

arXiv:2610.22372v1 Announce Type: cross Abstract: Personalization is essential for general-purpose robots to transition into household environments and achieve true practicality. While general robotic

arXiv:2610.19704v1 Announce Type: new Abstract: Large language models (LLMs) increasingly help users arrange appointments, plan journeys, and allocate limited resources. These tasks require models to

arXiv:2610.16191v1 Announce Type: new Abstract: A model may select the correct answer yet fail to construct it through a sequence of local decisions. We introduce Choice2Seq, a diagnostic benchmark th

arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily e

arXiv:2610.19185v1 Announce Type: new Abstract: Evaluation of large language models (LLMs) remains largely pre-scientific: it involves many design decisions, with little guidance on how to make them.

arXiv:2610.16782v1 Announce Type: new Abstract: Industrial manipulation in flexible manufacturing, e.g., high-mix, low-volume (HMLV) electronics assembly, requires robot policies to adapt to changing

arXiv:2610.06170v2 Announce Type: replace Abstract: Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple

arXiv:2610.17728v1 Announce Type: new Abstract: Egocentric multimodal assistants see through the user's eyes and guide what the user does next, so their outputs directly shape physical actions, includ

arXiv:2606.20235v2 Announce Type: replace-cross Abstract: Academic paper search is a core step in scientific research, and LLM-based search agents are emerging as a promising paradigm for iterative, i

arXiv:2610.19063v1 Announce Type: new Abstract: As video models enable longer and more realistic interactive experiences, maintaining a world consistent with its history becomes essential. This requir

arXiv:2610.18305v1 Announce Type: new Abstract: The world encompasses both physical environments and human actors. Predicting its evolution requires understanding both physical laws and how emotions s

arXiv:2610.17573v1 Announce Type: new Abstract: Self-supervised pretraining has expanded the pool of 3D medical foundation models, but published evaluations use different datasets, endpoints and fine-

Benchmarks are becoming the core competency of any company. Our company mission is to build the world’s best models for document processing. This also means that we have to build the world’s best benc

arXiv:2610.18844v1 Announce Type: new Abstract: Repetition counting requires identifying and enumerating action cycles over time. While state-of-the-art video language models describe actions fluently

arXiv:2610.13947v2 Announce Type: replace Abstract: In this paper, we study physically grounded articulated 3D asset generation---moving the goal from reproducing how objects look to capturing how the

Zai recently launched a domain‑specific language model called **GLM Legal**. According to a tweet posted at 4:13 PM on Oct 12, 2026, GLM Legal became the leading open‑weight model on both **Harvey’s L

The ultimate LLM benchmark: Chilean Spanish. 80% of AI models fail to recognize Chilean Spanish and insist on speaking like an Argentine 80% de modelos de IA no logra reconocer el español chileno e in

arXiv:2601.17645v2 Announce Type: replace-cross Abstract: Internet audio-visual clips convey meaning through time-varying sound and motion, which extend beyond what text alone can represent. To examin