Search: “benchmark”

arXiv:2610.05715v1 Announce Type: new Abstract: Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much. We introduce OR-Bench, a fine-gr

arXiv:2610.04836v1 Announce Type: new Abstract: Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quant

arXiv:2610.03625v1 Announce Type: new Abstract: Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language o

arXiv:2603.29112v2 Announce Type: replace Abstract: We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in rec

arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmar

arXiv:2610.00651v1 Announce Type: new Abstract: Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects o

arXiv:2610.02186v1 Announce Type: cross Abstract: Molecular learning models are strongly shaped by their underlying representations. Yet standard sequential and graph formalisms struggle to explicitly

arXiv:2606.19788v2 Announce Type: replace Abstract: We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed

I've run the benchmark on a RTX 3090, 1024 tokens in / 256 out, concurrency 1–32. If the model fits on vRAM (Gemma-4-26B-A4B, byte-identical GGUF on both engines): llama.cpp has 2.2–3.2× the throughpu

arXiv:2609.38372v1 Announce Type: new Abstract: A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronge

arXiv:2609.30287v1 Announce Type: cross Abstract: AI-generated text detectors achieve high accuracy on standard benchmarks, yet the internal representations that drive these predictions remain poorly

arXiv:2609.31140v1 Announce Type: new Abstract: Most Vision-Language Models (VLMs) are built by extending pretrained Large Language Models (LLMs) with visual modules and multimodal alignment. However,

arXiv:2609.30325v1 Announce Type: new Abstract: Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a

arXiv:2609.30670v1 Announce Type: new Abstract: Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying w

ggml-cpu: tiled mul_mat for k-quants (#27851) Added tiled mul_mat. For each mul_mat_one_chunk, quants are unpacked into (max) 256x256 tiles of int8, one routine per quent. Then microkernel computes 16

Can LLM agents actually get through a day in the life of a normal user? That question got me reading papers on Android agents and mobile benchmarks over the past few months. A few patterns kept showin

arXiv:2609.27288v1 Announce Type: new Abstract: The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI model

LlamaIndex 发布了一个面向 AI Agent 的文档解析评测基准「ParseBench」 @llama_index 100 种解析方法参与评测（vlm - 67、专用解析器 - 33），2000 页真实企业文档（人工两轮校验），169011 条确定性规则，完全不用 LLM 当裁判，可复现。评测五个维度取无加权平均：表格结构、图表抽取、内容忠实度、格式语义、视觉定位。 https://ww

arXiv:2609.17916v1 Announce Type: new Abstract: Temporal link prediction on the Temporal Graph Benchmark 2.0 (TGB 2.0) faces a scalability ceiling: on the benchmark's three largest datasets, every exi

Hey experts! I genuinely have this question and would love a general consensus from people actually using these models. LLMs have advanced a lot in benchmarks and in practical usage. Even GPT-4o and G

arXiv:2609.15007v1 Announce Type: new Abstract: Large language models are increasingly used as natural-language interfaces to structured data, yet they remain unreliable when answers require consisten

arXiv:2609.14811v1 Announce Type: new Abstract: We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete jour

arXiv:2609.14779v1 Announce Type: new Abstract: Performing deliberate mathematical reasoning in visual contexts is a hallmark of advanced Multimodal Large Language Models (MLLMs) and requires a sophis

arXiv:2609.13582v1 Announce Type: cross Abstract: A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents

arXiv:2609.12533v1 Announce Type: cross Abstract: Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare dat

arXiv:2609.12408v1 Announce Type: new Abstract: Hazardous-freight operations at petrochemical logistics nodes are safety-critical for intelligent transportation systems, yet existing vision benchmarks

arXiv:2505.12135v2 Announce Type: replace-cross Abstract: When an interactive benchmark reports a single success rate for a language-model agent, it is rarely clear what that number measures. A failur

arXiv:2601.08536v3 Announce Type: replace Abstract: Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmar

arXiv:2601.00702v3 Announce Type: replace-cross Abstract: Deformable scenes violate the rigidity assumptions underpinning classical visual--inertial odometry (VIO), often leading to over-fitting to lo

arXiv:2609.10451v1 Announce Type: new Abstract: Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance

arXiv:2609.09678v1 Announce Type: new Abstract: Clinical diagnosis agents must decide not only what test to request next, but also when to diagnose or defer. Existing agent benchmarks largely evaluate

arXiv:2609.06573v1 Announce Type: new Abstract: Recent studies report that automated red-teaming finds more vulnerabilities, at lower cost, than human red-teaming on standard AI safety benchmarks, and

arXiv:2609.10394v1 Announce Type: cross Abstract: Current audio-visual speech recognition (AVSR) benchmarks, like LRS3, rely heavily on clean, scripted and rehearsed speech. They fail to reflect the c

arXiv:2609.08174v1 Announce Type: new Abstract: We introduce OntologyBench, a tiered biomedical retrieval benchmark comprising 471,854 training and 125,744 evaluation query-document relevance pairs ac

arXiv:2609.09863v1 Announce Type: new Abstract: Choosing a deep learning architecture for label-free single-cell classification remains an open question, with microscopy benchmarks reporting conflicti

arXiv:2609.10397v1 Announce Type: cross Abstract: Exception Related Code (ERC), which includes throw statements, conditions (if statements) that guard those throw statements, and try/catch blocks, is

arXiv:2609.10356v1 Announce Type: new Abstract: Long-term change understanding from images of the same place revisited over time is a challenging task with applications in map maintenance and urban in

arXiv:2609.09349v1 Announce Type: new Abstract: Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating g

arXiv:2609.05441v1 Announce Type: new Abstract: Long-term memory for LLM agents is evaluated today by conversational recall benchmarks (LoCoMo, LongMemEval), which measure question answering over dial

arXiv:2609.05761v1 Announce Type: new Abstract: Visual geolocation benchmarks typically ask a model where an image was captured without accounting for the location context that users often provide. We

arXiv:2508.00935v3 Announce Type: replace-cross Abstract: Computer-using agents (CUAs), which can autonomously control computers to perform multi-step actions, might pose significant safety risks if m

arXiv:2609.02804v1 Announce Type: cross Abstract: For more than 20 years, the Model-RB benchmark frb100-40 remained an open challenge; since 2014, its public record had stood at 99 of 100 variables. W

arXiv:2609.01660v1 Announce Type: cross Abstract: Production deployments of large language model (LLM) agents remain unreliable on long, multi-step workflows even as benchmark success rates climb stea

I had to look twice at this table. I know they are just numbers, but how the heck do you beat a competitor model released just a couple of days ago on some of the toughest benchmarks out there? Someth

arXiv:2606.23092v2 Announce Type: replace Abstract: Humans possess an innate ability to understand fine-grained interpersonal relationships, which is central to everyday social interactions. Although

arXiv:2609.01985v1 Announce Type: new Abstract: As LLM coding agents increasingly perform end-to-end engineering work, we lack empirical characterization of how they behave on systems-level requiremen