Search: “benchmark”

arXiv:2604.18177v1 Announce Type: new Abstract: Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight

arXiv:2604.10261v1 Announce Type: new Abstract: Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of

arXiv:2604.09911v1 Announce Type: cross Abstract: In the psychological literature the term `general intelligence' describes correlations between abilities and not simply the number of abilities. This

arXiv:2407.08101v4 Announce Type: replace Abstract: Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, wher

arXiv:2604.06173v1 Announce Type: cross Abstract: Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning. In statutory do

arXiv:2610.22209v1 Announce Type: cross Abstract: Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy f

arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Y

arXiv:2610.19525v1 Announce Type: cross Abstract: Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underl

arXiv:2610.17331v1 Announce Type: new Abstract: General-purpose coding agents use standard shell and file interactions, yet formal theorem-proving agents often rely on specialized, ad-hoc tooling. Thi

arXiv:2610.14539v1 Announce Type: new Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experienc

arXiv:2610.16319v1 Announce Type: new Abstract: Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluate

arXiv:2610.20221v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in E

arXiv:2610.19795v1 Announce Type: new Abstract: Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires

arXiv:2610.14589v1 Announce Type: new Abstract: Walking sequences reveal human-interpretable attributes related to body structure, posture, and motion. Gait attribute recognition provides a concrete r

arXiv:2610.18624v1 Announce Type: new Abstract: Pretraining generalist Earth observation foundation models requires data that combine several sensors, varied temporal sampling and, increasingly, text,

arXiv:2610.15020v1 Announce Type: new Abstract: PATTERN, a synthetic node-classification benchmark still used to rank graph networks, plants a random subgraph in a random host graph and asks which nod

arXiv:2610.16338v1 Announce Type: new Abstract: Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated s

arXiv:2610.18789v1 Announce Type: new Abstract: Agentic remote sensing is evaluated on how many questions it can answer, whereas crisis response depends on whether its answers can be relied on and how

arXiv:2610.13148v1 Announce Type: cross Abstract: In this work, we revisit the classic Les Miserables benchmark network from the perspective of social communication. Using the 1998 film adaptation of

arXiv:2610.13811v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard t

arXiv:2610.09000v1 Announce Type: cross Abstract: As small language models (SLMs) are increasingly deployed on resource-constrained and on-device platforms, including as components of agentic systems,

arXiv:2610.10276v1 Announce Type: cross Abstract: An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to co

arXiv:2610.07355v1 Announce Type: cross Abstract: Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and co

arXiv:2610.04425v1 Announce Type: new Abstract: Agricultural visual models are typically evaluated for either recognition or localization, but reliable diagnosis requires identifying what is present a

arXiv:2610.06825v1 Announce Type: cross Abstract: Scientific figures often encode quantitative results that are not readily available in machine-readable form, making accurate plot digitization import

arXiv:2610.05715v1 Announce Type: new Abstract: Vision-language models (VLMs) can detect that an object has rotated across views, but cannot reliably tell by how much. We introduce OR-Bench, a fine-gr

arXiv:2610.04836v1 Announce Type: new Abstract: Remote sensing agents rely on perception, measurement, and raster analysis tools to solve Earth observation tasks. We refer to their judgments and quant

arXiv:2610.03625v1 Announce Type: new Abstract: Relational databases are among the most widely deployed forms of structured knowledge, and natural language access to them requires grounding language o

arXiv:2603.29112v2 Announce Type: replace Abstract: We introduce GISTBench, a benchmark for evaluating Large Language Models' (LLMs) ability to understand users from their interaction histories in rec

arXiv:2610.03448v1 Announce Type: cross Abstract: LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmar

arXiv:2610.00651v1 Announce Type: new Abstract: Agent evaluations are increasingly used to compare LLMs and inform deployment decisions, yet ranks can reflect not only the model but also the effects o

arXiv:2610.02186v1 Announce Type: cross Abstract: Molecular learning models are strongly shaped by their underlying representations. Yet standard sequential and graph formalisms struggle to explicitly

arXiv:2606.19788v2 Announce Type: replace Abstract: We present CombEval, a dynamic benchmark for evaluating combinatorial counting in large language models. CombEval represents each problem as a typed

arXiv:2609.38372v1 Announce Type: new Abstract: A harness is the code around a language-model agent that organizes prompts, calls tools, manages context, and controls execution. As models grow stronge

arXiv:2609.30287v1 Announce Type: cross Abstract: AI-generated text detectors achieve high accuracy on standard benchmarks, yet the internal representations that drive these predictions remain poorly

arXiv:2609.31140v1 Announce Type: new Abstract: Most Vision-Language Models (VLMs) are built by extending pretrained Large Language Models (LLMs) with visual modules and multimodal alignment. However,

arXiv:2609.30325v1 Announce Type: new Abstract: Agents are increasingly deployed with real autonomy in web application and network penetration testing, where a single out-of-scope action can breach a

arXiv:2609.30670v1 Announce Type: new Abstract: Streaming video understanding requires models to interpret evidence as it arrives, yet current evaluations often report task scores without specifying w

arXiv:2609.27288v1 Announce Type: new Abstract: The Abstraction and Reasoning Corpus (ARC) has become a prominent benchmark for evaluating general abstract reasoning and fluid intelligence in AI model

arXiv:2609.17916v1 Announce Type: new Abstract: Temporal link prediction on the Temporal Graph Benchmark 2.0 (TGB 2.0) faces a scalability ceiling: on the benchmark's three largest datasets, every exi

arXiv:2609.15007v1 Announce Type: new Abstract: Large language models are increasingly used as natural-language interfaces to structured data, yet they remain unreliable when answers require consisten

arXiv:2609.14811v1 Announce Type: new Abstract: We introduce Crypto Accounting Bench (CAB), a benchmark for assessing whether frontier and open-weight language models can reconstruct the complete jour

arXiv:2609.14779v1 Announce Type: new Abstract: Performing deliberate mathematical reasoning in visual contexts is a hallmark of advanced Multimodal Large Language Models (MLLMs) and requires a sophis

arXiv:2609.13582v1 Announce Type: cross Abstract: A clinical agent benchmark can report the same verdict on identical inputs while the agent files a materially different order on each run. Such agents

arXiv:2609.12533v1 Announce Type: cross Abstract: Real-world Earth observation (EO) agents must translate high-level scientific questions into executable workflows to acquire observations, prepare dat

arXiv:2609.12408v1 Announce Type: new Abstract: Hazardous-freight operations at petrochemical logistics nodes are safety-critical for intelligent transportation systems, yet existing vision benchmarks