Search: “benchmark”

arXiv:2607.21998v1 Announce Type: new Abstract: This paper introduces a new benchmark test, Medical-Checklist, for assessing medical multimodal models. The recent advancements in multimodal models hav

arXiv:2607.22393v1 Announce Type: cross Abstract: Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual res

Anyone building with AI eventually hits the same tradeoff: how to get the most intelligence per dollar, the right model at the right cost for each task. That’s what DigitalOcean Inference Engine is bu

arXiv:2512.01113v2 Announce Type: replace-cross Abstract: Algorithmic reasoning -- the ability to perform step-by-step logical inference -- is a synthetic benchmark for evaluating multi-step reasoning

arXiv:2607.12894v1 Announce Type: new Abstract: Building capable embodied agents requires not only multimodal perception and understanding, but also agentic capabilities for reasoning about actions, a

just ran this on my phone (17 pro) Every on-device AI benchmark you've seen was measured on someone else's phone. So we stopped posting numbers and shipped the benchmark instead. Built into the RunAny

arXiv:2607.07946v1 Announce Type: cross Abstract: DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents. Most public agentic coding benchmarks fo

arXiv:2606.24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare

arXiv:2603.10044v2 Announce Type: replace-cross Abstract: A safety score earned on a benchmark need not predict how the same model behaves once it is wrapped in an agentic scaffold the benchmark never

arXiv:2605.17652v1 Announce Type: new Abstract: There are not enough established benchmarks for the task fo speech summarization. Creating new benchmarks demands human annotation, as LLMs could embed

arXiv:2605.12022v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance on standard knowledge evaluation benchmarks, yet recent work shows that their knowledge capabili

arXiv:2605.10597v1 Announce Type: cross Abstract: Benchmarks for coding agents increasingly measure source-level software repair, and cybersecurity benchmarks increasingly measure broad capture-the-fl

arXiv:2602.22953v2 Announce Type: replace Abstract: General-purpose agents perform tasks in unfamiliar environments without domain-specific manual customization. Yet no study has systematically measur

arXiv:2605.04886v1 Announce Type: new Abstract: This position paper argues that the under-representation of social science tasks in contemporary LLM benchmarks limits advances in both LLM evaluation a

arXiv:2605.04454v1 Announce Type: cross Abstract: Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such

arXiv:2604.24038v1 Announce Type: new Abstract: Static benchmarks measure what AI agents can do at a fixed point in time but not how they are adopted, maintained, or experienced in deployment. We intr

arXiv:2604.22897v1 Announce Type: cross Abstract: Patent retrieval underpins critical decisions in innovation, examination, and IP strategy, yet progress has been hampered by the absence of benchmarks

arXiv:2501.03624v2 Announce Type: replace-cross Abstract: Large language models (LLMs) excel on many NLP benchmarks, but their behavior on real-world, semi-structured prediction remains underexplored.

arXiv:2604.19131v1 Announce Type: new Abstract: Automated essay scoring (AES) is commonly evaluated on public benchmarks using quadratic weighted kappa (QWK). However, because benchmark labels are ass

VideoGameBench is a new benchmark launched on Antim Labs, created by a1zhang, Thomas L. Griffiths, Karthik R. N, and Ofir Press. The benchmark likely evaluates AI model performance on video game-relat

arXiv:2511.16857v3 Announce Type: replace Abstract: Vision Language Models (VLMs) have achieved impressive performance on spatial reasoning benchmarks, yet these evaluations mask critical weaknesses i

arXiv:2604.18177v1 Announce Type: new Abstract: Benchmarks are often used as a standard to understand LLM capabilities in different domains. However, aggregate benchmark scores provide limited insight

ARC-AGI-3 is a new benchmark iteration designed so that any intelligent human applying genuine effort should achieve greater than 90% accuracy, maintaining the benchmark's core principle that tasks mu

arXiv:2604.10261v1 Announce Type: new Abstract: Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of

arXiv:2604.09911v1 Announce Type: cross Abstract: In the psychological literature the term `general intelligence' describes correlations between abilities and not simply the number of abilities. This

arXiv:2407.08101v4 Announce Type: replace Abstract: Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, wher

arXiv:2604.06173v1 Announce Type: cross Abstract: Legal QA benchmarks have predominantly focused on case law, overlooking the unique challenges of statute-centric regulatory reasoning. In statutory do

arXiv:2610.22209v1 Announce Type: cross Abstract: Agent memory benchmarks evaluate whether an agent's responses reflect correct knowledge, measuring downstream question-answering accuracy as a proxy f

arXiv:2610.19560v1 Announce Type: cross Abstract: In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Y

arXiv:2610.19525v1 Announce Type: cross Abstract: Verifiable instruction-following benchmarks emphasize output-constraint satisfaction but can overlook whether responses meaningfully answer the underl

arXiv:2610.17331v1 Announce Type: new Abstract: General-purpose coding agents use standard shell and file interactions, yet formal theorem-proving agents often rely on specialized, ad-hoc tooling. Thi

arXiv:2610.14539v1 Announce Type: new Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experienc

arXiv:2610.16319v1 Announce Type: new Abstract: Real-world hybrid web tasks need both information gathering and page interaction, yet search-based and operation-based agents are developed and evaluate

arXiv:2610.20221v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed across languages, yet their safety and reasoning capabilities remain predominantly evaluated in E

arXiv:2610.19795v1 Announce Type: new Abstract: Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires

arXiv:2610.14589v1 Announce Type: new Abstract: Walking sequences reveal human-interpretable attributes related to body structure, posture, and motion. Gait attribute recognition provides a concrete r

arXiv:2610.18624v1 Announce Type: new Abstract: Pretraining generalist Earth observation foundation models requires data that combine several sensors, varied temporal sampling and, increasingly, text,

arXiv:2610.15020v1 Announce Type: new Abstract: PATTERN, a synthetic node-classification benchmark still used to rank graph networks, plants a random subgraph in a random host graph and asks which nod

MacStudio 128GB M1 ultra I'm curious how people who can run both models are deciding which one to work with. DeepSeekv4 benchmarks right below GPT 5.5 high. Flash Next benchmarks right above 5.4/5.5 x

arXiv:2610.16338v1 Announce Type: new Abstract: Recent years have witnessed rapid progress in joint audio--visual generation. However, audio--visual synchronization remains a challenge, as generated s

arXiv:2610.18789v1 Announce Type: new Abstract: Agentic remote sensing is evaluated on how many questions it can answer, whereas crisis response depends on whether its answers can be relied on and how

arXiv:2610.13148v1 Announce Type: cross Abstract: In this work, we revisit the classic Les Miserables benchmark network from the perspective of social communication. Using the 1998 film adaptation of

arXiv:2610.13811v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong results in classical code generation and are increasingly used to write the unit tests that safeguard t

arXiv:2610.09000v1 Announce Type: cross Abstract: As small language models (SLMs) are increasingly deployed on resource-constrained and on-device platforms, including as components of agentic systems,

arXiv:2610.10276v1 Announce Type: cross Abstract: An LLM safety patch can pass a benchmark while still being a poor repair. This risk is especially acute for jailbreak repairs, where the goal is to co

arXiv:2610.07355v1 Announce Type: cross Abstract: Video world models track objects they can see; we ask what they keep of objects they cannot. We hide an object from a frozen V-JEPA 2 predictor and co