Search: “benchmark”

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on lon

arXiv:2610.02200v1 Announce Type: new Abstract: We show that multimodal models possess strong reasoning abilities and that an appropriate harness can unlock their potential to solve tasks across diver

arXiv:2609.38479v1 Announce Type: cross Abstract: This paper presents an explainable approach to pedestrian routing, in which perceived safety is estimated from street-level imagery through an explici

arXiv:2609.40230v1 Announce Type: cross Abstract: Visual perception is conventionally formulated as a one-shot prediction from a single glance at the image, under the assumption that the image content

arXiv:2609.39392v1 Announce Type: new Abstract: Autonomous research agents can generate hypotheses and conduct experiments, but experimentation remains a major source of computational cost. A fundamen

So everyone has been yelling about how I should be using Gufo instead of halogen because it's open source and it's 'just as good or better'. Checking in on their GitHub (GitHub.com/gufo-org/gufo) got

arXiv:2609.39639v1 Announce Type: new Abstract: Tabular learning uses structured data to predict target outcomes. Traditionally, this process has relied on labeled data. However, large language models

arXiv:2609.39229v1 Announce Type: cross Abstract: Automatic evaluation of faithfulness increasingly relies on a large language model acting as a judge, yet the most reliable judges are proprietary fro

arXiv:2609.39568v1 Announce Type: cross Abstract: Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable g

arXiv:2609.39846v1 Announce Type: cross Abstract: We investigate the problem of role-capability leakage (RCL), in which a role-prompted reasoning model generates convincing in-role text while continui

arXiv:2609.40325v1 Announce Type: new Abstract: As interactive 3D worlds are increasingly used to study intelligent behavior, it becomes important to develop efficient pipelines for identifying anomal

arXiv:2609.37143v1 Announce Type: cross Abstract: Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks wit

arXiv:2609.37356v1 Announce Type: new Abstract: Generating optimization instances that are both feasible and computationally challenging is crucial for benchmarking solvers and training learning-based

arXiv:2609.32103v1 Announce Type: new Abstract: Large language models can memorize private or harmful information, motivating machine unlearning methods that remove targeted knowledge while preserving

arXiv:2609.32391v1 Announce Type: cross Abstract: Continual-learning agents are systems of models, harnesses, and memory operating over long multi-session horizons. Evaluating and training them requir

arXiv:2609.31823v1 Announce Type: new Abstract: Large vision-language models (LVLMs) have demonstrated remarkable performance on multimodal reasoning benchmarks, yet their perceptual reliability under

arXiv:2609.31546v1 Announce Type: new Abstract: Electrocardiogram (ECG) foundation models typically tokenize the signal into fixed-length patches that ignore cardiac structure, so a patch may split a

arXiv:2506.04734v3 Announce Type: replace Abstract: Reasoning models represented by the Deepseek-R1-Distill series have been widely adopted by the open-source community due to their strong performance

arXiv:2504.06188v3 Announce Type: replace Abstract: AI agents can extend their capabilities at inference time by loading reusable skills into context, yet equipping an agent with too many skills, part

arXiv:2609.31298v1 Announce Type: cross Abstract: Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder for which early and accurate diagnosis is critical to improving long-term deve

Every week, I talk with founders who are building at an unbelievable pace. Teams are moving from inception to product-market fit faster than ever, with foundation models wired deeply into their core p

We comprehensively evaluated 16 recent frontier VLMs - including Opus 5.5 and GPT-6 Sol/Luna - on whether higher effort led to better document parsing performance. Higher effort typically leads to imp

arXiv:2609.26176v2 Announce Type: replace-cross Abstract: Encoded-prompt attacks are evaluated almost entirely on their harmful arm: a benchmark sends obfuscated harmful requests and reports how often

arXiv:2609.29465v1 Announce Type: new Abstract: Large language model based coding agents have made substantial progress on repository-level software engineering tasks. Existing repository benchmarks,

arXiv:2609.26900v1 Announce Type: cross Abstract: A language model agent acts through the tools it is given. The data it reads while working on a task can redirect what it does with those tools. A gro

arXiv:2609.28007v1 Announce Type: new Abstract: Most Turkish-capable large language models (LLMs) are evaluated using general-purpose benchmarks rather than long, structurally complex domain documents

arXiv:2609.27307v1 Announce Type: new Abstract: Graphical User Interface (GUI) agents enable the fulfillment of complex user instructions through multi-turn interactions with software environments, re

arXiv:2609.27041v1 Announce Type: new Abstract: Mathematical reasoning benchmarks are typically organized by topic, but language models may organize their internal computation by reusable reasoning ap

arXiv:2609.28197v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents that alter real-world states, ensuring operational safety across multi-step workflows has

arXiv:2609.28049v1 Announce Type: cross Abstract: Video understanding is usually benchmarked on curated, single-actor, or professionally filmed clips, and a strong score there is routinely read as evi

arXiv:2609.27606v1 Announce Type: new Abstract: We introduce State-Grounded Conditioning (SGC), a design principle for user-facing LLM agents that must condition on live user state (game state, sessio

arXiv:2603.08930v2 Announce Type: replace-cross Abstract: This paper introduces a benchmark for evaluating whether vision-language models (VLMs) can generate plant simulation configurations from image

arXiv:2503.15242v3 Announce Type: replace Abstract: We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generati

arXiv:2604.07733v2 Announce Type: replace Abstract: Evaluating strategic decision-making in LLM-based agents requires generative, competitive, and longitudinal environments, yet few benchmarks provide

arXiv:2410.14948v2 Announce Type: replace Abstract: Medical multimodal large language models (MLLMs) can perform well on existing medical visual question answering (MedVQA) benchmarks, but their train

Saw a new open 27B model release that looks like it could be interesting for the Ollama crowd: VeriLoop E2. It is a Qwen3.8-27B post-train aimed at code, mathematics, scientific reasoning, and agentic

I didn't know my set up was outperforming nearly everyone until reading another discussion where people were struggling getting half of that speed with half the context on the same hardware. I benchma

arXiv:2609.24972v1 Announce Type: cross Abstract: An LLM agent's capability is largely magnified by its harness, namely the prompts, control flow, tooling, memory, and context management surrounding t

On September 21 2026, Yohei reported results from a hard “insect‑identification” benchmark where model accuracy spanned 23 points across the evaluated systems. The Qwen 4B + Glance model scored 0.96,

I never thought there would be a model worse than the Gemini series, but here we are. I remember Elon tweeting that this model would beat Astra & 5.1 across all benchmarks. submitted by /u/prvthvm [li

I decided to run prism-ml/Ternary-Bonsai-2-27B-PQ2_0 through my own set of UNSCIENTIFIC benchmarks. I needed something to compare it to, so I decided I would compare with another 27B model by filesize

I just released a new dataset https://huggingface.co/datasets/LocalLLaMA/deepswe-mini it is meant to be more efficent when testing and benchmarking local models on DeepSwe https://deepswe.datacurve.ai

Vacations give you time to think and explore. I came across a few things and made a few discoveries of my own. It turns out that, as a programmer, I no longer need the smartest and most expensive AI m

arXiv:2609.19916v1 Announce Type: cross Abstract: Large language models (LLMs) are typically evaluated on static benchmarks, even though natural language constantly evolves through newly emerging word

arXiv:2605.13481v2 Announce Type: replace Abstract: We introduce PersonalAI 2.0 (PAI-2), a novel framework designed to enhance LLM-based systems through integration of external knowledge graphs (KGs).

arXiv:2609.18860v1 Announce Type: cross Abstract: When a large vision-language model misclassifies a harmful meme, the failure may reflect missing internal evidence or an inability to route represente