Search: “benchmark”

The Stepwise Informativeness Assumption: Why are Entropy Dynamics and Reasoning Correlated in LLMs?

arXiv:2604.06192v1 Announce Type: cross Abstract: Recent work uses entropy-based signals at multiple representation levels to study reasoning in large language models, but the field remains largely em

10 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

DGX agent

arXiv:2604.07223v1 Announce Type: cross Abstract: As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to int

model-releasesarxiv-cs-ai

10 Apr 2026

Data World: Scaling Data Environments from Scratch

arXiv:2610.21656v1 Announce Type: new Abstract: Data agents use code to process heterogeneous files and derive analytical insights. Scaling their training faces costly environment curation, privacy an

FDE-Gym: The Last Mile of AI Delivery

arXiv:2610.22350v1 Announce Type: new Abstract: Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world custom

Language Is a Security Surface for Agent Guards

arXiv:2610.22244v1 Announce Type: cross Abstract: Step-level guards decide, before each tool call, whether an agent's next action is safe given the user's request and the trajectory so far. Users writ

The actual state of open-weight models vs. frontier models

model-releasesr-localllama

It's clear that today’s SOTA (state-of-the-art) models remain ahead of open-source models. However, if we filter the benchmarks correctly, a different picture emerges compared to what we saw just a fe

ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

arXiv:2610.21098v1 Announce Type: cross Abstract: The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark tha

What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination

arXiv:2610.21229v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) math corpora are routinely screened for benchmark overlap, but the reliability and data-loss costs

Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given

arXiv:2610.16911v1 Announce Type: new Abstract: Does providing additional context to automated AI auditors improve their effectiveness? Using the Petri tool to audit LLM coding agents, we test the imp

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

arXiv:2610.17659v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong capabilities in interactive problem solving, tool use, and long-horizon task execution. However, the

Cowork-Gym: Synthesizing Interactive Work Environments at Scale

arXiv:2610.17670v1 Announce Type: new Abstract: Cowork agents navigate dependencies across tools and files while adapting to changing user requirements and edits to shared workspaces. Scaling such env

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

model-releasesarxiv-cs-lg

arXiv:2610.19240v1 Announce Type: new Abstract: Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predomin

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

arXiv:2610.14071v1 Announce Type: cross Abstract: As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue ha

Measuring Epistemic Humility in Multimodal Large Language Models

arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant risk

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

arXiv:2610.20548v1 Announce Type: new Abstract: Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and mult

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?

arXiv:2610.17837v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation t

Recursive Environment Scaling for General Long-Horizon Agents

arXiv:2610.16397v1 Announce Type: new Abstract: We study environment scaling as a way to increase agent-benchmark difficulty while holding core task objectives fixed. Our framework, Recursive Environm

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

arXiv:2610.17205v1 Announce Type: new Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typ

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

arXiv:2610.16634v1 Announce Type: cross Abstract: Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing close

14 Oct 2026

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

arXiv:2610.17079v1 Announce Type: new Abstract: Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual

14 Oct 2026

Web Scale on a Budget: Tail-Aware Retrieval Evaluation

arXiv:2610.17986v1 Announce Type: new Abstract: A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability. One prominent example is ILIAS,

14 Oct 2026

Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

arXiv:2610.13152v1 Announce Type: new Abstract: Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. Ho

12 Oct 2026

SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire

arXiv:2610.12596v1 Announce Type: new Abstract: Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short a

12 Oct 2026

WoundAmbit: Bridging State-of-the-Art Semantic Segmentation and Real-World Wound Care

arXiv:2504.06185v3 Announce Type: replace-cross Abstract: Chronic wounds affect a large population, particularly the elderly and diabetic patients, who often exhibit limited mobility and co-existing h

12 Oct 2026

UPDATE: Qwen 3.8 27B 140 tok/s on single RTX 3090 Megakernel: KL divergence 0.0009 vs llama.cpp

model-releasesr-localllama

This is a follow-up to my post from yesterday (https://www.reddit.com/r/LocalLLaMA/comments/1x2erdj/qwen3827b_on_a_single_3090_140_toks_on_code_with/) detailing the CUDA megakernel. Thank you all for

11 Oct 2026

Generative Adversarial Loops

arXiv:2610.11458v1 Announce Type: cross Abstract: AI research progress can be viewed as the interaction between two processes: benchmark creation and method discovery. Historically, both were driven b

9 Oct 2026

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete re

8 Oct 2026

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when u

8 Oct 2026

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

arXiv:2610.08680v1 Announce Type: cross Abstract: Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-speci

Logbook: Extremely Long-form Audio Event Understanding

arXiv:2610.07338v1 Announce Type: cross Abstract: Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we intro

Open SLM Evalulations

model-releasesr-localllama

Introducing Open SLM Evaluations! Open SLM Evaluations is a dataset containing benchmarks for 106 SLMs(150M parameters and below). It has benchmark scores for each model, links, parameter counts, and

PlotPick: AI-powered batch extraction of numerical data from scientific figures

arXiv:2605.06021v2 Announce Type: replace Abstract: Systematic reviews and meta-analyses often need numerical data reported only in figures, and extracting them with interactive digitisers usually req

SWE-Game: Can Coding Agents Build the Games We Want?

arXiv:2609.33678v2 Announce Type: replace Abstract: We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five t

The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory

arXiv:2610.07309v1 Announce Type: new Abstract: Long-term-memory agents can retrieve relevant information that is inadmissible for the current request because it belongs to another principal, violates

Environmental sensor readings in two crop disease image datasets identify the session in which each image was taken

arXiv:2610.06369v1 Announce Type: new Abstract: Integrating environmental sensor data with leaf imagery is widely reported to boost crop disease classification accuracy. In this work, we reveal that t

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

arXiv:2610.04139v1 Announce Type: new Abstract: Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language mode

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

arXiv:2507.07297v2 Announce Type: replace Abstract: Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, i

OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies

model-releasesarxiv-cs-lg

arXiv:2610.05878v1 Announce Type: new Abstract: Benchmarks expose vision-language-action (VLA) policies to few canonical instructions, while exhaustive deployment testing is impossible. We introduce O

ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory

arXiv:2610.05097v1 Announce Type: new Abstract: As multimodal large language models (MLLMs) reason for longer, attention to the initial visual input diminishes, weakening visual grounding. Visual memo

Agents

Lost in the Request: How Communication Variation Disrupts Retrieval and Action in Email Agents

arXiv:2610.02627v1 Announce Type: new Abstract: An email assistant should not complete less work simply because a user phrases the same request differently. Yet most benchmarks test each task with onl

agentsarxiv-cs-ai

5 Oct 2026

Syntheses

Wiki Lint Report — 2026-10-04

linthealth-checkautomated

Automated lint: 19 errors, 64915 warnings, 3 info

4 Oct 2026

Incident-Arena: Getting agents to the last nine of reliability

arXiv:2610.00648v1 Announce Type: new Abstract: AI coding agents are ubiquitous in engineering workflows amongst industry and academia. Yet, despite their use in app coding, relatively less attention

MECHVAR: Variance-Guided Mechanism Discrimination for Autonomous Machine Learning Experiment Selection

model-releasesarxiv-cs-lg

arXiv:2610.01819v1 Announce Type: new Abstract: Benchmark gains are often mechanism-ambiguous: reproducing an improvement does not by itself identify why it occurs. We study finite-library mechanism d

Predictive Credit: Measuring What Scientific Explanations Add to Experimental Forecasts

arXiv:2610.00314v1 Announce Type: new Abstract: Research agents explain planned experiments. We measure predictive credit with paired forecasts sharing an intervention, forecaster, and outcome while v

ROWBench: Do Video Models Render What the Program Specifies?

arXiv:2610.02205v1 Announce Type: new Abstract: Programmable world models separate executable dynamics from visual generation, offering a promising foundation for next-generation game engines. However

Scientific Agents: Evaluating Profession-Specific System Prompts on Scientific Tasks

arXiv:2610.00084v1 Announce Type: new Abstract: Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific A

DrivingBench: Can Vision-Language Models Drive a Toyota Corolla?

arXiv:2609.38948v1 Announce Type: cross Abstract: Frontier models excel at many digital benchmarks, yet their ability to drive a real car, an everyday human skill, remains largely untested. We present