Search: “benchmark”

ARGOS: Who, Where, and When in Agentic Multi-Camera Person Search

arXiv:2604.12762v1 Announce Type: cross Abstract: We introduce ARGOS, the first benchmark and framework that reformulates multi-camera person search as an interactive reasoning problem requiring an ag

15 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Filtered Reasoning Score: Evaluating Reasoning Quality on a Model's Most-Confident Traces

DGX agent

arXiv:2604.11996v1 Announce Type: cross Abstract: Should we trust Large Language Models (LLMs) with high accuracy? LLMs achieve high accuracy on reasoning benchmarks, but correctness alone does not re

model-releasesarxiv-cs-ai

15 Apr 2026

Mobile GUI Agents under Real-world Threats: Are We There Yet?

arXiv:2507.04227v2 Announce Type: replace-cross Abstract: Recent years have witnessed a rapid development of mobile GUI agents powered by large language models (LLMs), which can autonomously execute d

15 Apr 2026

Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Music

arXiv:2604.10905v1 Announce Type: cross Abstract: We present Audio Flamingo Next (AF-Next), the next-generation and most capable large audio-language model in the Audio Flamingo series, designed to ad

OOWM: Structuring Embodied Reasoning and Planning via Object-Oriented Programmatic World Modeling

arXiv:2604.09580v1 Announce Type: new Abstract: Standard Chain-of-Thought (CoT) prompting empowers Large Language Models (LLMs) with reasoning capabilities, yet its reliance on linear natural language

TraversalBench: Challenging Paths to Follow for Vision Language Models

arXiv:2604.10999v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on many multimodal benchmarks. However, the ability to follow complex visual paths -- a task that human o

You Only Judge Once: Multi-response Reward Modeling in a Single Forward Pass

arXiv:2604.10966v1 Announce Type: cross Abstract: We present a discriminative multimodal reward model that scores all candidate responses in a single forward pass. Conventional discriminative reward m

Fine-Grained Action Segmentation for Renorrhaphy in Robot-Assisted Partial Nephrectomy

arXiv:2604.09051v1 Announce Type: new Abstract: Fine-grained action segmentation during renorrhaphy in robot-assisted partial nephrectomy requires frame-level recognition of visually similar suturing

13 Apr 2026

GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents

arXiv:2604.07429v1 Announce Type: new Abstract: Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse f

MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference

arXiv:2509.22750v3 Announce Type: replace Abstract: Real-world multi-hop QA is naturally linked with ambiguity, where a single query can trigger multiple reasoning paths that require independent resol

PeReGrINE: Evaluating Personalized Review Fidelity with User Item Graph Context

arXiv:2604.07788v1 Announce Type: cross Abstract: We introduce PeReGrINE, a benchmark and evaluation framework for personalized review generation grounded in graph-structured user--item evidence. PeRe

Self-Preference Bias in Rubric-Based Evaluation of Large Language Models

arXiv:2604.06996v1 Announce Type: cross Abstract: LLM-as-a-judge has become the de facto approach for evaluating LLM outputs. However, judges are known to exhibit self-preference bias (SPB): they tend

The Stepwise Informativeness Assumption: Why are Entropy Dynamics and Reasoning Correlated in LLMs?

arXiv:2604.06192v1 Announce Type: cross Abstract: Recent work uses entropy-based signals at multiple representation levels to study reasoning in large language models, but the field remains largely em

TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories

arXiv:2604.07223v1 Announce Type: cross Abstract: As large language models (LLMs) evolve from static chatbots into autonomous agents, the primary vulnerability surface shifts from final outputs to int

Data World: Scaling Data Environments from Scratch

arXiv:2610.21656v1 Announce Type: new Abstract: Data agents use code to process heterogeneous files and derive analytical insights. Scaling their training faces costly environment curation, privacy an

FDE-Gym: The Last Mile of AI Delivery

arXiv:2610.22350v1 Announce Type: new Abstract: Existing agent benchmarks typically evaluate agents on well-specified tasks defined by issue reports, specifications, or task tickets. Real-world custom

Language Is a Security Surface for Agent Guards

arXiv:2610.22244v1 Announce Type: cross Abstract: Step-level guards decide, before each tool call, whether an agent's next action is safe given the user's request and the trajectory so far. Users writ

ToolBottleneckBench: Measuring Tool-Agent Performance under Controlled Pressures and Fixed Harness Budgets

arXiv:2610.21098v1 Announce Type: cross Abstract: The success score of a tool agent reflects both its model and the budgets of the harness that runs it. We present ToolBottleneckBench, a benchmark tha

What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination

arXiv:2610.21229v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR) math corpora are routinely screened for benchmark overlap, but the reliability and data-loss costs

Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given

arXiv:2610.16911v1 Announce Type: new Abstract: Does providing additional context to automated AI auditors improve their effectiveness? Using the Petri tool to audit LLM coding agents, we test the imp

Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents

arXiv:2610.17659v1 Announce Type: new Abstract: Large language model (LLM) agents have shown strong capabilities in interactive problem solving, tool use, and long-horizon task execution. However, the

Cowork-Gym: Synthesizing Interactive Work Environments at Scale

arXiv:2610.17670v1 Announce Type: new Abstract: Cowork agents navigate dependencies across tools and files while adapting to changing user requirements and edits to shared workspaces. Scaling such env

FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

model-releasesarxiv-cs-lg

arXiv:2610.19240v1 Announce Type: new Abstract: Financial disclosures express the underlying relationships of double-entry accounting through unstructured text. While financial LLM benchmarks predomin

Google Earth AI Geospatial Reasoning Agent: Capabilities and a Systematic Evaluation

arXiv:2610.15750v1 Announce Type: new Abstract: Google Earth AI is a family of geospatial AI models, together with an agent that orchestrates them. We present this agent, Google Earth AI's Geospatial

Loose Lips: Understanding Privacy Leakage in Agentic Dialogue

arXiv:2610.14071v1 Announce Type: cross Abstract: As LLM agents are increasingly deployed in high-stakes enterprise settings, understanding how they handle sensitive information in agentic dialogue ha

Measuring Epistemic Humility in Multimodal Large Language Models

arXiv:2509.09658v3 Announce Type: replace Abstract: Hallucinations in multimodal large language models (MLLMs), where a model generates content inconsistent with the input image, pose significant risk

PerSo: Investigating Facets of LLMs' Social Intelligence in Real-World Persuasion Scenarios

arXiv:2610.20548v1 Announce Type: new Abstract: Understanding the social intelligence of large language models (LLMs) is increasingly important for applications involving human-AI interaction and mult

ProtoArena: Can AI Agents Discover and Exploit Real-World Cross-Component Protocol Vulnerabilities?

arXiv:2610.17837v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly used for vulnerability analysis, yet existing benchmarks provide limited coverage of exploitation t

Recursive Environment Scaling for General Long-Horizon Agents

arXiv:2610.16397v1 Announce Type: new Abstract: We study environment scaling as a way to increase agent-benchmark difficulty while holding core task objectives fixed. Our framework, Recursive Environm

Releasing the Inner Jev: A Simple Recipe for Open Decision Models

arXiv:2610.17205v1 Announce Type: new Abstract: LLM-based decision-making systems, also called System One models, have recently gained popularity, most notably through Jev (Almeida, 2026). Given a typ

Language2Drive: Do Vision-Language-Action Models Really Follow the Instructions in Autonomous Driving?

arXiv:2610.16634v1 Announce Type: cross Abstract: Recent Vision-Language-Action (VLA) models increasingly use natural-language instructions to condition autonomous driving behavior. Yet existing close

14 Oct 2026

Rethinking LLM-Guided Visual Question Answering: Does Deeper Knowledge Integration Really Help?

arXiv:2610.17079v1 Announce Type: new Abstract: Knowledge-based Visual Question Answering (VQA) requires reasoning that extends beyond visible image content, demanding external commonsense and factual

14 Oct 2026

Web Scale on a Budget: Tail-Aware Retrieval Evaluation

arXiv:2610.17986v1 Announce Type: new Abstract: A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability. One prominent example is ILIAS,

14 Oct 2026

Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

arXiv:2610.13152v1 Announce Type: new Abstract: Open coding agents are commonly trained through supervised fine-tuning on interaction trajectories whose final patches pass the benchmark test suite. Ho

12 Oct 2026

SpireBench: Evaluating Long-Horizon Strategic Play of Agentic Models with Slay the Spire

arXiv:2610.12596v1 Announce Type: new Abstract: Agentic large language models now tackle complex tasks, yet strategic reasoning over long horizons is under-studied: most agentic benchmarks are short a

12 Oct 2026

WoundAmbit: Bridging State-of-the-Art Semantic Segmentation and Real-World Wound Care

arXiv:2504.06185v3 Announce Type: replace-cross Abstract: Chronic wounds affect a large population, particularly the elderly and diabetic patients, who often exhibit limited mobility and co-existing h

12 Oct 2026

Generative Adversarial Loops

arXiv:2610.11458v1 Announce Type: cross Abstract: AI research progress can be viewed as the interaction between two processes: benchmark creation and method discovery. Historically, both were driven b

9 Oct 2026

Large-scale Repository Engineering via Agent-Native Reusable Code Primitives

arXiv:2610.09079v1 Announce Type: cross Abstract: Large language models equipped with development environments have moved code generation toward repository-scale construction, yet building complete re

8 Oct 2026

ToolRACER: A Robust Agentic Conversation Emulation Resource for Agent Training and Evaluation

arXiv:2610.09163v1 Announce Type: new Abstract: Task-oriented conversational agents remain fragile under real world conversation scenarios as they rarely follow a predictable script, especially when u

8 Oct 2026

A Systematic Study of Small Language Models on Abstract Reasoning Tasks

arXiv:2610.08680v1 Announce Type: cross Abstract: Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-speci

Logbook: Extremely Long-form Audio Event Understanding

arXiv:2610.07338v1 Announce Type: cross Abstract: Audio benchmarks are built around short, pre-segmented clips, limiting model design to brief inputs or fixed vocabularies. To close this gap, we intro

PlotPick: AI-powered batch extraction of numerical data from scientific figures

arXiv:2605.06021v2 Announce Type: replace Abstract: Systematic reviews and meta-analyses often need numerical data reported only in figures, and extracting them with interactive digitisers usually req

SWE-Game: Can Coding Agents Build the Games We Want?

arXiv:2609.33678v2 Announce Type: replace Abstract: We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five t

The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory

arXiv:2610.07309v1 Announce Type: new Abstract: Long-term-memory agents can retrieve relevant information that is inadmissible for the current request because it belongs to another principal, violates

Environmental sensor readings in two crop disease image datasets identify the session in which each image was taken

arXiv:2610.06369v1 Announce Type: new Abstract: Integrating environmental sensor data with leaf imagery is widely reported to boost crop disease classification accuracy. In this work, we reveal that t

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

arXiv:2610.04139v1 Announce Type: new Abstract: Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language mode

MagiC: Evaluating Multimodal Cognition Toward Grounded Visual Reasoning

arXiv:2507.07297v2 Announce Type: replace Abstract: Recent advances in large vision-language models have led to impressive performance in visual question answering and multimodal reasoning. However, i

OGAM: Connecting Systematic Testing to Runtime Assurance through Object-Grounded Attention Monitoring for VLA Policies

model-releasesarxiv-cs-lg

arXiv:2610.05878v1 Announce Type: new Abstract: Benchmarks expose vision-language-action (VLA) policies to few canonical instructions, while exhaustive deployment testing is impossible. We introduce O