Search: “benchmark”

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

arXiv:2610.13361v1 Announce Type: cross Abstract: As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their us

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

DGX agent

arXiv:2610.13022v1 Announce Type: cross Abstract: As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users ofte

model-releasesarxiv-cs-ai

12 Oct 2026

LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge

arXiv:2506.09956v2 Announce Type: replace-cross Abstract: Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the pra

MAGMA-Bench: Evaluating Long-Horizon Embodied Agents under Dynamic Tasks and Execution Failures

arXiv:2610.13534v1 Announce Type: new Abstract: Long-horizon embodied agents must maintain task-relevant information, coordinate heterogeneous robots, and adapt their decisions as execution unfolds. E

Measuring Artifact Reliance in Machine-Generated Text Detectors

arXiv:2610.13583v1 Announce Type: new Abstract: Detectors of machine-generated text score almost perfectly on scientific-literature benchmarks, implying we can reliably distinguish with these detector

researcharxiv-cs-cl

perturber: A Stateless Service for Semantics-Preserving Text Perturbations

arXiv:2610.12667v1 Announce Type: new Abstract: Large language model (LLM) evaluations on clean benchmarks overstate model robustness: real inputs arrive perturbed, and these meaning-preserving varian

RoleWorld: Knowledge Transfer in World Modeling with Functional Role Abstractions

arXiv:2610.13172v1 Announce Type: new Abstract: World models enable large language model (LLM) agents to anticipate action effects and make informed decisions. For generalization, it is desirable for

Same Query, Different Answer: Run-to-Run Variance in Semantic Query Processing

arXiv:2610.13823v1 Announce Type: cross Abstract: Semantic query engines ask a large language model (LLM) to judge individual rows, and benchmarks compare these engines on the quality of a few executi

researcharxiv-cs-ai

When VLMs Destroy What They Know: Spatial Representations Fall Below Lexical Priors

arXiv:2610.13910v1 Announce Type: new Abstract: Vision-Language Models (VLMs) increasingly analyze and reason about images, yet it remains unclear whether their internal representations faithfully enc

Evaluating Exact Output and Checkpoint-State Prediction in Real Programs

arXiv:2610.11889v1 Announce Type: cross Abstract: We present a benchmark for predicting final output and checkpoint state from source and input alone. It extends CRUXEval-style output prediction with

9 Oct 2026

$OneMillion-Bench: How Far are Language Agents from Human Experts?

arXiv:2603.07980v2 Announce Type: replace-cross Abstract: As language models (LMs) evolve from chat assistants to long-horizon agents capable of multi-step reasoning and tool use, existing benchmarks

9 Oct 2026

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

arXiv:2610.12402v1 Announce Type: cross Abstract: Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intel

9 Oct 2026

BEACON-SP: Ontology-Grounded GraphRAG Framework for Clinical Suicide Risk Assessment

arXiv:2610.09026v1 Announce Type: cross Abstract: We present BEACON-SP, an ontology-grounded Graph Retrieval-Augmented Generation (GraphRAG) framework for clinician-facing decision support in behavior

8 Oct 2026

Kuration SDK: Addressing the Virtual2Real Gap via Data Curation

arXiv:2610.09305v1 Announce Type: cross Abstract: Benchmarks for measuring the quality of action-conditioned world models are still evolving and shifting away from visual similarity-based metrics to a

researcharxiv-cs-cv

8 Oct 2026

Rubric Spans are Label Representations: Joint LLM Encoding for Short Answer Scoring

arXiv:2610.09660v1 Announce Type: new Abstract: Automatic Short Answer Scoring (ASAS) requires models that can score student responses against question-specific criteria while remaining efficient and

8 Oct 2026

Artifact removal improves electrodermal waveforms but not downstream classification in a virtual-reality balance task

model-releasesarxiv-cs-lg

arXiv:2610.07438v1 Announce Type: cross Abstract: Artifact removal routinely precedes the classification of electrodermal activity (EDA), on the assumption that a cleaner signal supports a better deci

BluffJAX: Adversarial Imperfect Information Games in JAX

arXiv:2610.07686v1 Announce Type: new Abstract: We introduce BluffJAX: an open-source suite of adversarial imperfect information games in JAX. We provide canonical implementations of games designed fo

DAEDALUS: Bootstrapping Agent Memory from Self-Generated Tasks

arXiv:2610.08048v1 Announce Type: new Abstract: LLM agents often lack the operational knowledge to act reliably in new environments, as they must discover specific tool behaviors or environment conven

Do Higher-Order Models Win for Higher-Order Reasons? Rethinking Performance Gains in Hypergraph Learning

arXiv:2610.07981v1 Announce Type: new Abstract: Higher-order models (e.g., hypergraph neural networks) often outperform lower-order baselines on hypergraph learning benchmarks, and their advantages ar

researcharxiv-cs-lg

DSV-Mem: Evaluating Multimodal Memory in Professional Workflows for MLLM Agents

arXiv:2610.08102v1 Announce Type: new Abstract: Conversational MLLM agents are increasingly expected to assist in professional workflows, from AI research and engineering design to product management

EPOCH: Reliable Discovery through Evidence-Governed Search

arXiv:2610.06986v1 Announce Type: new Abstract: AI research agents are increasingly used to search over programs, mathematical constructions, and proofs. However, existing systems typically optimize e

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

arXiv:2610.06910v1 Announce Type: new Abstract: Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in web front-end execution, with browser-based game generation

Leveraging a four-quadrant approach for evaluating Redpine Science

arXiv:2610.07937v1 Announce Type: new Abstract: Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Pr

Multigroup Fairness and Omniprediction: Separations and Equivalences

model-releasesarxiv-cs-lg

arXiv:2610.07374v1 Announce Type: new Abstract: Omniprediction is a learning guarantee which requires a single predictor to be competitive relative to the best hypothesis from a benchmark class for an

Natural Language Questions as an Interface for Knowledge Graphs: QRAKEN Graph Distillation and Semantic Self-Healing

arXiv:2610.08095v1 Announce Type: new Abstract: Natural-language access to RDF knowledge graphs is a core Semantic Web ambition. Large language models (LLMs) have advanced Text-to-SPARQL, yet on unfam

Quantifying the Gap between Understanding and Generation within Unified Multimodal Models

arXiv:2602.02140v2 Announce Type: replace Abstract: Recent advances in unified multimodal models (UMM) have demonstrated remarkable progress in both understanding and generation tasks. However, whethe

ServeLearnBench: How Well Can Agents Self-Improve from Serving Experience?

arXiv:2610.07792v1 Announce Type: cross Abstract: Large language model agents are increasingly deployed to perform complex tasks in real-world environments. However, the knowledge required for correct

World Models' Last Exam in Physics

arXiv:2610.08791v1 Announce Type: new Abstract: Video world models can produce visually convincing yet physically inconsistent sequences, raising concerns about their reliability for prediction and pl

Analysis of SWIR Imaging Detection Performance Under Adverse Environmental Conditions for Autonomous Driving Systems

arXiv:2610.06596v1 Announce Type: new Abstract: Short-wave infrared (SWIR) imaging has emerged as a promising modality for autonomous driving, yet its practical benefits over RGB remain poorly charact

Answer with Evidence: Consistency-Aware Grounded Visual Question Answering for Roadside Traffic Scenes

arXiv:2610.05274v1 Announce Type: new Abstract: Roadside traffic reasoning requires every free-form textual claim to be backed by visual evidence. Existing grounded multimodal large language models (M

Are We Measuring Anticipation? Auditing Privileged Information in Procedural Video Evaluation

arXiv:2610.03826v1 Announce Type: new Abstract: Benchmark scores license claims about the capabilities being evaluated. We audit the inference licensed by an evaluation protocol, rather than the predi

Beyond Plausibility: Verifiable Fine-Grained Image Editing on Structured Assets

arXiv:2610.04381v1 Announce Type: new Abstract: Fine-grained image editing requires more than producing a visually plausible result: an editor must execute the requested attribute change precisely whi

MTOR: Generalizable AI-Generated Video Detection with Multimodal Semantics and Temporal Over-Regularity

arXiv:2610.06378v1 Announce Type: new Abstract: The rapid evolution of video generation has narrowed the perceptual gap between authentic and synthetic videos, making generalizable AI-generated video

On Hyperparameter Tuning on the Test Set

arXiv:2610.05902v1 Announce Type: cross Abstract: 'Don't tune hyperparameters on the test set' is often stated in machine learning textbooks. Violating it is considered a cardinal sin that produces mi

AptMQL-Bench: From Text-to-SQL to Text-to-MQL via Access-Pattern Schema Design and Data-Preserving Migration

arXiv:2610.02770v1 Announce Type: new Abstract: Document databases such as MongoDB are core infrastructure for modern applications, and natural-language interfaces to them---text-to-MQL---would let no

HazardWeaver: Scientific Route Selection for Hazard Analysis Agents

arXiv:2610.03591v1 Announce Type: new Abstract: Understanding and assessing natural hazards is essential for disaster preparedness and risk reduction. Recent advances in large language models have spu

Ontological Instability and Statistical Amplification: The Paradox of 'Humanizing' LLM-Generated Text

arXiv:2610.03110v1 Announce Type: new Abstract: Supervised AI-text detectors report high benchmark accuracy, but it is not clear what their decisions are based on. We analyze a RoBERTa-based detector

Output Language Confusion under Multilingual Prompt Contamination

arXiv:2610.02926v1 Announce Type: new Abstract: Standard factual benchmarks assume clean monolingual prompts and exact-match scoring, two assumptions that break simultaneously in real-world multilingu

Seeing, Saying, but Not Using: From Reportable Spatial Facts to Usable States in Multimodal Large Language Models

arXiv:2610.02876v1 Announce Type: new Abstract: A multimodal large language model that correctly reports a spatial fact does not necessarily use that fact in subsequent reasoning. To study this distin

Trading Strategy Optimization via Textual Gradient

arXiv:2610.03128v1 Announce Type: new Abstract: Quantitative trading strategy design aims to discover trading programs from historical data that remain effective in future markets, which can be viewed

Understanding Clinical Cognitive Dialogues Using Large Language Models

arXiv:2609.34125v2 Announce Type: replace Abstract: In-person cognitive assessment is both a test and an interaction. Clinicians explain tasks, repair misunderstandings, and adapt to patient responses

Decoding the Disaster: Multi-Task Geospatial Reasoning with Vision-Language Models and Crowdsourced Imagery for Disaster Mapping

arXiv:2610.00302v1 Announce Type: cross Abstract: Crowdsourced imagery provides timely, fine-grained, street-level observations for disaster mapping, complementing conventional remote sensing imagery

Evaluating the Robustness of Anti-UAV Detection under Controlled Fog Degradation: Fog-Aware Training and Clear-Sky Tradeoff

arXiv:2610.00141v1 Announce Type: new Abstract: Vision-based anti-UAV systems must function in poor visibility, yet most benchmarks use only clear-sky footage, and previous robustness studies treat ad

False Floors: LLM Safety Routing Evaluations Break Under Distribution Shift

arXiv:2610.01535v1 Announce Type: cross Abstract: Safety routers send each request to one of several models and are judged against the best single model. A major routing benchmark picks that comparato

GIFTBench: Diagnosing Generalization in Image Forgery Localization and Informing Model Design

arXiv:2610.01778v1 Announce Type: new Abstract: Reliable evaluation of image forgery localization (IFL) requires assessing models under diverse distribution changes, yet existing benchmarks often cove

Is Success All You Need? Investigating the Impact of Input Perturbations on VLA Behaviour in Tabletop Manipulation Tasks

model-releasesarxiv-cs-ro

arXiv:2610.01351v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models have achieved high task success rates on robot manipulation task benchmarks. More recently, there has been an emphas

NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

arXiv:2610.01461v1 Announce Type: new Abstract: We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do

VISTA: A Visual Harness for Reasoning in an Interactive World

arXiv:2610.02200v1 Announce Type: new Abstract: We show that multimodal models possess strong reasoning abilities and that an appropriate harness can unlock their potential to solve tasks across diver