Search: “benchmark”

Beyond Semantic Similarity: Reducing Unnecessary API Calls via Behavior-Aligned Retriever

arXiv:2508.14323v3 Announce Type: replace Abstract: Tool-augmented LLMs invoke external functions to extend their capabilities, but errors in the invocation decision, such as calling a tool when none

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning

DGX agent

arXiv:2608.30686v1 Announce Type: cross Abstract: Coding agents are increasingly used for software engineering tasks, including bootstrapping projects from third-party repositories whose integrity can

model-releasesarxiv-cs-cl

1 Sept 2026

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

arXiv:2608.30730v1 Announce Type: cross Abstract: Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies

FractureFields: Contact-Aware Binary Multi-Field Transfer for Fractured 3D Gaussian Simulation

arXiv:2608.28982v1 Announce Type: new Abstract: Physics-integrated 3D Gaussian representations make it possible to simulate image-reconstructed assets directly as particles, but current Gaussia-MPM pi

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

arXiv:2608.30672v1 Announce Type: new Abstract: Recent advances in large language models and multimodal models have pushed remote sensing (RS) processing from simple perception models to agentic syste

IndicDetect: Evaluating Cross-Lingual LLM-Generated Text Detection for Hindi, Telugu, and Tamil

arXiv:2608.29919v1 Announce Type: cross Abstract: The rapid proliferation of LLMs has further heightened the need to develop dependable AI-generated text detection, especially beyond English. Neverthe

More Capable, Less Faithful: A Multilingual Analysis of Mathematical (Un)Solvability Detection in LLMs

arXiv:2608.30463v1 Announce Type: new Abstract: Solvability detection is one of the most challenging aspects of mathematical reasoning for Large Language Models (LLMs). While prior work has studied th

nnMNet: Baseline for Martian Terrain Semantic Segmentation

arXiv:2608.29609v1 Announce Type: new Abstract: Semantic segmentation is a crucial task for understanding Mars, the most Earth-like planet in our solar system. However, it is challenging because the M

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

arXiv:2505.03654v3 Announce Type: replace-cross Abstract: Multimodal Large Language Models have shown strong performance across multimodal tasks, and recent personalized MLLMs can recognize user-speci

Reward-Oracle MCTS for Formal Theorem Proving: Sample-Efficient Search and the Need for Kernel-Level Proof Auditing

arXiv:2608.28639v1 Announce Type: new Abstract: Formal theorem proving with large language models remains challenging due to the difficulty of navigating large proof search spaces efficiently. Existin

Selection-Aware Stress Testing for Interactive Agents

arXiv:2608.30916v1 Announce Type: new Abstract: Agent evaluations often use one benchmark to choose a workflow and then search for task types where its advantage weakens, so both conclusions are selec

SVI2LoD3: Agent-Driven Reconstruction of LoD3 Facade Openings in Semantic 3D City Models from Volunteered Street View Imagery using Large Language and Visual Models

arXiv:2608.29992v1 Announce Type: new Abstract: This paper presents an end-to-end, agent-driven pipeline for the LoD3 reconstruction of facade openings in 3D city models, producing directly usable Cit

Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

arXiv:2603.04191v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as personal assistants, where users may share individual preferences over extended interaction

ToxLens: A Reproducible Graph-Learning Framework for Leakage-Aware, Uncertainty-Calibrated Molecular Toxicity Prediction

arXiv:2608.30472v1 Announce Type: new Abstract: Molecular toxicity prediction is increasingly used to prioritise compounds before experimental testing, but conventional benchmark performance can overs

When Patients Cut In: Extending Clinical Conversational AI Safety to Interruptions

arXiv:2608.29241v1 Announce Type: new Abstract: Clinical voice agents are now deployed in routine care, where real patients do not wait their turn: they interrupt. These systems typically use a cascad

What Do Interaction Representations Actually Measure? Pre-Event Separability in Weakly-Supervised Violence Detection

arXiv:2608.27879v1 Announce Type: new Abstract: Articulated human pose provides detailed body-configuration information beyond coarse spatial relationships, but whether this detail yields greater disc

31 Aug 2026

Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory

arXiv:2608.26807v1 Announce Type: cross Abstract: Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences. Existing agents either rely on e

Evaluating Language Models in Realistic Conversational Contexts

arXiv:2608.26131v1 Announce Type: new Abstract: As Large Language Models (LLMs) are increasingly deployed to serve open-ended, multi-turn interactions, evaluating conversational quality at human scale

GROUND: Reducing Hallucinations in LLM-Based Enterprise Analytics Through Governed Semantic Definitions

arXiv:2608.26157v1 Announce Type: new Abstract: Natural-language analytics over enterprise data warehouses is increasingly important, but production use is limited by hallucinated metrics, invalid joi

Agents

How Do LLM Agents Actually Get the Flag? Trace-Level Provenance for Agentic Offensive Security Evaluation

arXiv:2608.26237v1 Announce Type: cross Abstract: Capture-the-Flag (CTF) benchmarks are widely used to assess the offensive security capabilities of autonomous language-model agents. Evaluations rely

agentsarxiv-cs-ai

PAWBench: How Far Are We from Probabilistically Aligned World Modeling?

arXiv:2608.27345v1 Announce Type: cross Abstract: Recent video generation models are increasingly framed as world models. Many physical processes can unfold in more than one valid way. Therefore, a wo

ReViCo: Unveiling the Limitations of VLMs in Visual Text Understanding via Error Correction

arXiv:2608.27154v1 Announce Type: new Abstract: Vision Language Models (VLMs) have shown great success in general visual tasks, yet they still struggle to deeply understand text within images. In this

The Cold-Start Safety Gap in LLM Agents

arXiv:2606.07867v2 Announce Type: replace Abstract: Are tool-calling LLM agents equally safe throughout a conversation? We discover they are not: agents are most vulnerable at the very start of a sess

Evidence-Grounded Mapping of Multimodal Human Sensing Psychological Transdiagnostic Dimensions

arXiv:2608.24903v1 Announce Type: cross Abstract: Mobile and wearable sensing enables longitudinal observation of behavior, yet translating these signals into meaningful mental health constructs remai

27 Aug 2026

FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

arXiv:2608.25158v1 Announce Type: cross Abstract: Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate thi

27 Aug 2026

Video-IFBench: Evaluating Instruction Following of Multimodal LLMs in Video Understanding Scenarios

arXiv:2608.25529v1 Announce Type: new Abstract: Multimodal Large Language Models (MLLMs) have shown strong performance in video understanding. However, their ability to follow instructions in this dom

27 Aug 2026

Design-to-Plan: A Large Language Model-Based Multi-Agent Framework for Manufacturing Process Planning from 3D CAD Models and 2D Engineering Drawings

arXiv:2608.24039v1 Announce Type: cross Abstract: Manufacturing process planning transforms heterogeneous design information into coherent manufacturing decisions. However, existing approaches focus o

Gated Activation Steering for Reducing Sycophancy & Hallucination in Medical Question Answering

arXiv:2608.23666v1 Announce Type: new Abstract: Sycophancy and hallucination are persistent failure modes of Large Language Models (LLMs) across domains. However, it becomes particularly consequential

OmniJudge or OmniBias? Diagnosing Multimodal Judges through Balanced, Decoupled Lenses

arXiv:2608.24160v1 Announce Type: new Abstract: Multimodal understanding models that can jointly judge text-to-image (T2I), text-to-video (T2V) and text-to-speech (TTS) generation are increasingly use

QiMeng-ChipV-RTL: Exploiting Information Locality for IP-level Verilog Generation

arXiv:2602.00704v2 Announce Type: replace Abstract: The generation of Register-Transfer Level (RTL) code is a crucial yet labor-intensive step in digital hardware design, traditionally requiring engin

ClawProBench: Trace-Aware Evaluation of AI Agents with Runtime Coverage and Frozen Workplace-Style Holdouts

arXiv:2608.22510v1 Announce Type: new Abstract: Agent benchmarks often evaluate only final answers even when agents run on stateful runtimes. We argue this under-specifies what is being evaluated: the

25 Aug 2026

There Is No Neutral Harness: Modern LLM Leaderboards Are Manufactured by Config-Fragile Items

arXiv:2608.21382v1 Announce Type: new Abstract: Multiple-choice benchmarks fix the questions and the correct answers, but not the harness: the order of the options, the wording of the prompt, and whet

25 Aug 2026

Aggregate, Don't Adapt: Subject-Level Posterior Aggregation and Transductive Calibration for Cross-Site Parkinsonian Gait Severity

arXiv:2608.20587v1 Announce Type: cross Abstract: We describe the winning entry to the MoCha 2026 Benchmark and Challenge on Parkinsonian Gait, which predicts MDS-UPDRS gait severity from canonicalize

Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

arXiv:2608.20797v1 Announce Type: new Abstract: Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. H

Personalized Privacy Control in LLMs via Attention Head Intervention

arXiv:2608.21209v1 Announce Type: new Abstract: The rise of agentic AI enables LLMs to access diverse user data, raising critical privacy concerns. Prior work on contextual privacy studies whether LLM

RiskTraf: Risk-Extrapolated Residual Learning for Multi-Variate Traffic Flow Prediction

arXiv:2608.20656v1 Announce Type: cross Abstract: Traffic sensors commonly record flow, speed, and occupancy, but standard traffic flow forecasting benchmarks and models rarely exploit all three raw m

SlidesGen-Bench: Evaluating Slides Generation via Computational and Quantitative Metrics

arXiv:2601.09487v2 Announce Type: replace Abstract: The rapid evolution of Large Language Models (LLMs) has fostered diverse paradigms for automated slide generation, ranging from code-driven layouts

CAViAR: A Causal Video Dataset for Fine-Grained Accident Reasoning in Real-World Scenarios

arXiv:2608.19380v1 Announce Type: new Abstract: While modern autonomous driving systems excel at perception tasks such as object detection and trajectory prediction, they lack the high-level causal re

21 Aug 2026

PL-NBA: A Possession-level Universal Basketball Video Dataset Supporting Multiple Visual Understanding Tasks

arXiv:2608.19646v1 Announce Type: new Abstract: Visual understanding in sports has emerged as a hot topic in computer vision in recent years. Most existing basketball video datasets adopt single actio

21 Aug 2026

XDen-1K: A Density Field Dataset of Real-World Objects

arXiv:2512.10668v2 Announce Type: replace Abstract: A deep understanding of the physical world is essential for robotic manipulation and physically realistic simulation. While current methods, includi

21 Aug 2026

CL4D: Contrastive Language-4D Pretraining for Vision-Language Reasoning in Dynamic Scenes

arXiv:2608.18734v1 Announce Type: new Abstract: 4D understanding and reasoning is a fundamental capability for embodied AI agents operating in dynamic physical environments. However, existing vision e

20 Aug 2026

Grouping the Stochastic Machine: Precision, Not Capability, as the Frontier Metric for AI Systems

arXiv:2608.19140v1 Announce Type: new Abstract: Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve. I argue this measures t

20 Aug 2026

ControlledShifts: Towards Standardizing Robustness Evaluation in Trajectory Prediction Under Distribution Shifts

model-releasesarxiv-cs-ro

arXiv:2608.17882v1 Announce Type: new Abstract: Trajectory prediction is central to safety in autonomous driving, yet learning-based predictors tend to degrade sharply when encountering scenarios poor

EvoTS-Agent: A Self-Evolving LLM Agent for Financial Time Series Change Point Detection

arXiv:2608.17933v1 Announce Type: new Abstract: Financial time series exhibit non-stationary and heterogeneous statistical properties, making change-point detection challenging because no single unsup

Exploring Efficient Open-Vocabulary Segmentation in the Remote Sensing

arXiv:2509.12040v3 Announce Type: replace-cross Abstract: Open-Vocabulary Remote Sensing Image Segmentation (OVRSIS), an emerging task that adapts Open-Vocabulary Segmentation (OVS) to the remote sens

TSQueryBench: LLM-as-a-Judge for Time Series Explanations

arXiv:2604.02118v2 Announce Type: replace Abstract: Natural language explanations of time series data are increasingly produced by foundation models in high stakes domains, making factual correctness

Augmenting Text to Increase Translation Difficulty

arXiv:2608.15932v1 Announce Type: new Abstract: As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models

18 Aug 2026

Beyond Accuracy: Assessing Calibration of Geospatial Foundation Models and Their Sensitivity to Distribution Shifts

arXiv:2608.16614v1 Announce Type: new Abstract: Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show tha