Search: “benchmark”

VeriHarness: Scaling Agentic Verification for Long-Horizon Tasks

arXiv:2610.00972v1 Announce Type: new Abstract: As LLM agents undertake increasingly complex, long-horizon tasks, verifying their outputs becomes increasingly challenging. We study how verification ca

2 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

VideoSTF: Stress-Testing Output Repetition in Video Large Language Models

DGX agent

arXiv:2602.10639v2 Announce Type: replace Abstract: Video Large Language Models (VideoLLMs) have achieved strong performance on video understanding tasks, yet existing benchmarks evaluate only what mo

researcharxiv-cs-cv

2 Oct 2026

Safety

BARRAC: Adaptation of an English Aspect-based Sentiment Analysis Approach for Classification Tasks in Arabic Dialects

arXiv:2609.38820v1 Announce Type: cross Abstract: With the rapid growth of Arabic NLP, several models, datasets and benchmarks have been reported. This paper asks whether approaches developed for majo

safetyarxiv-cs-ai

Behavior-Grounded Semantic Enrichment for Financial Fraud Modeling and Reasoning

arXiv:2609.34211v2 Announce Type: replace Abstract: In financial fraud detection, rich semantic context can provide important evidence for transaction behavior modeling and fraud reasoning. However, p

Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models

arXiv:2609.38485v1 Announce Type: new Abstract: Unified multimodal models (UMMs) train image understanding and autoregressive image generation on shared parameters, and the two objectives are known to

Beyond Model Ranking: Regime Diagnosis for Distributional-Statistical Misspecification in Industrial Time-Series Forecasting

arXiv:2609.40117v1 Announce Type: new Abstract: Time-series forecasting models achieve strong benchmark performance but exhibit severe systematic bias in industrial deployments. This train--deploy gap

From Compound Figures to Medical Multi-image Reasoning: Scaling Multimodal Large Language Models with Biomedical Literature

arXiv:2511.22232v2 Announce Type: replace-cross Abstract: Multimodal large language models (MLLMs) are increasingly capable in medical imaging, yet most focus on single-image settings. Clinical interp

From Given to Gathered Evidence: Agentic Learning for Longitudinal Medical Reasoning

arXiv:2609.39566v1 Announce Type: new Abstract: Foundation models can serve as clinical agents through tool-use harnesses. However, conventional medical benchmarks assess reasoning over preselected ev

LLM Persona Unlearning

model-releasesarxiv-cs-cl

arXiv:2609.39882v1 Announce Type: new Abstract: Pre-training equips large language models (LLMs) with a broad repertoire of behavioral patterns associated with roles, styles, values, and goals. Post-t

Agents

MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories

arXiv:2609.40195v1 Announce Type: cross Abstract: Long-term egocentric video enables personalized AI assistants to reason about daily life. However, as video histories grow to hundreds of hours spanni

agentsarxiv-cs-ai

Right Answers, Costly Models: The Efficiency Gap in LLM-based Optimization Modeling

arXiv:2609.38884v1 Announce Type: cross Abstract: Optimization modeling formulates real-world decision problems as mathematical programs that solvers can use to find optimal decisions. Large language

TrafficSignBench: Rule-Centric Closed-Loop Evaluation of Traffic-Sign Compliance in Autonomous Driving

arXiv:2609.38463v1 Announce Type: cross Abstract: Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not

Uruqi: Learning Spatial Cognition from Visual Experience

arXiv:2609.39195v1 Announce Type: new Abstract: Spatial intelligence requires maintaining a coherent understanding of the world as the embodied agent moves. Like humans, the agent must use its own mot

Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos

model-releasesarxiv-cs-ro

arXiv:2609.38966v1 Announce Type: new Abstract: We present Video2SwimFish, an automated pipeline and benchmark for building controllable fish assets from real-fish videos for underwater embodied AI. G

An Exact Generate - Transform Decomposition of Small-LLM Team Scaling Across Orchestration Architectures

arXiv:2609.36104v1 Announce Type: new Abstract: Replacing one LLM agent with a collaborating team can raise accuracy, but whether scaling the team helps, and which architecture to scale, is unclear. S

Constructing Challenging Browser-Use Tasks by Controlled Environment Interventions

arXiv:2609.35814v1 Announce Type: cross Abstract: As browser-use agents improve, benchmarks keep pace by collecting new tasks, websites, and applications, often making tasks longer or more novel. This

Do LLM Agents Execute the Plans They Declare? From Planning-Mode Declaration to Pattern-Specific Execution

arXiv:2609.38108v1 Announce Type: new Abstract: Large language models (LLMs) enable agents to solve long-horizon tasks by generating a plan and then executing it in an environment. However, successful

Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessment

arXiv:2609.37576v1 Announce Type: new Abstract: With the rapid advancement of text-to-image (T2I) generation, robust evaluation becomes critical yet challenging, as traditional metrics fail to capture

Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

arXiv:2609.37470v1 Announce Type: new Abstract: A probability used for a decision should refer to the same event across equivalent requests. We introduce probability contracts, a benchmark connecting

RunyaNER: Auxiliary Language Selection for Runyankore NER

model-releasesarxiv-cs-cl

arXiv:2609.37543v1 Announce Type: new Abstract: Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resou

ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

arXiv:2609.34547v1 Announce Type: new Abstract: Video-capable vision-language models score above 80% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

arXiv:2609.33303v1 Announce Type: new Abstract: Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing

HM-ROUTER: Joint Model and Harness Routing for Agentic Systems

arXiv:2609.32213v1 Announce Type: new Abstract: Agent performance depends on both the underlying model and the harness that manages its tool use and execution. Selecting a suitable pair requires accou

Research

Mixed-Prior Decision Risk for Open-Set Recognition

arXiv:2609.35043v1 Announce Type: new Abstract: In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexis

researcharxiv-cs-cv

PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

arXiv:2609.34314v1 Announce Type: new Abstract: Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

arXiv:2609.32734v1 Announce Type: new Abstract: AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allo

Recipe-Matching, Not Equivalence

arXiv:2609.31927v1 Announce Type: cross Abstract: MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold doc

SAGE: Symbolic Action-Gating and Editing for LLM Task Planners

model-releasesarxiv-cs-ro

arXiv:2609.34268v1 Announce Type: new Abstract: Large language models (LLMs) are now the default cognitive core of embodied household agents, yet the plans they emit are rarely checked against a groun

ScreenHaystack: Finding Blind Zones in GUI Grounding

arXiv:2609.32036v1 Announce Type: new Abstract: We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each targe

Evaluating Cultural Awareness of LLMs for Haitian Creole

arXiv:2609.31506v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they

Agents

Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes

arXiv:2609.30798v1 Announce Type: new Abstract: Real-time voice agents have moved from research prototypes to production deployments, yet the literature describing them is fragmented across three comm

agentsarxiv-cs-ai

LUCID: Learning Under Confounding for Inference and Discovery in Time Series

arXiv:2609.31315v1 Announce Type: new Abstract: Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct e

MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

arXiv:2609.30967v1 Announce Type: new Abstract: Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model tha

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

arXiv:2609.31148v1 Announce Type: new Abstract: Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply wh

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

arXiv:2609.30981v1 Announce Type: new Abstract: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

arXiv:2609.31024v1 Announce Type: cross Abstract: Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audi

TemplateCraft: Agentic Visual Template Generation

arXiv:2609.31451v1 Announce Type: cross Abstract: The growing popularity of short videos has driven demand for one-click content creation. Visual templates turn uploaded images into personalized conte

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

arXiv:2609.30709v1 Announce Type: cross Abstract: Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretati

Agents

Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

arXiv:2609.28991v1 Announce Type: cross Abstract: Video understanding is increasingly performed by multi-stage LLM agents that separate temporal grounding, visual observation, and reasoning. Yet these

agentsarxiv-cs-ai

Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles

arXiv:2609.29974v1 Announce Type: new Abstract: Estimating heterogeneous treatment effects from observational data is difficult because the most appropriate inductive bias varies with overlap, treatme

EnigmaForge: The Question Is Hidden in the Story

arXiv:2609.30144v1 Announce Type: new Abstract: Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and lo

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

model-releasesarxiv-cs-cl

arXiv:2609.29573v1 Announce Type: new Abstract: Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that disto

Task-Resolved Fisher Spectroscopy for Quantum Reservoir Computing

arXiv:2609.29570v1 Announce Type: cross Abstract: Quantum reservoir computing (QRC) uses fixed quantum dynamics to encode a time series and trains only a classical readout, but a benchmark capacity al

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

arXiv:2609.28725v1 Announce Type: cross Abstract: Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models lear

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

arXiv:2609.28335v1 Announce Type: new Abstract: Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is

CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments

arXiv:2609.27273v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly act on behalf of users online. What happens when the environments they operate in have incentives that do not al

Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution

arXiv:2609.26952v1 Announce Type: new Abstract: Dependency conflicts in Python ecosystems arise from incompatible version constraints, missing packages, and undocumented compatibility relationships, c

Learning the Cost of Reliable Inference

arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on th