Search: “benchmark”

Grid2Matrix: Revealing Digital Agnosia in Vision-Language Models

arXiv:2604.09687v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) excel on many multimodal reasoning benchmarks, but these evaluations often do not require an exhaustive readout of the i

14 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

4D-RGPT: Toward Region-level 4D Understanding via Perceptual Distillation

DGX agent

arXiv:2512.17012v3 Announce Type: replace Abstract: Despite advances in Multimodal LLMs (MLLMs), their ability to reason over 3D structures and temporal dynamics remains limited, constrained by weak 4

model-releasesarxiv-cs-cv

13 Apr 2026

CAD 100K: A Comprehensive Multi-Task Dataset for Car Related Visual Anomaly Detection

arXiv:2604.09023v1 Announce Type: new Abstract: Multi-task visual anomaly detection is critical for car-related manufacturing quality assessment. However, existing methods remain task-specific, hinder

13 Apr 2026

EigentSearch-Q+: Enhancing Deep Research Agents with Structured Reasoning Tools

arXiv:2604.07927v2 Announce Type: replace Abstract: Deep research requires reasoning over web evidence to answer open-ended questions, and it is a core capability for AI agents. Yet many deep research

13 Apr 2026

AgentOpt v0.1 Technical Report: Client-Side Optimization for LLM-Based Agent

arXiv:2604.06296v1 Announce Type: cross Abstract: AI agents are increasingly deployed in real-world applications, including systems such as Manus, OpenClaw, and coding agents. Existing research has pr

ClawsBench: Evaluating Capability and Safety of LLM Productivity Agents in Simulated Workspaces

arXiv:2604.05172v2 Announce Type: replace Abstract: Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evalu

FORGE:Fine-grained Multimodal Evaluation for Manufacturing Scenarios

arXiv:2604.07413v1 Announce Type: new Abstract: The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution,

k-server-bench: Automating Potential Discovery for the k-Server Conjecture

arXiv:2604.07240v1 Announce Type: cross Abstract: We introduce a code-based challenge for automated, open-ended mathematical discovery based on the k-server conjecture, a central open problem in com

Matrix Profile for Anomaly Detection on Multidimensional Time Series

arXiv:2409.09298v2 Announce Type: replace-cross Abstract: The Matrix Profile (MP), a versatile tool for time series data mining, has been shown effective in time series anomaly detection (TSAD). This

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale

arXiv:2604.06814v1 Announce Type: cross Abstract: While traditional tree-based ensemble methods have long dominated tabular tasks, deep neural networks and emerging foundation models have challenged t

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604.07343v1 Announce Type: cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a cen

VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

arXiv:2604.07634v1 Announce Type: new Abstract: Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core

AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations

arXiv:2610.21624v1 Announce Type: new Abstract: Real-world tasks are often complex and noisy, posing substantial challenges for LLM agents. Yet existing evaluations lack a systematic framework for joi

ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing

arXiv:2610.21581v1 Announce Type: new Abstract: Mathematical agents are commonly evaluated by answer accuracy or proof acceptance. We introduce ClaimScope, a 72-task benchmark for auditing proposed cl

Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

arXiv:2610.22043v1 Announce Type: cross Abstract: Which inputs drive a language model's financial forecast? We construct synthetic earnings announcements that independently reverse 20-day momentum, no

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

arXiv:2610.21380v1 Announce Type: new Abstract: Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty

Prism-1: Position- and Rival-Independent Scoring for System One Models

arXiv:2610.22066v1 Announce Type: new Abstract: Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in o

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

arXiv:2610.21614v1 Announce Type: cross Abstract: Large audio language models (LALMs) have made substantial progress in audio understanding. However, in high-throughput applications requiring frequent

Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

arXiv:2610.21135v1 Announce Type: cross Abstract: LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves

SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

arXiv:2610.20038v2 Announce Type: replace-cross Abstract: Historical code changes provide convenient task boundaries, but do not identify which production functions selected tests exercise or support

Verdict: Encode Once, Decide Many

arXiv:2610.20778v1 Announce Type: cross Abstract: Automating organizational work means answering many small typed questions about one piece of state (does this email need a reply, how urgent is it, wh

AgentStripes: An Evidence-Graded Maturity Model for AI Agents

arXiv:2610.18603v1 Announce Type: new Abstract: Benchmarks report what an Artificial Intelligence (AI) agent can do on a task suite, but not whether one released version has been engineered with the c

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these inform

Calibrated Typed Decisions for Entity, Relation, and Event Extraction

arXiv:2610.17113v1 Announce Type: cross Abstract: Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remain

claimcheck: Measuring and Improving Specification Faithfulness

arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author meant

CommerceGen-Studio: An End-to-End Framework for Commercial-Grade Main-Image Generation

arXiv:2610.20603v1 Announce Type: new Abstract: We present CommerceGen-Studio, an end-to-end framework for generating commercial-grade main images for e-commerce Stock Keeping Units (SKUs). Our framew

CompanionSafe: From Multi-turn Safety Evaluation to Guardrails for AI Companions

arXiv:2610.18095v1 Announce Type: new Abstract: AI companions provide users with emotional support and ongoing companionship, but their sustained relational interactions also introduce distinct safety

CWE-bench: Refocusing Agentic Security from Exploitation to Defense

arXiv:2610.17020v1 Announce Type: cross Abstract: Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An a

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

arXiv:2610.17256v1 Announce Type: new Abstract: Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each an

DynThinker: Unifying Static 3D and Dynamic 4D Spatial Reasoning

arXiv:2610.20598v1 Announce Type: new Abstract: Achieving general spatial intelligence requires a unified understanding of both static 3D and dynamic 4D. However, we find that existing models struggle

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fin

PsyMemBehav: Psychology-Informed and Memory-Grounded Two-Level Identity-to-Action Reasoning for Social-Media Behavior Simulation

arXiv:2610.14570v1 Announce Type: new Abstract: Large language models (LLMs) support social simulation, but faithfully reproducing real users requires grounding both structured platform decisions and

Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate

arXiv:2610.15710v1 Announce Type: new Abstract: Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair a

Unconstrained CTI-REALM with Open-Weights

arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and f

What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the ag

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

model-releasesarxiv-cs-cl

arXiv:2610.20195v1 Announce Type: new Abstract: Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing e

A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?

arXiv:2610.15562v1 Announce Type: new Abstract: Large-scale video generation models are increasingly viewed as a promising path toward world models, yet their ability to capture causally coherent phys

Auditing Failure Attribution: Label Predictability and Judge Behavior

arXiv:2610.17747v1 Announce Type: cross Abstract: Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step again

Geometric Parrots: Rule Absence vs. Rule Fragility in Spatial and Physical Reasoning for Vision-Language Models

arXiv:2610.16170v1 Announce Type: new Abstract: Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

arXiv:2610.18694v1 Announce Type: new Abstract: Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models rec

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

arXiv:2610.18238v1 Announce Type: new Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page

LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents

arXiv:2610.17685v1 Announce Type: cross Abstract: Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local

Learn from Move: the Next Step for GUI Agents

arXiv:2610.16720v1 Announce Type: new Abstract: Existing benchmarks for GUI agents primarily focus on increasing task difficulty by extending task horizons. Besides, enabling agents to explore unfamil

QTrack: Query-Driven Reasoning for Multi-modal MOT

arXiv:2603.13759v2 Announce Type: replace Abstract: Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets sp

Remembering What Matters for Long-Horizon Robotic World Models

arXiv:2610.14583v1 Announce Type: cross Abstract: Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated h

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

arXiv:2610.18845v1 Announce Type: new Abstract: Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

arXiv:2610.16441v1 Announce Type: new Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explai

VitaBench: Memory Lifecycles Beyond Retrieval

arXiv:2610.14731v1 Announce Type: new Abstract: Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant