Search: “benchmark”

Matrix Profile for Anomaly Detection on Multidimensional Time Series

arXiv:2409.09298v2 Announce Type: replace-cross Abstract: The Matrix Profile (MP), a versatile tool for time series data mining, has been shown effective in time series anomaly detection (TSAD). This

10 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

OmniTabBench: Mapping the Empirical Frontiers of GBDTs, Neural Networks, and Foundation Models for Tabular Data at Scale

DGX agent

arXiv:2604.06814v1 Announce Type: cross Abstract: While traditional tree-based ensemble methods have long dominated tabular tasks, deep neural networks and emerging foundation models have challenged t

model-releasesarxiv-cs-ai

10 Apr 2026

Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

arXiv:2604.07343v1 Announce Type: cross Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a cen

10 Apr 2026

VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

arXiv:2604.07634v1 Announce Type: new Abstract: Streaming vision-language models (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core

10 Apr 2026

AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations

arXiv:2610.21624v1 Announce Type: new Abstract: Real-world tasks are often complex and noisy, posing substantial challenges for LLM agents. Yet existing evaluations lack a systematic framework for joi

ClaimScope: Label-Blind Witness Verification for Mathematical Claim Auditing

arXiv:2610.21581v1 Announce Type: new Abstract: Mathematical agents are commonly evaluated by answer accuracy or proof acceptance. We introduce ClaimScope, a 72-task benchmark for auditing proposed cl

It really just needs some encouragement (Qwen3.8-27B-UD-IQ4_XS)

model-releasesr-localllama

I know I am probably missing the point of this benchmark, but I wanted to see if the model is capable of generating something better or is that the best it can do. So I gave it some encouragement. I f

Momentum, Commentary Tone, and Earnings Surprise in LLM Financial Forecasts

arXiv:2610.22043v1 Announce Type: cross Abstract: Which inputs drive a language model's financial forecast? We construct synthetic earnings announcements that independently reverse 20-day momentum, no

Perception or Reasoning? Diagnosing and Tackling Reinforcement Learning Bottlenecks in VLMs

arXiv:2610.21380v1 Announce Type: new Abstract: Reinforcement learning has brought strong gains to language model reasoning, but its improvements on visual tasks remain less consistent. One difficulty

Prism-1: Position- and Rival-Independent Scoring for System One Models

arXiv:2610.22066v1 Announce Type: new Abstract: Given a state and a set of options, such as intents, tools or interface elements, System One decision models return a distribution over the options in o

Qwen-Audio-Jev: A Unified Interface for Audio Decisions

arXiv:2610.21614v1 Announce Type: cross Abstract: Large audio language models (LALMs) have made substantial progress in audio understanding. However, in high-throughput applications requiring frequent

Stronger Judges Do Not See More: Deployment-Integrity Blindness in LLM Evaluation

arXiv:2610.21135v1 Announce Type: cross Abstract: LLM-agent evaluation reports behavioral scores but leaves the identity of the deployed runtime stack unreported. A tampered deployment that preserves

SWE-Nexus: Automated Composition of Software Engineering Tasks from Test Execution

arXiv:2610.20038v2 Announce Type: replace-cross Abstract: Historical code changes provide convenient task boundaries, but do not identify which production functions selected tests exercise or support

Verdict: Encode Once, Decide Many

arXiv:2610.20778v1 Announce Type: cross Abstract: Automating organizational work means answering many small typed questions about one piece of state (does this email need a reply, how urgent is it, wh

AgentStripes: An Evidence-Graded Maturity Model for AI Agents

arXiv:2610.18603v1 Announce Type: new Abstract: Benchmarks report what an Artificial Intelligence (AI) agent can do on a task suite, but not whether one released version has been engineered with the c

AM-Bench: Understanding Agents under Task Underspecification across Models and Harnesses

arXiv:2610.18090v1 Announce Type: new Abstract: Ambiguous or incomplete task specifications complicate both execution and clarification for language-model agents. Yet agents' responses to these inform

Calibrated Typed Decisions for Entity, Relation, and Event Extraction

arXiv:2610.17113v1 Announce Type: cross Abstract: Information extraction models often perform well on the datasets they are trained on, but transferring to new domains and annotation guidelines remain

claimcheck: Measuring and Improving Specification Faithfulness

arXiv:2610.17358v1 Announce Type: cross Abstract: Formal verification proves that a program satisfies its specification, but says nothing about whether the specification captures what the author meant

CommerceGen-Studio: An End-to-End Framework for Commercial-Grade Main-Image Generation

arXiv:2610.20603v1 Announce Type: new Abstract: We present CommerceGen-Studio, an end-to-end framework for generating commercial-grade main images for e-commerce Stock Keeping Units (SKUs). Our framew

CompanionSafe: From Multi-turn Safety Evaluation to Guardrails for AI Companions

arXiv:2610.18095v1 Announce Type: new Abstract: AI companions provide users with emotional support and ongoing companionship, but their sustained relational interactions also introduce distinct safety

CWE-bench: Refocusing Agentic Security from Exploitation to Defense

arXiv:2610.17020v1 Announce Type: cross Abstract: Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An a

Does a Governed Confidence Score Know When a Text-to-SQL Answer Is Right? An Exploratory Study on BIRD and a Pre-registered Confirmation on Spider

arXiv:2610.17256v1 Announce Type: new Abstract: Conversational analytics lets people ask questions of records spread across many systems and get answers in plain language; it is useful only if each an

DynThinker: Unifying Static 3D and Dynamic 4D Spatial Reasoning

arXiv:2610.20598v1 Announce Type: new Abstract: Achieving general spatial intelligence requires a unified understanding of both static 3D and dynamic 4D. However, we find that existing models struggle

Long-Horizon-Terminal-Bench 2.0: Automating Diverse Task Construction and Measuring Sustained Agent Progress

arXiv:2610.14146v1 Announce Type: new Abstract: Long-horizon terminal benchmarks measure whether an agent can sustain progress through extended workflows with process rewards, not only whether its fin

PsyMemBehav: Psychology-Informed and Memory-Grounded Two-Level Identity-to-Action Reasoning for Social-Media Behavior Simulation

arXiv:2610.14570v1 Announce Type: new Abstract: Large language models (LLMs) support social simulation, but faithfully reproducing real users requires grounding both structured platform decisions and

Qwen3.8FN vLLM vs. llama.cpp vs Ninfer6000

model-releasesr-localllama

I saw u/lkarlslund 's post just over a week ago regarding Ninfer6000 (a fork of Ninfer specifically tuned for 6000 blackwell and Qwen3.8-Flash-Next) and figured 'Hey I'm an idiot & bought one of those

Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate

arXiv:2610.15710v1 Announce Type: new Abstract: Long-term memory for assistants needs a gate that decides when a new utterance retires an old stored claim. Benchmark labels for this step mark a pair a

This was my first blog post at @llama_index! Go check out the cool diagrams and stuff

model-releasesjerry-liu--x

This was my first blog post at @llama_index! Go check out the cool diagrams and stuff Media Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a mas

Unconstrained CTI-REALM with Open-Weights

arXiv:2610.14180v1 Announce Type: new Abstract: Organisations that want language-model agents for security work face a choice between open-weight models, which they can run and audit themselves, and f

What Counts as the Same Task? Executable Contracts for Robustness Evaluation of Tool-Using Agents

arXiv:2610.14477v1 Announce Type: cross Abstract: Robustness tests for tool-using agents often change the agent's environment, for example by hiding a hostile note in an invoice, and then judge the ag

What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

model-releasesarxiv-cs-cl

arXiv:2610.20195v1 Announce Type: new Abstract: Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing e

A Plausible Future Is Not a Causal Future: Can Video Models Generate How the World Unfolds?

arXiv:2610.15562v1 Announce Type: new Abstract: Large-scale video generation models are increasingly viewed as a promising path toward world models, yet their ability to capture causally coherent phys

Auditing Failure Attribution: Label Predictability and Judge Behavior

arXiv:2610.17747v1 Announce Type: cross Abstract: Failure-attribution benchmarks ask an LLM judge which step or agent caused a failure. A high score should mean the judge checked the faulty step again

Geometric Parrots: Rule Absence vs. Rule Fragility in Spatial and Physical Reasoning for Vision-Language Models

arXiv:2610.16170v1 Announce Type: new Abstract: Vision-language models (VLMs) can answer many geometric and physical reasoning questions correctly, yet still behave inconsistently in interaction, for

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

arXiv:2610.18694v1 Announce Type: new Abstract: Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models rec

Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory

arXiv:2610.18238v1 Announce Type: new Abstract: Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page

LatentRiskBench: Measuring State-Dependent Risks in Enterprise Agents

arXiv:2610.17685v1 Announce Type: cross Abstract: Enterprise agents face safety risks that cannot be determined from individual actions alone. Existing safety benchmarks largely evaluate action-local

Learn from Move: the Next Step for GUI Agents

arXiv:2610.16720v1 Announce Type: new Abstract: Existing benchmarks for GUI agents primarily focus on increasing task difficulty by extending task horizons. Besides, enabling agents to explore unfamil

muse wonder sota on hard document ocr tasks the more you know!

model-releasesjerry-liu--x

muse wonder sota on hard document ocr tasks the more you know! We benchmarked Gemini 4 Argon, Muse Wonder, and Muse Spark comprehensively on hard document OCR tasks. They're all pretty good! - Muse Sp

QTrack: Query-Driven Reasoning for Multi-modal MOT

arXiv:2603.13759v2 Announce Type: replace Abstract: Category-level multi-object trackers estimate trajectories for all detected instances, whereas some applications require propagating only targets sp

Remembering What Matters for Long-Horizon Robotic World Models

arXiv:2610.14583v1 Announce Type: cross Abstract: Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated h

The Last Remote Sensing Model Built by Humans? Casting Any Earth Observation Task as Recursive Self-Improvement

arXiv:2610.18845v1 Announce Type: new Abstract: Earth observation (EO) models are still built by hand. For each new task, experts design a training recipe and tune it against a benchmark. We show that

VisExMEM: Visual Evidence-Based Explainable Multimodal Evaluation Metric

arXiv:2610.16441v1 Announce Type: new Abstract: Current image-text evaluation often relies on a global score, which gives limited insight into which semantic details are supported by the image. Explai

VitaBench: Memory Lifecycles Beyond Retrieval

arXiv:2610.14731v1 Announce Type: new Abstract: Long-term memory benchmarks for language agents primarily evaluate whether information from earlier interactions can be retrieved and used when relevant

AgentSWE: Can Coding Agents Build the Agent You Actually Want?

arXiv:2610.13361v1 Announce Type: cross Abstract: As the production of software passes into the hands of software itself, coding agents are increasingly commissioned to build agents fitted to their us

Don't Guess, Just Ask: Evaluating Interactive Requirement Clarification for Repository-Level Software Development

arXiv:2610.13022v1 Announce Type: cross Abstract: As LLMs become increasingly capable of implementing software, a critical bottleneck emerges: whether they truly understand what users want. Users ofte

How to test agent tool descriptions with Harbor and Phoenix

Build a reproducible benchmark, compare tool use across agents, and test whether a description change reduces unnecessary work while preserving outcomes. The post How to test agent tool descriptions w

model-releasesarize-ai

LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge

arXiv:2506.09956v2 Announce Type: replace-cross Abstract: Indirect prompt injection attacks exploit a fundamental weakness of LLMs: executing unauthorized tasks from untrusted sources. Despite the pra