Search: “benchmark”

arXiv:2609.33303v1 Announce Type: new Abstract: Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing

arXiv:2609.32213v1 Announce Type: new Abstract: Agent performance depends on both the underlying model and the harness that manages its tool use and execution. Selecting a suitable pair requires accou

arXiv:2609.35043v1 Announce Type: new Abstract: In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexis

arXiv:2609.34314v1 Announce Type: new Abstract: Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether

arXiv:2609.32734v1 Announce Type: new Abstract: AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allo

arXiv:2609.31927v1 Announce Type: cross Abstract: MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold doc

arXiv:2609.34268v1 Announce Type: new Abstract: Large language models (LLMs) are now the default cognitive core of embodied household agents, yet the plans they emit are rarely checked against a groun

arXiv:2609.32036v1 Announce Type: new Abstract: We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each targe

arXiv:2609.31506v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they

arXiv:2609.30798v1 Announce Type: new Abstract: Real-time voice agents have moved from research prototypes to production deployments, yet the literature describing them is fragmented across three comm

arXiv:2609.31315v1 Announce Type: new Abstract: Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct e

arXiv:2609.30967v1 Announce Type: new Abstract: Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model tha

arXiv:2609.31148v1 Announce Type: new Abstract: Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply wh

arXiv:2609.30981v1 Announce Type: new Abstract: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing

arXiv:2609.31024v1 Announce Type: cross Abstract: Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audi

arXiv:2609.31451v1 Announce Type: cross Abstract: The growing popularity of short videos has driven demand for one-click content creation. Visual templates turn uploaded images into personalized conte

arXiv:2609.30709v1 Announce Type: cross Abstract: Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretati

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything t

Been doing a bunch of vibe coding lately. Had my agents host a private WoW server for me, then built out a web browser client so you can play without installing the game and it has mobile controls. Af

Hey r/LocalLLaMA! ​I wanted to share a small project I’ve been working on called BeeNara ​Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contrac

arXiv:2609.28991v1 Announce Type: cross Abstract: Video understanding is increasingly performed by multi-stage LLM agents that separate temporal grounding, visual observation, and reasoning. Yet these

arXiv:2609.29974v1 Announce Type: new Abstract: Estimating heterogeneous treatment effects from observational data is difficult because the most appropriate inductive bias varies with overlap, treatme

arXiv:2609.30144v1 Announce Type: new Abstract: Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and lo

arXiv:2609.29573v1 Announce Type: new Abstract: Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that disto

Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read thi

arXiv:2609.29570v1 Announce Type: cross Abstract: Quantum reservoir computing (QRC) uses fixed quantum dynamics to encode a time series and trains only a classical readout, but a benchmark capacity al

arXiv:2609.28725v1 Announce Type: cross Abstract: Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models lear

arXiv:2609.28335v1 Announce Type: new Abstract: Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is

arXiv:2609.27273v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly act on behalf of users online. What happens when the environments they operate in have incentives that do not al

arXiv:2609.26952v1 Announce Type: new Abstract: Dependency conflicts in Python ecosystems arise from incompatible version constraints, missing packages, and undocumented compatibility relationships, c

arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on th

arXiv:2609.27372v1 Announce Type: cross Abstract: Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by comple

arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and u

arXiv:2609.27408v1 Announce Type: cross Abstract: Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is tr

arXiv:2609.26826v1 Announce Type: cross Abstract: Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass

arXiv:2410.07421v2 Announce Type: replace Abstract: Instance segmentation is a core computer vision task with great practical significance. Recent advances, driven by large-scale benchmark datasets, h

arXiv:2511.17909v2 Announce Type: replace Abstract: Chemical reasoning inherently integrates visual, textual, and symbolic modalities, yet existing benchmarks rarely capture this complexity, often rel

arXiv:2609.26792v1 Announce Type: new Abstract: Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene featur

arXiv:2609.25645v1 Announce Type: cross Abstract: Exhaustively evaluating every candidate LLM configuration on every benchmark item to identify a high-performing one is costly. We formulate configurat

arXiv:2609.22682v1 Announce Type: new Abstract: Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknow

This is both fun and a useful measure of how much better multimodal AI has gotten In a short time. Can AI tell if you've built your IKEA furniture wrong? Our new benchmark, the Furniture Assembly Benc

arXiv:2609.22114v1 Announce Type: new Abstract: Context compression is widely proposed as a way to cut the token bill of LLM coding agents, and public benchmarks report that aggressive compression pre

arXiv:2609.22767v1 Announce Type: new Abstract: The IEEE BigData Cup benchmark combines three prediction problems with different output structures: ordinal suicide-risk classification, multi-label psy

Hi everyone, I wanted to ask you a question. I’m developing an app, and I’m currently putting together a list of the best AI models, especially, but not necessarily, in GGUF format and using Q4_K_M qu

arXiv:2609.22788v1 Announce Type: new Abstract: Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes.

arXiv:2609.22157v1 Announce Type: cross Abstract: KV-cache eviction methods decide which tokens to keep but not whether to evict at all, so a benchmark mean can hide a class of inputs on which compres