Search: “benchmark”

RunyaNER: Auxiliary Language Selection for Runyankore NER

arXiv:2609.37543v1 Announce Type: new Abstract: Cross-lingual zero-shot transfer and multilingual fine-tuning are promising approaches for NLP tasks such as Named Entity Recognition (NER) in low-resou

30 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

System one models are extremely useful for a variety of document tasks that require fast decisions: * orientation detection * language detec…

DGX agent

System one models are extremely useful for a variety of document tasks that require fast decisions: * orientation detection * language detection * classification * splitting We benchmarked Jev with ot

model-releasesjerry-liu--x

30 Sept 2026

ActionLens: Diagnosing Spatial-Temporal Binding Failures in Vision-Language Models

arXiv:2609.34547v1 Announce Type: new Abstract: Video-capable vision-language models score above 80% on popular benchmarks yet struggle with spatial-temporal binding: associating the right action with

BITS: Rethinking Fair and Comprehensive Evaluation for Irregular Time Series Forecasting

arXiv:2609.33303v1 Announce Type: new Abstract: Despite recent progress in irregular time series forecasting, the field still lacks a unified benchmark for fair and comprehensive evaluation. Existing

HM-ROUTER: Joint Model and Harness Routing for Agentic Systems

arXiv:2609.32213v1 Announce Type: new Abstract: Agent performance depends on both the underlying model and the harness that manages its tool use and execution. Selecting a suitable pair requires accou

Research

Mixed-Prior Decision Risk for Open-Set Recognition

arXiv:2609.35043v1 Announce Type: new Abstract: In open-set recognition (OSR), a probe must either be identified as one of the known gallery classes or rejected as unknown, so three error types coexis

researcharxiv-cs-cv

PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

arXiv:2609.34314v1 Announce Type: new Abstract: Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

arXiv:2609.32734v1 Announce Type: new Abstract: AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allo

Recipe-Matching, Not Equivalence

arXiv:2609.31927v1 Announce Type: cross Abstract: MathNet-Retrieve asks a retriever to find, for a math problem, a document stating the same problem. An LLM under one fixed prompt writes each gold doc

SAGE: Symbolic Action-Gating and Editing for LLM Task Planners

model-releasesarxiv-cs-ro

arXiv:2609.34268v1 Announce Type: new Abstract: Large language models (LLMs) are now the default cognitive core of embodied household agents, yet the plans they emit are rarely checked against a groun

ScreenHaystack: Finding Blind Zones in GUI Grounding

arXiv:2609.32036v1 Announce Type: new Abstract: We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each targe

Evaluating Cultural Awareness of LLMs for Haitian Creole

arXiv:2609.31506v1 Announce Type: cross Abstract: Large language models (LLMs) exhibit substantial performance disparities between high- and low-resource languages. Beyond lower task performance, they

Evaluating Real-Time Voice Agents: From Component Quality to Grounded Outcomes

arXiv:2609.30798v1 Announce Type: new Abstract: Real-time voice agents have moved from research prototypes to production deployments, yet the literature describing them is fragmented across three comm

LUCID: Learning Under Confounding for Inference and Discovery in Time Series

arXiv:2609.31315v1 Announce Type: new Abstract: Unobserved common causes are pervasive in real-world time series and can induce spurious associations that causal discovery methods mistake for direct e

MoMHa: Multi-Objective Optimization of LLM Harnesses over Accuracy, Safety, and Tokens

arXiv:2609.30967v1 Announce Type: new Abstract: Most work on improving large language models treats accuracy as the sole objective. We argue that the harness, the Python code surrounding the model tha

Seeing Semantic Shift: Difference-Aware Sentence-Level Temporal Segmentation of Sign Language Videos

arXiv:2609.31148v1 Announce Type: new Abstract: Recent advances in sign language understanding have achieved impressive success on short, single-sentence videos, yet their performance drops sharply wh

STORM-Bench: Evaluating Online Video QA under Evolving and Incomplete Evidence

arXiv:2609.30981v1 Announce Type: new Abstract: Reliable online video question answering requires tracking state transitions while selectively abstaining when visual evidence is insufficient. Existing

Synth-JEPA: Joint Embedding Prediction for Renderer-Free Synthesizer Parameter Search

arXiv:2609.31024v1 Announce Type: cross Abstract: Sound matching can be formulated as optimizing synthesizer parameters against an audio-domain objective. However, objectives derived from generic audi

TemplateCraft: Agentic Visual Template Generation

arXiv:2609.31451v1 Announce Type: cross Abstract: The growing popularity of short videos has driven demand for one-click content creation. Visual templates turn uploaded images into personalized conte

VLALight: Lightweight Vision-Language-Action Models for Emergency-Aware Traffic Signal Control

arXiv:2609.30709v1 Announce Type: cross Abstract: Traffic signal control (TSC) is essential for mitigating urban congestion. Recent advances in vision-language models (VLMs) enable richer interpretati

2026 in LLMs (so far)

model-releasessimon-willison

On Friday I gave the closing keynote at the WeAreDevelopers World Congress North America in San Jose. I tied together the key trends from the past year into a chronological exploration of everything t

27 Sept 2026

Qwen plays World of Warcraft

model-releasesr-localllama

Been doing a bunch of vibe coding lately. Had my agents host a private WoW server for me, then built out a web browser client so you can play without installing the game and it has mobile controls. Af

27 Sept 2026

I built a tiny (332MB) CPU-friendly model for document sorting that actually knows when to say 'none fits' (BeeNara)

model-releasesr-localllama

Hey r/LocalLLaMA! I wanted to share a small project I’ve been working on called BeeNara Why I built this: I was looking for a way to automatically sort my local documents (invoices, letters, contrac

26 Sept 2026

Beneath the Scores: Rethinking Hallucination Evaluation for Video Understanding Models

arXiv:2609.28991v1 Announce Type: cross Abstract: Video understanding is increasingly performed by multi-stage LLM agents that separate temporal grounding, visual observation, and reasoning. Yet these

Diverse Geometries, Frozen Weights: Robust Heterogeneous Treatment-Effect Estimation via Causal Expert Ensembles

arXiv:2609.29974v1 Announce Type: new Abstract: Estimating heterogeneous treatment effects from observational data is difficult because the most appropriate inductive bias varies with overlap, treatme

EnigmaForge: The Question Is Hidden in the Story

arXiv:2609.30144v1 Announce Type: new Abstract: Most benchmarks hand the model a question. EnigmaForge hands it a stack of old documents and no question at all. Buried in the letters, receipts, and lo

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

arXiv:2609.29573v1 Announce Type: new Abstract: Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that disto

Qwen3.8-27B: Using KV Cache Transplants to Boost Output Quality

model-releasesr-localllama

Since my last post, I've been thinking about different options for dynamic performance degradation, trying to squeeze as much high-quality inference out of my GPU as I can. Over the weekend I read thi

Task-Resolved Fisher Spectroscopy for Quantum Reservoir Computing

arXiv:2609.29570v1 Announce Type: cross Abstract: Quantum reservoir computing (QRC) uses fixed quantum dynamics to encode a time series and trains only a classical readout, but a benchmark capacity al

Unmasking Shortcut Learning in IoT Intrusion Detection: A Forensic, Multi-Paradigm Evaluation of Feature Dependence and Data Leakage

arXiv:2609.28725v1 Announce Type: cross Abstract: Machine learning-based Network Intrusion Detection Systems often report near-perfect performance on IoT benchmarks. However, whether these models lear

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

arXiv:2609.28335v1 Announce Type: new Abstract: Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is

CAVEAT: Towards Robust Computer-Use Agents in Incentive-Misaligned Environments

arXiv:2609.27273v1 Announce Type: new Abstract: Computer-use agents (CUAs) increasingly act on behalf of users online. What happens when the environments they operate in have incentives that do not al

Escaping Python Dependency Hell: A Hybrid Replay-and-Repair Pipeline for Python Dependency Resolution

arXiv:2609.26952v1 Announce Type: new Abstract: Dependency conflicts in Python ecosystems arise from incompatible version constraints, missing packages, and undocumented compatibility relationships, c

Learning the Cost of Reliable Inference

arXiv:2609.28322v1 Announce Type: new Abstract: Benchmarking and routing platforms increasingly act as intermediaries connecting large language model providers with end-users. However, providers on th

Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models

arXiv:2609.27372v1 Announce Type: cross Abstract: Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by comple

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and u

What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit

arXiv:2609.27408v1 Announce Type: cross Abstract: Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is tr

What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

arXiv:2609.26826v1 Announce Type: cross Abstract: Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass

Bayesian Fusion of Active Contour Models and ConvNet Priors for Standing Dead Tree Segmentation

arXiv:2410.07421v2 Announce Type: replace Abstract: Instance segmentation is a core computer vision task with great practical significance. Recent advances, driven by large-scale benchmark datasets, h

ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry

arXiv:2511.17909v2 Announce Type: replace Abstract: Chemical reasoning inherently integrates visual, textual, and symbolic modalities, yet existing benchmarks rarely capture this complexity, often rel

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

model-releasesarxiv-cs-ro

arXiv:2609.26792v1 Announce Type: new Abstract: Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene featur

Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices

arXiv:2609.25645v1 Announce Type: cross Abstract: Exhaustively evaluating every candidate LLM configuration on every benchmark item to identify a high-performing one is costly. We formulate configurat

Self-Organizing Agent Teams Learn to Reason Together

arXiv:2609.22682v1 Announce Type: new Abstract: Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknow

This is both fun and a useful measure of how much better multimodal AI has gotten In a short time.

model-releasesethan-mollick--x

This is both fun and a useful measure of how much better multimodal AI has gotten In a short time. Can AI tell if you've built your IKEA furniture wrong? Our new benchmark, the Furniture Assembly Benc

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

arXiv:2609.22114v1 Announce Type: new Abstract: Context compression is widely proposed as a way to cut the token bill of LLM coding agents, and public benchmarks report that aggressive compression pre

Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection

arXiv:2609.22767v1 Announce Type: new Abstract: The IEEE BigData Cup benchmark combines three prediction problems with different output structures: ordinal suicide-risk classification, multi-label psy

Hi Ollama People!

Hi everyone, I wanted to ask you a question. I’m developing an app, and I’m currently putting together a list of the best AI models, especially, but not necessarily, in GGUF format and using Q4_K_M qu

model-releasesr-ollama

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

arXiv:2609.22788v1 Announce Type: new Abstract: Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes.