Search: “benchmark”

Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Models

arXiv:2609.27372v1 Announce Type: cross Abstract: Benchmarks for full-duplex spoken dialogue models score turn-taking with binary fixed-window rules that reward immediate response or silence by comple

24 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

DGX agent

arXiv:2609.27510v1 Announce Type: cross Abstract: Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and u

model-releasesarxiv-cs-ai

24 Sept 2026

What Looks Like a Capability Limit in Vision-Language Models Is a Readout Limit

arXiv:2609.27408v1 Announce Type: cross Abstract: Benchmarks for vision-language models offer their answer choices in some convention: a letter, a color name, a pixel coordinate. That convention is tr

24 Sept 2026

Agents

What Makes a Terminal-Bench Task Hard? Separating Genuine Hardness from Fake-Hardness on an Adjudicated Agentic Corpus

arXiv:2609.26826v1 Announce Type: cross Abstract: Frontier benchmarks need tasks that current models cannot solve. But a task that no model solves is not automatically a hard task. The same zero pass

agentsarxiv-cs-ai

24 Sept 2026

Bayesian Fusion of Active Contour Models and ConvNet Priors for Standing Dead Tree Segmentation

arXiv:2410.07421v2 Announce Type: replace Abstract: Instance segmentation is a core computer vision task with great practical significance. Recent advances, driven by large-scale benchmark datasets, h

ChemVTS-Bench: Evaluating Visual-Textual-Symbolic Reasoning of Multimodal Large Language Models in Chemistry

arXiv:2511.17909v2 Announce Type: replace Abstract: Chemical reasoning inherently integrates visual, textual, and symbolic modalities, yet existing benchmarks rarely capture this complexity, often rel

DreamStream: Towards Policy-Oriented Generative Simulation for End-to-End Driving

model-releasesarxiv-cs-ro

arXiv:2609.26792v1 Announce Type: new Abstract: Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene featur

Efficient Cost-Aware LLM Evaluation via Bayesian Bandit Gittins Indices

arXiv:2609.25645v1 Announce Type: cross Abstract: Exhaustively evaluating every candidate LLM configuration on every benchmark item to identify a high-performing one is costly. We formulate configurat

Agents

Self-Organizing Agent Teams Learn to Reason Together

arXiv:2609.22682v1 Announce Type: new Abstract: Collective intelligence depends not only on what team members know, but also on how they organize their work. When the structure of a solution is unknow

agentsarxiv-cs-ai

An Empirical Cost Attribution of Context-Compression Gateways in Multi-Turn Coding Agents

arXiv:2609.22114v1 Announce Type: new Abstract: Context compression is widely proposed as a way to cut the token bill of LLM coding agents, and public benchmarks report that aggressive compression pre

Beyond Final-Token Classification: Heterogeneous Readouts for Evidence-Grounded Suicide Risk Detection

arXiv:2609.22767v1 Announce Type: new Abstract: The IEEE BigData Cup benchmark combines three prediction problems with different output structures: ordinal suicide-risk classification, multi-label psy

Human-Level Accuracy, Non-Human Strategies: Revealing Model-Human Divergence in Video Physical Reasoning

arXiv:2609.22788v1 Announce Type: new Abstract: Video foundation models now reach human-level accuracy on physical-reasoning benchmarks, yet such tasks require predicting unobserved physical outcomes.

PAGE: Partition-Aware Gated KV-Cache Eviction

arXiv:2609.22157v1 Announce Type: cross Abstract: KV-cache eviction methods decide which tokens to keep but not whether to evict at all, so a benchmark mean can hide a class of inputs on which compres

Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they ha

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

arXiv:2609.23570v1 Announce Type: cross Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluation

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlled

Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

arXiv:2609.22259v1 Announce Type: new Abstract: Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/with

Co-Evolving Zero-Day Jamming: Adaptive Attack Synthesis and Graph Attention-Based Online Detection

arXiv:2609.21334v1 Announce Type: cross Abstract: Effective evaluation of zero-day jamming detectors requires robust adversarial models. However, existing attack models often assume prior knowledge of

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

arXiv:2609.21562v1 Announce Type: cross Abstract: Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game

Research

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

arXiv:2609.21181v1 Announce Type: new Abstract: The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether

researcharxiv-cs-ai

Local Ai

Refining Ground Truth Poses in Autonomous Driving Datasets via Neural Rendering

arXiv:2504.15776v2 Announce Type: replace Abstract: Public autonomous driving datasets underpin the training and benchmarking of perception, mapping, and localization algorithms, yet residual inaccura

local-aiarxiv-cs-cv

Understanding Structural Representation in Foundation Models for Polymers

arXiv:2512.11881v2 Announce Type: replace-cross Abstract: From the relative scarcity of training data to the lack of standardized benchmarks, the creation of effective foundation models for polymers f

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

arXiv:2506.17277v2 Announce Type: replace-cross Abstract: The retrieval stage of retrieval-augmented generation (RAG) for scientific question answering depends on how documents are segmented and how c

MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents

arXiv:2609.20152v1 Announce Type: new Abstract: Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

arXiv:2609.20758v1 Announce Type: cross Abstract: Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in

Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

arXiv:2609.20700v1 Announce Type: new Abstract: Episodic test-time adaptation resets a frozen segmenter to source weights M_0 on each case and adapts for a fixed step count. A fixed horizon conflates

Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models

arXiv:2602.11244v2 Announce Type: replace Abstract: Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

arXiv:2609.18052v1 Announce Type: cross Abstract: This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specifica

Research

Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning

arXiv:2609.18991v1 Announce Type: new Abstract: Multimodal LLMs report strong performance on scientific reasoning benchmarks, yet most treat perception and reasoning as a single measurable process. We

researcharxiv-cs-ai

MSR: Multiple Subject Reference for Video Generation

arXiv:2609.18393v1 Announce Type: new Abstract: Conditioning a video generator on multiple images requires preserving appearance while associating each reference with its intended role. We present MSR

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

arXiv:2609.18805v1 Announce Type: cross Abstract: Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents ma

SEA-LION-v4.8: A Technical Report

arXiv:2609.18310v1 Announce Type: new Abstract: We introduce Nemotron-SEA-LION-v4.8, a family of Southeast Asian Languages in One Network (SEA-LION) built upon NVIDIA Nemotron 3. The family includes 3

Suppressed, Not Erased: A Representational Trace of Edited Facts Survives Even Weight-Free Knowledge Editing

arXiv:2609.18985v1 Announce Type: new Abstract: Knowledge-editing benchmarks certify local correctness, whether an edited model produces the new fact on near-edit prompts but not how much of the origi

Hardware

TabICLv2: A better, faster, scalable, and open tabular foundation model

arXiv:2602.11139v2 Announce Type: replace Abstract: Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstr

hardwarearxiv-cs-lg

Safety

Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognition

arXiv:2510.09653v4 Announce Type: replace-cross Abstract: This paper presents a comprehensive overview of the Ultralytics YOLO family, emphasizing architectural evolution, benchmarking, deployment, an

safetyarxiv-cs-ai

Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale

arXiv:2609.18769v1 Announce Type: new Abstract: Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document i

WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratories

arXiv:2609.18435v1 Announce Type: new Abstract: Automating biological research requires general-purpose, reproducible robot systems that allow individual wet-lab researchers to delegate robot tasks wi

WordPolo: Evaluating Language Models Through Iterative Semantic Feedback

arXiv:2609.19006v1 Announce Type: cross Abstract: Large Language Models (LLMs) and Large Reasoning Models (LRMs) are typically evaluated on challenging benchmarks through dataset accuracy alone, provi

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what

Bridging the Perceptual Gap: Residual-Enhanced Downscaling and Manifold-Aware Perception Alignment Adaptation for NR-IQA

arXiv:2609.16664v1 Announce Type: new Abstract: Leveraging Large Vision-Language Models like CLIP has recently set new benchmarks for No-Reference Image Quality Assessment (NR-IQA). However, the contr

Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning

arXiv:2609.16255v1 Announce Type: cross Abstract: We present an efficient method to distill reasoning capabilities into compact video-language models (VLMs) for video question answering (VideoQA). Our

How Good Are Time-Series Foundation Models for Pedestrian Crowd Count Forecasting? A Cross-Dataset Comparative Study

arXiv:2609.16415v1 Announce Type: cross Abstract: Pedestrian-count forecasting supports pedestrian-oriented Intelligent Transportation Systems (ITS), including crowd monitoring, pedestrian-traffic sta

JewelTry: Mask-Free Scale Aware Jewelry Virtual Try-On

arXiv:2609.16626v1 Announce Type: new Abstract: Virtual try-on (VTON) enables customers to visualize how fashion products appear when worn and has become an important technology for online shopping. W

Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?

arXiv:2507.23248v2 Announce Type: replace Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task

Tables Decoded: DELTA for Structure, TARQA for Understanding

arXiv:2609.17458v1 Announce Type: new Abstract: Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering (Ta

Building Legal Reward Models for Grounding and Abstention

arXiv:2609.14739v1 Announce Type: cross Abstract: Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and ab

15 Sept 2026

Research

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation

researcharxiv-cs-ai

15 Sept 2026

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

arXiv:2609.13773v1 Announce Type: cross Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set ma