Search: “benchmark”

Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act

arXiv:2609.22588v1 Announce Type: new Abstract: Vision-language models (VLMs) perform strongly on visual question answering benchmarks, yet often make decisions that contradict visual evidence they ha

22 Sept 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

DGX agent

arXiv:2609.23570v1 Announce Type: cross Abstract: Coding agents operate on real repository coding tasks, and persistent memory systems promise to reuse experience across tasks. Yet existing evaluation

model-releasesarxiv-cs-cl

22 Sept 2026

Vimarsha: Faithful ASR Evaluation for Indian Languages with Demographic Diversity, In-the-Wild Audio and Spelling Variations

arXiv:2609.24199v1 Announce Type: new Abstract: Evaluation benchmarks for Indian language automatic speech recognition (ASR) suffer from two systematic biases: optimistic scores from clean, controlled

22 Sept 2026

Which Part of the Context Layer Does the Work? Separating Semantic Content from Retrieval Scaffolding in Text-to-SQL Agents

arXiv:2609.22259v1 Announce Type: new Abstract: Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/with

22 Sept 2026

Co-Evolving Zero-Day Jamming: Adaptive Attack Synthesis and Graph Attention-Based Online Detection

arXiv:2609.21334v1 Announce Type: cross Abstract: Effective evaluation of zero-day jamming detectors requires robust adversarial models. However, existing attack models often assume prior knowledge of

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

arXiv:2609.21562v1 Announce Type: cross Abstract: Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game

Implicit Rule Induction with Test-Time Task Embeddings in ARC-like Tasks

arXiv:2609.21181v1 Announce Type: new Abstract: The Abstraction and Reasoning Corpus and related benchmarks evaluate whether AI models can solve novel reasoning tasks, but often leave unclear whether

Local Ai

Refining Ground Truth Poses in Autonomous Driving Datasets via Neural Rendering

arXiv:2504.15776v2 Announce Type: replace Abstract: Public autonomous driving datasets underpin the training and benchmarking of perception, mapping, and localization algorithms, yet residual inaccura

local-aiarxiv-cs-cv

Understanding Structural Representation in Foundation Models for Polymers

arXiv:2512.11881v2 Announce Type: replace-cross Abstract: From the relative scarcity of training data to the lack of standardized benchmarks, the creation of effective foundation models for polymers f

Finally got Qwen 3.8 Next running on my v100 6gpu setup (TP2 PP3)

model-releasesr-localllama

Hey everyone, After wrestling with hardware and engine issues for days, I finally got Qwen 3.8 Next running properly on my multi-GPU rig. Thought I’d share the setup journey, benchmarks, and thermal r

20 Sept 2026

we made a 27b model for creative writing. performs as good as claude fable 5, at a 40x cheaper price, open weights.

hey! we're a small swiss and south african lab and we just put out hemmingway-1. it's a 27b model we trained specifically for roleplay, creative writing, dialogue, stories, and for everyday texting. f

model-releasesr-ollama

20 Sept 2026

Applications

I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradi…

applicationsethan-mollick--x

I think there are lots of good reasons to worry about whether AI is giving correct answers (especially free models), but, given the contradiction with other recent research suggesting AI is now doing

19 Sept 2026

Chunk Twice, Embed Once: A Systematic Study of Segmentation and Representation Trade-offs in Chemistry-Aware Retrieval-Augmented Generation

arXiv:2506.17277v2 Announce Type: replace-cross Abstract: The retrieval stage of retrieval-augmented generation (RAG) for scientific question answering depends on how documents are segmented and how c

MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agents

arXiv:2609.20152v1 Announce Type: new Abstract: Generally, most voice agents are cascaded systems, i.e., an ASR model transcribes the caller's audio, a language model reads the transcript and decides

Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation

arXiv:2609.20758v1 Announce Type: cross Abstract: Evaluating an AI system requires disaggregated assessment, as performance varies across domains such as benchmark task types or conversation types in

Should This Case Be Adapted? Prediction Fragmentation Controls Test-Time Adaptation

arXiv:2609.20700v1 Announce Type: new Abstract: Episodic test-time adaptation resets a frozen segmenter to source weights M_0 on each case and adapts for a fixed step count. A fixed horizon conflates

Stress Tests REVEAL Fragile Temporal and Visual Grounding in Video-Language Models

arXiv:2602.11244v2 Announce Type: replace Abstract: Video-Language Models (VidLMs) achieve strong benchmark scores, yet these scores often hide whether models use the video at all. We show that VidLM

An Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming Tasks

arXiv:2609.18052v1 Announce Type: cross Abstract: This study empirically evaluates whether cost-efficient Large Language Models (LLMs) can be trusted to generate enterprise code to a written specifica

Lost in Perception: Isolating Perceptual and Reasoning Failures in Multimodal Physics and Geometry Reasoning

arXiv:2609.18991v1 Announce Type: new Abstract: Multimodal LLMs report strong performance on scientific reasoning benchmarks, yet most treat perception and reasoning as a single measurable process. We

MSR: Multiple Subject Reference for Video Generation

arXiv:2609.18393v1 Announce Type: new Abstract: Conditioning a video generator on multiple images requires preserving appearance while associating each reference with its intended role. We present MSR

ProgramDistill: From Interactive Web Apps to Verifiable Reference-Guided SWE Tasks

arXiv:2609.18805v1 Announce Type: cross Abstract: Coding agents are typically evaluated with desired behavior specified through issues or instructions. In practical web development, however, agents ma

SEA-LION-v4.8: A Technical Report

arXiv:2609.18310v1 Announce Type: new Abstract: We introduce Nemotron-SEA-LION-v4.8, a family of Southeast Asian Languages in One Network (SEA-LION) built upon NVIDIA Nemotron 3. The family includes 3

Suppressed, Not Erased: A Representational Trace of Edited Facts Survives Even Weight-Free Knowledge Editing

arXiv:2609.18985v1 Announce Type: new Abstract: Knowledge-editing benchmarks certify local correctness, whether an edited model produces the new fact on near-edit prompts but not how much of the origi

Hardware

TabICLv2: A better, faster, scalable, and open tabular foundation model

arXiv:2602.11139v2 Announce Type: replace Abstract: Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstr

hardwarearxiv-cs-lg

Safety

Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognition

arXiv:2510.09653v4 Announce Type: replace-cross Abstract: This paper presents a comprehensive overview of the Ultralytics YOLO family, emphasizing architectural evolution, benchmarking, deployment, an

safetyarxiv-cs-ai

Version- and Scope-Aware Question Answering over Normative Documents: A Deployed System and an End-to-End Evaluation at Production Scale

arXiv:2609.18769v1 Announce Type: new Abstract: Correctly answering a question grounded in normative documents often depends on information outside any single passage: whether the retrieved document i

WetRobo: A Reproducible Robot Kit for Coding Agents in Biological Laboratories

arXiv:2609.18435v1 Announce Type: new Abstract: Automating biological research requires general-purpose, reproducible robot systems that allow individual wet-lab researchers to delegate robot tasks wi

WordPolo: Evaluating Language Models Through Iterative Semantic Feedback

arXiv:2609.19006v1 Announce Type: cross Abstract: Large Language Models (LLMs) and Large Reasoning Models (LRMs) are typically evaluated on challenging benchmarks through dataset accuracy alone, provi

Beyond Cultural Knowledge: Evaluating Arabic Cultural Appropriateness of Large Language Models

arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what

Bridging the Perceptual Gap: Residual-Enhanced Downscaling and Manifold-Aware Perception Alignment Adaptation for NR-IQA

arXiv:2609.16664v1 Announce Type: new Abstract: Leveraging Large Vision-Language Models like CLIP has recently set new benchmarks for No-Reference Image Quality Assessment (NR-IQA). However, the contr

Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning

arXiv:2609.16255v1 Announce Type: cross Abstract: We present an efficient method to distill reasoning capabilities into compact video-language models (VLMs) for video question answering (VideoQA). Our

How Good Are Time-Series Foundation Models for Pedestrian Crowd Count Forecasting? A Cross-Dataset Comparative Study

arXiv:2609.16415v1 Announce Type: cross Abstract: Pedestrian-count forecasting supports pedestrian-oriented Intelligent Transportation Systems (ITS), including crowd monitoring, pedestrian-traffic sta

JewelTry: Mask-Free Scale Aware Jewelry Virtual Try-On

arXiv:2609.16626v1 Announce Type: new Abstract: Virtual try-on (VTON) enables customers to visualize how fashion products appear when worn and has become an important technology for online shopping. W

Script Fragmentation and Format: What Drives the English-Bengali Performance Gap in Open LLMs?

arXiv:2507.23248v2 Announce Type: replace Abstract: Bengali is spoken by more than 230 million people, yet no standardized instrument evaluates large language models (LLMs) on Bengali across the task

Tables Decoded: DELTA for Structure, TARQA for Understanding

arXiv:2609.17458v1 Announce Type: new Abstract: Table understanding is a core task in document intelligence, encompassing two key subtasks: table reconstruction and table visual question answering (Ta

Testing Qwen 3.8 27B running locally on a single 5090 via Ollama

I wanted to test what Qwen 3.8 27B running locally on a single 5090 could do in Row-Bot. I gave it just one prompt: 'I want you to create an animation showing what the current local model can do. Test

model-releasesr-ollama

Building Legal Reward Models for Grounding and Abstention

arXiv:2609.14739v1 Announce Type: cross Abstract: Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and ab

Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores

arXiv:2601.13885v2 Announce Type: replace-cross Abstract: Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation

Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging

arXiv:2609.13773v1 Announce Type: cross Abstract: LLM-as-a-Judge evaluators are increasingly used to score open-ended generation, yet a judge's correlation with human ratings on its development set ma

DreamSat-Bench: Development and Initial Testing of a Testbed for AI-Based Pose Estimation from 3D Reconstruction

model-releasesarxiv-cs-ro

arXiv:2609.14183v1 Announce Type: new Abstract: This paper presents the development and initial testing of DreamSat-Bench, a modular rendezvous and proximity operation testbed designed to benchmark AI

Enhancing Large Language Model-Based Systems for End-to-End Circuit Analysis Problem Solving

arXiv:2512.10159v3 Announce Type: replace-cross Abstract: LLMs have shown strong performance in data-rich domains such as programming, but their reliability in engineering tasks remains limited. Circu

Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth

arXiv:2609.13745v1 Announce Type: new Abstract: Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover

Realtime-Venus: A full-duplex interaction system with asynchronous delegation

arXiv:2609.13814v1 Announce Type: new Abstract: Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and lin

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

arXiv:2609.13463v1 Announce Type: new Abstract: The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reli

Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judges

arXiv:2602.13576v2 Announce Type: replace-cross Abstract: Evaluation and alignment pipelines for large language models increasingly rely on LLM-based judges, whose behavior is guided by natural-langua

Self-Evolving Memory for Generative Recommendation

arXiv:2609.15598v1 Announce Type: cross Abstract: Generative recommendation has emerged as a promising end-to-end paradigm for personalized recommendation. However, user preferences continuously evolv

Sharing standardized image-derived data in computational pathology using DICOM

arXiv:2609.14530v1 Announce Type: cross Abstract: Development and evaluation of computational pathology methods require access to large and diverse datasets. Over the past decade, various initiatives

Stellar Colosseum: A Many-Agent Harness for Long-Horizon Research in Mathematics and Theoretical Computer Science

arXiv:2609.15983v1 Announce Type: new Abstract: Language models can produce plausible short proofs, but may still be unreliable on long-horizon research problems, where progress depends on a sequence