Search: “benchmark”

SWE-Race: a coding-agent benchmark of 188 real concurrency bugs, with results from three models [P]

model-releasesr-machinelearning

We've been building a benchmark out of real concurrency bugs (race conditions, deadlocks, cancellation issues) taken from merged PRs in about 100 Python projects. Each task gets graded by the project'

6 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos

DGX agent

arXiv:2610.04432v1 Announce Type: new Abstract: Building interactive simulators from real-world observations is a promising way to scale embodied data, but current pipelines still rely heavily on manu

model-releasesarxiv-cs-cv

6 Oct 2026

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

arXiv:2610.03639v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to support legal practice, education, and research, yet their reliability in national legal systems o

5 Oct 2026

From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

arXiv:2610.03524v1 Announce Type: new Abstract: General-purpose Text-to-SQL systems achieve strong performance on academic benchmarks like Spider and BIRD, where schemas are relatively shallow and col

5 Oct 2026

HakemBench: A Turkish Benchmark of Typed Decisions

arXiv:2610.02293v1 Announce Type: new Abstract: HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a pr

5 Oct 2026

JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion

arXiv:2610.00353v1 Announce Type: new Abstract: A sound judgment applies the law to established facts and weighs the circumstances in which they arose. However, existing methods swing between rigid st

2 Oct 2026

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

arXiv:2610.00531v1 Announce Type: new Abstract: AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however

2 Oct 2026

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

arXiv:2610.01499v1 Announce Type: new Abstract: Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standar

2 Oct 2026

Research

EHR-RobustGym: Benchmarking and Training Agents for Robust Clinical Reasoning

arXiv:2609.39371v1 Announce Type: new Abstract: In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements refer

researcharxiv-cs-ai

GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning

arXiv:2609.39777v1 Announce Type: new Abstract: LLM-based multi-agent systems coordinate specialized reasoning through aggregation, interaction, and adaptive control, yet their potential for graph lea

MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

arXiv:2511.00421v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise in medical applications, but their ability to detect and correct errors in clinical texts remains un

NodeGround: A Node Classification Benchmark in the Graph Foundation Model Era

arXiv:2609.39673v1 Announce Type: new Abstract: Can a pretrained graph model replace training and tuning a separate predictor for each dataset? Answering this requires evaluating prediction quality al

We benchmarked 18 RAG pipelines against an agent loop on Google's FRAMES. The best pipeline hit 78.9%. The agent loop hit 92.7%.

model-releasesr-localllama

Hybrid search, reranking, query decomposition, and query expansion are often treated as must-haves for good RAG. We wanted to see how much each actually helped, so we tested them. Same model, same emb

AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

arXiv:2607.11849v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved remarkable performance on high-school and competition-level mathematics, yet their capabilities on advanc

au-Multilingual: Benchmarking Voice Agents Across Languages

arXiv:2609.35820v1 Announce Type: cross Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce au-Multilingual, extending au-Voice to Spanish, Brazilian Por

Research

Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics

arXiv:2609.36492v1 Announce Type: cross Abstract: We benchmarked vision-language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse dete

researcharxiv-cs-ai

CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies

arXiv:2609.32192v2 Announce Type: replace Abstract: Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

arXiv:2609.37658v1 Announce Type: new Abstract: LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-ter

A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

arXiv:2609.32890v1 Announce Type: new Abstract: Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured

CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration

arXiv:2609.34782v1 Announce Type: cross Abstract: Many physical tasks in human environments require collaboration, from assisting a partner to jointly manipulating an object. Yet, existing humanoid be

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

arXiv:2609.31632v1 Announce Type: new Abstract: Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence a

Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

arXiv:2609.31648v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are studied mainly in robotics, where visual observations and language instructions are mapped to physical actions.

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

arXiv:2609.32352v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insuf

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

arXiv:2609.33158v1 Announce Type: new Abstract: Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance repor

Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

arXiv:2609.33812v1 Announce Type: cross Abstract: Repeated runs of the same coding agent are known to give different benchmark scores. We ask what that variation means for a team running an agent on i

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

arXiv:2609.32813v1 Announce Type: new Abstract: Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still str

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

arXiv:2609.34480v1 Announce Type: new Abstract: Reliable visual question answering requires correct answers when evidence is sufficient and abstention when it is not. We introduce a benchmark that con

Benchmarking Attention for Tabular Foundation Models

arXiv:2609.31306v1 Announce Type: new Abstract: Tabular in-context learners such as TabPFN, Mitra, or ConTextTab rely on alternating row and column attention over 2D sequences of latent embeddings. Th

28 Sept 2026

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

arXiv:2609.31587v1 Announce Type: cross Abstract: We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it

28 Sept 2026

PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents

arXiv:2609.31468v1 Announce Type: cross Abstract: LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferen

28 Sept 2026

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

arXiv:2609.28813v1 Announce Type: new Abstract: Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotio

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

arXiv:2603.18612v2 Announce Type: replace Abstract: We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev an

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

arXiv:2609.28876v1 Announce Type: new Abstract: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questi

If you want to see interactive explainers of benchmarks and demos I built, you can read it on our new dev site here: https://claude.dev/blog…

Thariq (@trq212) posted on September 25 , 2026 that readers can view interactive explainers of benchmarks and demos he built on the new Claude dev site (https://claude.dev/blog). The tweet includes a

model-releasesthariq--x

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

arXiv:2609.29028v1 Announce Type: new Abstract: In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassi

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

arXiv:2609.29445v1 Announce Type: new Abstract: We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sente

Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court

arXiv:2609.26945v1 Announce Type: new Abstract: Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the

Industry

Companies are increasingly using AI benchmarking services to aggregate public job listings and other payroll data to identify under- and over-paid employees (Callum Borchers/Wall Street Journal)

Callum Borchers / Wall Street Journal: Companies are increasingly using AI benchmarking services to aggregate public job listings and other payroll data to identify under- and over-paid employees — Ne

industrytechmeme

Hardware

Folks, have you purchased the Mac M5 Ultra with 256GB yet? We need serious benchmarks, because we only get YouTube clowns influencers results

Ok, so the Mac M5 Ultra (256GB) hit the market, but the only publicly available benchmarks material are flashy YouTube 'clown influencers' videos. We need serious numbers to evaluate whether Apple’s s

hardwarer-localllama

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

arXiv:2609.26942v1 Announce Type: cross Abstract: Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a rec

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

arXiv:2508.13680v5 Announce Type: replace Abstract: We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) in

Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework

arXiv:2609.22696v1 Announce Type: new Abstract: Decentralized social media platforms create new opportunities and challenges for computational mental health research because data access, moderation, l

IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its te

MOLA LiDAR-Inertial Odometry (MOLA-LIO) on the COMFORT Localization Benchmark

model-releasesarxiv-cs-ro

arXiv:2609.25813v1 Announce Type: new Abstract: This short report documents our entry to the COMFORT Localization Benchmark (IROS 2026), evaluated on the GrandTour dataset recorded with the Boxi paylo

Most mental health benchmarks focus on emergency situations. MentalHealthBench is designed to cover the full spectrum of mental health conve…

Most mental health benchmarks focus on emergency situations. MentalHealthBench is designed to cover the full spectrum of mental health conversations that people bring to AI - from everyday support to

model-releasesopenai--x

Ask Before It Tells: Benchmark-to-Robot Body-Cue Transfer for a Question-First Bedside Robot

model-releasesarxiv-cs-ro

arXiv:2609.24099v1 Announce Type: new Abstract: Body-cue recognition can support assistive robots, but benchmark accuracy does not guarantee reliable behavior under a robot-camera viewpoint. We presen

22 Sept 2026

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

arXiv:2609.23231v1 Announce Type: new Abstract: Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a diff

22 Sept 2026

Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking

arXiv:2609.23825v1 Announce Type: new Abstract: We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architec