Search: “benchmark”

From Benchmarks to Production: A Text-to-SQL System for Complex Financial Data

arXiv:2610.03524v1 Announce Type: new Abstract: General-purpose Text-to-SQL systems achieve strong performance on academic benchmarks like Spider and BIRD, where schemas are relatively shallow and col

5 Oct 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

HakemBench: A Turkish Benchmark of Typed Decisions

DGX agent

arXiv:2610.02293v1 Announce Type: new Abstract: HakemBench is a Turkish benchmark of typed decisions, in which the model under test reads a text, a question and a fixed set of options and returns a pr

model-releasesarxiv-cs-cl

5 Oct 2026

JusticeAxis: Benchmarking Legal Judgment between Rigid Rule Application and Ungrounded Discretion

arXiv:2610.00353v1 Announce Type: new Abstract: A sound judgment applies the law to established facts and weighs the circumstances in which they arose. However, existing methods swing between rigid st

2 Oct 2026

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

arXiv:2610.00531v1 Announce Type: new Abstract: AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however

2 Oct 2026

VTR-Bench: A Systematic Benchmark for Evaluating Visual Text Rendering in Video Generation

arXiv:2610.01499v1 Announce Type: new Abstract: Recent video generation models can produce highly realistic videos from natural language instructions, with visual quality approaching cinematic standar

2 Oct 2026

Research

EHR-RobustGym: Benchmarking and Training Agents for Robust Clinical Reasoning

arXiv:2609.39371v1 Announce Type: new Abstract: In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements refer

researcharxiv-cs-ai

GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning

arXiv:2609.39777v1 Announce Type: new Abstract: LLM-based multi-agent systems coordinate specialized reasoning through aggregation, interaction, and adaptive control, yet their potential for graph lea

MedRECT: A Bilingual Medical Reasoning Benchmark for Error Correction in Clinical Texts

arXiv:2511.00421v2 Announce Type: replace-cross Abstract: Large language models (LLMs) show promise in medical applications, but their ability to detect and correct errors in clinical texts remains un

NodeGround: A Node Classification Benchmark in the Graph Foundation Model Era

arXiv:2609.39673v1 Announce Type: new Abstract: Can a pretrained graph model replace training and tuning a separate predictor for each dataset? Answering this requires evaluating prediction quality al

AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

arXiv:2607.11849v2 Announce Type: replace Abstract: Large language models (LLMs) have achieved remarkable performance on high-school and competition-level mathematics, yet their capabilities on advanc

au-Multilingual: Benchmarking Voice Agents Across Languages

arXiv:2609.35820v1 Announce Type: cross Abstract: English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce au-Multilingual, extending au-Voice to Spanish, Brazilian Por

Research

Benchmarking Vision-Language Models on Synapse Detection and Proofreading in Connectomics

arXiv:2609.36492v1 Announce Type: cross Abstract: We benchmarked vision-language models (VLMs) on the decisions annotators take when inspecting electron microscopy images in connectomics: synapse dete

researcharxiv-cs-ai

CoMemBench: Benchmarking Collaborative Memory Boundaries across Multi-Agent Workflow Topologies

arXiv:2609.32192v2 Announce Type: replace Abstract: Multi-agent workflows require task-relevant information to be shared across agents, while irrelevant, stale, unverified, or incompatible information

EnterpriseBench: Benchmarking LLM Agents on Enterprise-Level Strategic Reasoning and Decision-Making

arXiv:2609.37658v1 Announce Type: new Abstract: LLM agents are increasingly expected to support enterprise workflows, where tasks often involve missing information, uncertainty, feedback, and long-ter

A Function-Level Vulnerability Score Measures Flag Rate More Than the Model: Protocol Effects on Paired Benchmarks

arXiv:2609.32890v1 Announce Type: new Abstract: Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured

CoHuB: A Simulation Benchmark for Multi-Humanoid Collaboration

arXiv:2609.34782v1 Announce Type: cross Abstract: Many physical tasks in human environments require collaboration, from assisting a partner to jointly manipulating an object. Yet, existing humanoid be

EEGAgentBench: Benchmarking LLM Agents on Short- and Long-Horizon EEG Analysis

arXiv:2609.31632v1 Announce Type: new Abstract: Electroencephalography (EEG) analysis is evolving from short-segment classification toward long-horizon interpretation that demands iterative evidence a

Energy Vision--Language--Action: A Controlled Multimodal Benchmark for Intent-Conditioned Residential Energy Management

arXiv:2609.31648v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are studied mainly in robotics, where visual observations and language instructions are mapped to physical actions.

EyeVQA: Benchmarking Ophthalmic Vision-Language Models from Recognition to Spatial Grounding

arXiv:2609.32352v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown increasing potential for medical image understanding, yet their capabilities in ophthalmic imaging remain insuf

FOCUS: Benchmarking Retinal Model Generalization from Foundation Vision Encoders to Multimodal LLMs

arXiv:2609.33158v1 Announce Type: new Abstract: Progress in AI-based retinal image analysis has advanced with foundation models, yet evaluating their reliability remains challenging. Performance repor

Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models

arXiv:2609.33812v1 Announce Type: cross Abstract: Repeated runs of the same coding agent are known to give different benchmark scores. We ask what that variation means for a team running an agent on i

USAI-Quant: A Quantitative Reasoning Benchmark for Vision-Language Models in Built Environments

arXiv:2609.32813v1 Announce Type: new Abstract: Large vision-language models (VLMs) have emerged as a powerful paradigm for urban and spatial AI. However, current state-of-the-art large VLMs still str

When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenes

arXiv:2609.34480v1 Announce Type: new Abstract: Reliable visual question answering requires correct answers when evidence is sufficient and abstention when it is not. We introduce a benchmark that con

Benchmarking Attention for Tabular Foundation Models

arXiv:2609.31306v1 Announce Type: new Abstract: Tabular in-context learners such as TabPFN, Mitra, or ConTextTab rely on alternating row and column attention over 2D sequences of latent embeddings. Th

28 Sept 2026

Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

arXiv:2609.31587v1 Announce Type: cross Abstract: We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it

28 Sept 2026

PriceBench: A Diagnostic Benchmark for Price, Quality, and Brand Preferences in LLM Booking Agents

arXiv:2609.31468v1 Announce Type: cross Abstract: LLMs increasingly act as purchasing agents, which makes the LLM, not the user, the one choosing among the options that satisfy a request; its preferen

28 Sept 2026

CinematicVQA: Benchmarking Film-Grammar Reasoning in Large Vision-Language Models

arXiv:2609.28813v1 Announce Type: new Abstract: Cinematography, the craft of visual storytelling through framing, lighting, and camera operation, fundamentally shapes how audiences perceive and emotio

DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

arXiv:2603.18612v2 Announce Type: replace Abstract: We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev an

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

arXiv:2609.28876v1 Announce Type: new Abstract: We introduce Forecast-Dojo, a replayable environment for benchmarking and training LLM forecasting agents. It combines resolved prediction-market questi

RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation

arXiv:2609.29028v1 Announce Type: new Abstract: In this paper, we propose RGBD20K, a novel dataset for facilitating the development of more robust and general RGB-D semantic segmentation by encompassi

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

arXiv:2609.29445v1 Announce Type: new Abstract: We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sente

Classifying Interpretive Canons at the Sentence Level: A Benchmark from the German Federal Constitutional Court

arXiv:2609.26945v1 Announce Type: new Abstract: Judicial reasoning remains challenging for large language models (LLMs) to analyze. This paper contributes a sentence-level benchmark for evaluating the

24 Sept 2026

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

arXiv:2609.26942v1 Announce Type: cross Abstract: Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a rec

24 Sept 2026

VMMU: A Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark

arXiv:2508.13680v5 Announce Type: replace Abstract: We introduce VMMU, a Vietnamese Multitask Multimodal Understanding and Reasoning Benchmark designed to evaluate how vision-language models (VLMs) in

24 Sept 2026

Building Trustworthy Mental Health Benchmarks on Bluesky: A Validation-Aware Weak-Supervision Framework

arXiv:2609.22696v1 Announce Type: new Abstract: Decentralized social media platforms create new opportunities and challenges for computational mental health research because data access, moderation, l

23 Sept 2026

IntLawNER: A Named Entity Recognition Dataset and Benchmark in International Law

arXiv:2609.22529v1 Announce Type: new Abstract: International law provides the normative framework through which states coordinate action, regulate armed conflict, and protect human rights, yet its te

23 Sept 2026

MOLA LiDAR-Inertial Odometry (MOLA-LIO) on the COMFORT Localization Benchmark

model-releasesarxiv-cs-ro

arXiv:2609.25813v1 Announce Type: new Abstract: This short report documents our entry to the COMFORT Localization Benchmark (IROS 2026), evaluated on the GrandTour dataset recorded with the Boxi paylo

23 Sept 2026

Ask Before It Tells: Benchmark-to-Robot Body-Cue Transfer for a Question-First Bedside Robot

model-releasesarxiv-cs-ro

arXiv:2609.24099v1 Announce Type: new Abstract: Body-cue recognition can support assistive robots, but benchmark accuracy does not guarantee reliable behavior under a robot-camera viewpoint. We presen

ChemCLIR-Bench: Benchmarking Cross-Lingual Information Retrieval in Multilingual Chemical Patents

arXiv:2609.23231v1 Announce Type: new Abstract: Cross-lingual information retrieval (CLIR) is increasingly important in multi-national industries, where critical technical evidence may exist in a diff

Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarking

arXiv:2609.23825v1 Announce Type: new Abstract: We present a comprehensive benchmark of Federated Learning (FL) for multilingual Automatic Speech Recognition (ASR), evaluating four Speech-LLM architec

GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents

arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research

Leakage-Safe Empirical Benchmarking of EEG-Based Machine Learning Pipelines for Dementia Classification

arXiv:2609.22092v1 Announce Type: cross Abstract: Electroencephalography (EEG) is a low-cost and non-invasive signal source for dementia screening, yet existing EEG-based studies remain difficult to c

MMS-VPR: A Fine-Grained Multimodal Street-Level Visual Place Recognition Dataset and Evaluation Benchmark for Dense Pedestrian Environments

arXiv:2505.12254v3 Announce Type: replace Abstract: Existing visual place recognition (VPR) datasets predominantly rely on vehicle-mounted imagery, offer limited multimodal diversity, and underreprese

PII-TRACE: A Benchmark for Context-Aware PII Detection in Multi-Turn LLM Conversations

arXiv:2609.22200v1 Announce Type: new Abstract: LLM assistants and agentic systems log long multi-turn conversations. AI providers often scan these conversations for Personally Identifiable Informatio

Safety

RSPDBench: Benchmarking Vision Foundation Models on Earth Observation Tasks Under Physically Grounded Remote-Sensing Product Degradations

arXiv:2609.23427v1 Announce Type: new Abstract: Vision foundation models targeting Earth observation (EO) tasks are commonly evaluated on clean downstream benchmarks, but operational EO products can a

safetyarxiv-cs-cv

Research

TempCore: Are Video QA Benchmarks Temporally Grounded?

arXiv:2509.01167v3 Announce Type: replace-cross Abstract: Vision-language models (VLMs) can ingest only a limited number of video frames, making frame selection a practical necessity. But do current V

researcharxiv-cs-cl

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

arXiv:2609.21763v1 Announce Type: new Abstract: A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about

21 Sept 2026

BrainWideBench: Benchmarking large-scale pretraining and across-animal transfer in multi-region neural recordings

arXiv:2609.22064v1 Announce Type: new Abstract: Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question o