Search: “benchmark”

MedOpenClaw and MedFlowBench: Auditing Medical Agents in Full-Study Workflows

arXiv:2603.24649v2 Announce Type: replace Abstract: Medical imaging benchmarks often evaluate VLMs on pre-selected 2D images, slices, crops, or patches, making evaluation closer to visual recognition.

14 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism

DGX agent

arXiv:2605.12524v1 Announce Type: cross Abstract: We introduce ProofGrid, a benchmark suite for evaluating LLM reasoning through machine-checkable proofs rather than final answers alone. ProofGrid con

model-releasesarxiv-cs-ai

14 May 2026

Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents

arXiv:2602.16246v3 Announce Type: replace Abstract: Interactive large language model (LLM) agents operating via multi-turn dialogue and multi-step tool calling are increasingly used in production. Ben

14 May 2026

ClinicalBench: Stress-Testing Assertion-Aware Retrieval for Cross-Admission Clinical QA on MIMIC-IV

arXiv:2605.11143v1 Announce Type: new Abstract: Reasoning benchmarks measure clinical performance on clean inputs. We evaluate the step before reasoning: retrieval over real EHR notes, where negation,

GKnow: Measuring the Entanglement of Gender Bias and Factual Gender

arXiv:2605.12299v1 Announce Type: new Abstract: Recent works have analyzed the impact of individual components of neural networks on gendered predictions, often with a focus on mitigating gender bias.

Nautilus: From One Prompt to Plug-and-Play Robot Learning

model-releasesarxiv-cs-ro

arXiv:2605.11665v1 Announce Type: new Abstract: Robot learning research is fragmented across policy families, benchmark suites, and real robots; each implementation is entangled with the others in a c

NavOL: Navigation Policy with Online Imitation Learning

model-releasesarxiv-cs-ro

arXiv:2605.11762v1 Announce Type: new Abstract: Learning robust navigation policies remains a core challenge in robotics. Offline imitation learning suffers from distribution shift and compounding err

The Scaling Law of Evaluation Failure: Why Simple Averaging Collapses Under Data Sparsity and Item Difficulty Gaps, and How Item Response Theory Recovers Ground Truth Across Domains

arXiv:2605.11205v1 Announce Type: new Abstract: Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substan

AlphaExploitem: Going Beyond the Nash Equilibrium in Poker by Learning to Exploit Suboptimal Play

arXiv:2605.09150v1 Announce Type: new Abstract: Poker is an imperfect information game that has served as a long-standing benchmark for decision-making under uncertainty. To maximize utility beyond th

Artificial Intelligence in Number Theory: LLMs for Algorithm Generation and Ensemble Methods for Conjecture Verification

arXiv:2504.19451v3 Announce Type: cross Abstract: This paper presents two concrete applications of Artificial Intelligence to algorithmic and analytic number theory. Recent benchmarks of large languag

Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

arXiv:2605.09292v1 Announce Type: new Abstract: Large language models now achieve high final-answer accuracy on mathematical reasoning benchmarks, but accuracy alone does not capture reasoning flexibi

Beyond Local Edits: Embedding-Virtualized Knowledge for Broader Evaluation and Preservation of Model Editing

arXiv:2602.01977v2 Announce Type: replace Abstract: Knowledge editing methods for large language models are commonly evaluated using predefined benchmarks that assess edited facts together with a limi

BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

arXiv:2605.09041v1 Announce Type: new Abstract: Bias audits of large language models now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in

Chinese Cyberbullying Detection: Dataset, Method, and Validation

arXiv:2505.20654v2 Announce Type: replace-cross Abstract: Existing cyberbullying detection benchmarks were organized by the polarity of speech, such as 'offensive' and 'non-offensive', which were esse

Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation

arXiv:2605.08522v1 Announce Type: new Abstract: The evaluation of Large Language Models (LLMs) faces a critical challenge in construct validity, where fragmented benchmarks and ad hoc metrics frequent

Equilibrium Residuals Expose Three Regimes of Matrix-Game Strategic Reasoning in Language Models

arXiv:2605.10410v1 Announce Type: new Abstract: Large language models can score well on named game-theory benchmarks while failing on the same strategic computation once semantic cues are removed. We

From Pixels to Concepts: Do Segmentation Models Understand What They Segment?

arXiv:2605.09591v1 Announce Type: new Abstract: Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything M

MaD Physics: Evaluating information seeking under constraints in physical environments

arXiv:2605.10820v1 Announce Type: new Abstract: Scientific discovery is fundamentally a resource-constrained process that requires navigating complex trade-offs between the quality and quantity of mea

Phoenix-VL 1.5 Medium Technical Report

arXiv:2605.10391v1 Announce Type: cross Abstract: We introduce Phoenix-VL 1.5 Medium, a 123B-parameter natively multimodal and multilingual foundation model, adapted to regional languages and the Sing

Prediction Bottlenecks Don't Discover Causal Structure (But Here's What They Actually Do)

arXiv:2605.09169v1 Announce Type: cross Abstract: A Mamba state-space model trained only for next-step prediction appears to recover Granger-causal structure through a simple readout S = |W_{out} W_{i

Reasoning emerges from constrained inference manifolds in large language models

arXiv:2605.08142v1 Announce Type: cross Abstract: Reasoning in large language models is predominantly evaluated through labeled benchmarks, conflating task performance with the quality of internal inf

Research

SciLT: Long-tailed Image Classification under Scientific Image Domains

arXiv:2604.03687v2 Announce Type: replace Abstract: Long-tailed recognition has benefited from foundation models and fine-tuning paradigms, yet existing studies and benchmarks are mainly confined to n

researcharxiv-cs-cv

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

arXiv:2605.09266v1 Announce Type: new Abstract: We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical in

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT nicode{x2013} Multitracer Multicenter Generalization

arXiv:2605.05775v2 Announce Type: replace-cross Abstract: We report the design and results of the third autoPET challenge (MICCAI 2024), which benchmarked automated lesion segmentation in whole-body P

Research

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605.09883v1 Announce Type: cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores gen

researcharxiv-cs-ai

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

arXiv:2605.10405v1 Announce Type: new Abstract: Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on ever

Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

arXiv:2605.07251v1 Announce Type: new Abstract: Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous

Safety

Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs

arXiv:2602.22831v2 Announce Type: replace-cross Abstract: Moral benchmarks for LLMs typically score models on context-free prompts, implicitly treating the measured choice rate as stable. We test this

safetyarxiv-cs-ai

FactoryBench: Evaluating Industrial Machine Understanding

arXiv:2605.07675v1 Announce Type: new Abstract: We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs

Safety

From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

arXiv:2605.07268v1 Announce Type: new Abstract: Multiple-choice reasoning benchmarks face dual challenges: rapid saturation from advancing models and data contamination that undermines static evaluati

safetyarxiv-cs-cl

Query-efficient model evaluation using cached responses

arXiv:2605.07096v1 Announce Type: cross Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, gen

Structure Over Scale: Learning Visual Reasoning from Pedagogical Video

arXiv:2601.23251v2 Announce Type: replace Abstract: State-of-the-art vision-language models (VLMs) score impressively on video benchmarks yet stumble on basic visual reasoning tasks involving spatial

The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

arXiv:2605.07186v1 Announce Type: cross Abstract: Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect t

Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

arXiv:2605.07013v1 Announce Type: new Abstract: Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregres

Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priors

arXiv:2512.06393v5 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve high accuracy on many reasoning benchmarks but remain brittle under structural perturbations of rule-base

7 May 2026

AcademiClaw: When Students Set Challenges for AI Agents

arXiv:2605.02661v1 Announce Type: new Abstract: Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw

ISAAC: Auditing Causal Reasoning in Deep Models for Drug-Target Interaction

arXiv:2605.02962v1 Announce Type: new Abstract: Deep learning models for drug--target interaction (DTI) prediction often achieve strong benchmark performance without necessarily relying on mechanistic

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

arXiv:2605.00964v1 Announce Type: cross Abstract: Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human-AI wor

Local Ai

SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification

arXiv:2605.03301v1 Announce Type: new Abstract: De-identification of clinical text remains essential for secondary use of electronic health records (EHRs), yet public benchmarks such as i2b2 2006/2014

local-aiarxiv-cs-cl

StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

arXiv:2605.03927v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown remarkable performance in various robotic tasks, as they can perceive visual information and understand natural

TriBench-Ko: Evaluating LLM Risks in Judicial Workflows

arXiv:2605.03792v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into legal workflows. However, existing benchmarks primarily address proxy tasks, such as bar e

Agents

WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking

arXiv:2603.27343v2 Announce Type: replace Abstract: Existing large language models (LLMs) evaluations use fixed-difficulty benchmarks that cannot adapt as models improve, and rarely isolate specific c

agentsarxiv-cs-ai

Research

Counting as a minimal probe of language model reliability

arXiv:2605.02028v1 Announce Type: new Abstract: Large language models perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instr

researcharxiv-cs-cl

Hardware

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

arXiv:2510.09883v2 Announce Type: replace Abstract: Large reasoning models (LRMs) achieve state-of-the-art performance on challenging benchmarks by generating long chains of intermediate steps, but th

hardwarearxiv-cs-cl

Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks

arXiv:2605.01484v1 Announce Type: new Abstract: With the rapidly improving reasoning abilities of Large Language Models (LLMs), there is also a rising demand to use them in a wide variety of domains.

MolViBench: Evaluating LLMs on Molecular Vibe Coding

arXiv:2605.02351v1 Announce Type: new Abstract: Molecular Vibe Coding, a paradigm where chemists interact with LLMs to generate executable programs for molecular tasks, has emerged as a flexible alter

Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison Triggers They Fail to Detect

arXiv:2605.01017v1 Announce Type: new Abstract: We introduce Xiaohongshu Social Comparison Reader Elicitation (XHS-SCoRE), a reader-grounded benchmark for detecting if a text-only Xiaohongshu (RedNote

Toward Culturally Grounded Natural Language Processing

arXiv:2603.26013v2 Announce Type: replace Abstract: Multilingual NLP is often treated as a route to global inclusion, but linguistic coverage and cultural competence frequently diverge. This paper syn