Search: “benchmark”

Artificial Intelligence in Number Theory: LLMs for Algorithm Generation and Ensemble Methods for Conjecture Verification

arXiv:2504.19451v3 Announce Type: cross Abstract: This paper presents two concrete applications of Artificial Intelligence to algorithmic and analytic number theory. Recent benchmarks of large languag

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Beyond Accuracy: Evaluating Strategy Diversity in LLM Mathematical Reasoning

DGX agent

arXiv:2605.09292v1 Announce Type: new Abstract: Large language models now achieve high final-answer accuracy on mathematical reasoning benchmarks, but accuracy alone does not capture reasoning flexibi

model-releasesarxiv-cs-ai

12 May 2026

Beyond Local Edits: Embedding-Virtualized Knowledge for Broader Evaluation and Preservation of Model Editing

arXiv:2602.01977v2 Announce Type: replace Abstract: Knowledge editing methods for large language models are commonly evaluated using predefined benchmarks that assess edited facts together with a limi

BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

arXiv:2605.09041v1 Announce Type: new Abstract: Bias audits of large language models now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in

Chinese Cyberbullying Detection: Dataset, Method, and Validation

arXiv:2505.20654v2 Announce Type: replace-cross Abstract: Existing cyberbullying detection benchmarks were organized by the polarity of speech, such as 'offensive' and 'non-offensive', which were esse

Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluation

arXiv:2605.08522v1 Announce Type: new Abstract: The evaluation of Large Language Models (LLMs) faces a critical challenge in construct validity, where fragmented benchmarks and ad hoc metrics frequent

Equilibrium Residuals Expose Three Regimes of Matrix-Game Strategic Reasoning in Language Models

arXiv:2605.10410v1 Announce Type: new Abstract: Large language models can score well on named game-theory benchmarks while failing on the same strategic computation once semantic cues are removed. We

From Pixels to Concepts: Do Segmentation Models Understand What They Segment?

model-releasesarxiv-cs-cv

arXiv:2605.09591v1 Announce Type: new Abstract: Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything M

MaD Physics: Evaluating information seeking under constraints in physical environments

arXiv:2605.10820v1 Announce Type: new Abstract: Scientific discovery is fundamentally a resource-constrained process that requires navigating complex trade-offs between the quality and quantity of mea

Phoenix-VL 1.5 Medium Technical Report

arXiv:2605.10391v1 Announce Type: cross Abstract: We introduce Phoenix-VL 1.5 Medium, a 123B-parameter natively multimodal and multilingual foundation model, adapted to regional languages and the Sing

Prediction Bottlenecks Don't Discover Causal Structure (But Here's What They Actually Do)

arXiv:2605.09169v1 Announce Type: cross Abstract: A Mamba state-space model trained only for next-step prediction appears to recover Granger-causal structure through a simple readout S = |W_{out} W_{i

Reasoning emerges from constrained inference manifolds in large language models

arXiv:2605.08142v1 Announce Type: cross Abstract: Reasoning in large language models is predominantly evaluated through labeled benchmarks, conflating task performance with the quality of internal inf

SciLT: Long-tailed Image Classification under Scientific Image Domains

arXiv:2604.03687v2 Announce Type: replace Abstract: Long-tailed recognition has benefited from foundation models and fine-tuning paradigms, yet existing studies and benchmarks are mainly confined to n

researcharxiv-cs-cv

SeePhys Pro: Diagnosing Modality Transfer and Blind-Training Effects in Multimodal RLVR for Physics Reasoning

arXiv:2605.09266v1 Announce Type: new Abstract: We introduce SeePhys Pro, a fine-grained modality transfer benchmark that studies whether models preserve the same reasoning capability when critical in

The autoPET3 Challenge: Automated Lesion Segmentation in Whole-Body PET/CT nicode{x2013} Multitracer Multicenter Generalization

arXiv:2605.05775v2 Announce Type: replace-cross Abstract: We report the design and results of the third autoPET challenge (MICCAI 2024), which benchmarked automated lesion segmentation in whole-body P

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605.09883v1 Announce Type: cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores gen

researcharxiv-cs-ai

Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization

arXiv:2605.10405v1 Announce Type: new Abstract: Selecting the best large language model (LLM) for a fixed benchmark is often expensive, since exhaustive evaluation requires running every model on ever

Can Agents Price a Reaction? Evaluating LLMs on Chemical Cost Reasoning

arXiv:2605.07251v1 Announce Type: new Abstract: Large Language Models (LLMs) have become increasingly capable as tool-using agents, with benchmarks spanning diverse general agentic tasks. Yet rigorous

Safety

Direction-Flipped Influence Audits Reveal Hidden Structure in Moral Choices of LLMs

arXiv:2602.22831v2 Announce Type: replace-cross Abstract: Moral benchmarks for LLMs typically score models on context-free prompts, implicitly treating the measured choice rate as stable. We test this

safetyarxiv-cs-ai

FactoryBench: Evaluating Industrial Machine Understanding

arXiv:2605.07675v1 Announce Type: new Abstract: We introduce FactoryBench, a benchmark for evaluating time-series models and LLMs on machine understanding over industrial robotic telemetry. Q&A pairs

Safety

From 0-Order Selection to 2-Order Judgment: Combinatorial Hardening Exposes Compositional Failures in Frontier LLMs

arXiv:2605.07268v1 Announce Type: new Abstract: Multiple-choice reasoning benchmarks face dual challenges: rapid saturation from advancing models and data contamination that undermines static evaluati

safetyarxiv-cs-cl

Query-efficient model evaluation using cached responses

arXiv:2605.07096v1 Announce Type: cross Abstract: Evaluating a new model on an existing benchmark is often necessary to understand its behavior before deployment. For modern evaluation frameworks, gen

Structure Over Scale: Learning Visual Reasoning from Pedagogical Video

model-releasesarxiv-cs-cv

arXiv:2601.23251v2 Announce Type: replace Abstract: State-of-the-art vision-language models (VLMs) score impressively on video benchmarks yet stumble on basic visual reasoning tasks involving spatial

The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

arXiv:2605.07186v1 Announce Type: cross Abstract: Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect t

Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion

arXiv:2605.07013v1 Announce Type: new Abstract: Diffusion language models (DLMs) promise parallel, order-agnostic generation, but on standard benchmarks they have historically lagged behind autoregres

Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priors

arXiv:2512.06393v5 Announce Type: replace-cross Abstract: Large language models (LLMs) achieve high accuracy on many reasoning benchmarks but remain brittle under structural perturbations of rule-base

7 May 2026

AcademiClaw: When Students Set Challenges for AI Agents

arXiv:2605.02661v1 Announce Type: new Abstract: Benchmarks within the OpenClaw ecosystem have thus far evaluated exclusively assistant-level tasks, leaving the academic-level capabilities of OpenClaw

ISAAC: Auditing Causal Reasoning in Deep Models for Drug-Target Interaction

arXiv:2605.02962v1 Announce Type: new Abstract: Deep learning models for drug--target interaction (DTI) prediction often achieve strong benchmark performance without necessarily relying on mechanistic

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

arXiv:2605.00964v1 Announce Type: cross Abstract: Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human-AI wor

Local Ai

SHIELD: A Diverse Clinical Note Dataset and Distilled Small Language Models for Enterprise-Scale De-identification

arXiv:2605.03301v1 Announce Type: new Abstract: De-identification of clinical text remains essential for secondary use of electronic health records (EHRs), yet public benchmarks such as i2b2 2006/2014

local-aiarxiv-cs-cl

StateVLM: A State-Aware Vision-Language Model for Robotic Affordance Reasoning

model-releasesarxiv-cs-cv

arXiv:2605.03927v1 Announce Type: new Abstract: Vision-language models (VLMs) have shown remarkable performance in various robotic tasks, as they can perceive visual information and understand natural

TriBench-Ko: Evaluating LLM Risks in Judicial Workflows

arXiv:2605.03792v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly integrated into legal workflows. However, existing benchmarks primarily address proxy tasks, such as bar e

Agents

WMF-AM: Probing LLM Working Memory via Depth-Parameterized Cumulative State Tracking

arXiv:2603.27343v2 Announce Type: replace Abstract: Existing large language models (LLMs) evaluations use fixed-difficulty benchmarks that cannot adapt as models improve, and rarely isolate specific c

agentsarxiv-cs-ai

Counting as a minimal probe of language model reliability

arXiv:2605.02028v1 Announce Type: new Abstract: Large language models perform strongly on benchmarks in mathematical reasoning, coding and document analysis, suggesting a broad ability to follow instr

researcharxiv-cs-cl

Hardware

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

arXiv:2510.09883v2 Announce Type: replace Abstract: Large reasoning models (LRMs) achieve state-of-the-art performance on challenging benchmarks by generating long chains of intermediate steps, but th

hardwarearxiv-cs-cl

Evaluating LLMs on Large-Scale Graph Property Estimation via Random Walks

arXiv:2605.01484v1 Announce Type: new Abstract: With the rapidly improving reasoning abilities of Large Language Models (LLMs), there is also a rising demand to use them in a wide variety of domains.

MolViBench: Evaluating LLMs on Molecular Vibe Coding

arXiv:2605.02351v1 Announce Type: new Abstract: Molecular Vibe Coding, a paradigm where chemists interact with LLMs to generate executable programs for molecular tasks, has emerged as a flexible alter

Psychologically Potent, Computationally Invisible: LLMs Generate Social-Comparison Triggers They Fail to Detect

arXiv:2605.01017v1 Announce Type: new Abstract: We introduce Xiaohongshu Social Comparison Reader Elicitation (XHS-SCoRE), a reader-grounded benchmark for detecting if a text-only Xiaohongshu (RedNote

Toward Culturally Grounded Natural Language Processing

arXiv:2603.26013v2 Announce Type: replace Abstract: Multilingual NLP is often treated as a route to global inclusion, but linguistic coverage and cultural competence frequently diverge. This paper syn

Evaluating the Architectural Reasoning Capabilities of LLM Provers via the Obfuscated Natural Number Game

arXiv:2605.00677v1 Announce Type: new Abstract: While Large Language Models have achieved notable success on formal mathematics benchmarks such as MiniF2F, it remains unclear whether these results ste

4 May 2026

ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation

arXiv:2507.14201v3 Announce Type: replace-cross Abstract: We present ExCyTIn-Bench, the first benchmark to Evaluate an LLM agent X on the task of Cyber Threat Investigation through security questions

4 May 2026

SCAN: Structured Capability Assessment and Navigation for LLMs

arXiv:2505.06698v4 Announce Type: replace Abstract: Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives t

researcharxiv-cs-cl

4 May 2026

Activation Function Design Sustains Plasticity in Continual Learning

arXiv:2509.22562v4 Announce Type: replace-cross Abstract: In independent, identically distributed (i.i.d.) training regimes, activation functions have been benchmarked extensively, and their differenc

researcharxiv-cs-ai

1 May 2026

AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answering

arXiv:2601.12248v2 Announce Type: replace-cross Abstract: Recent advances in audio-aware large language models have shown strong performance on audio question answering. However, existing benchmarks m

Evaluating LLM Safety Under Repeated Inference via Accelerated Prompt Stress Testing

arXiv:2602.11786v2 Announce Type: replace Abstract: Traditional benchmarks for large language models (LLMs), such as HELM and AIR-BENCH, primarily assess safety through breadth-oriented evaluation acr

Frontier Coding Agents Can Now Implement an AlphaZero Self-Play Machine Learning Pipeline For Connect Four That Performs Comparably to an External Solver

arXiv:2604.25067v1 Announce Type: cross Abstract: Forecasting when AI systems will become capable of meaningfully accelerating AI research is a central challenge for AI safety. Existing benchmarks mea

Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models

arXiv:2604.25591v1 Announce Type: cross Abstract: Recent audio-aware large language models (ALLMs) have demonstrated strong capabilities across diverse audio understanding and reasoning tasks, but the

A Large-Scale, Cross-Disciplinary Corpus of Systematic Reviews