Search: “benchmark”

The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue

arXiv:2606.01901v1 Announce Type: cross Abstract: We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image ge

2 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

A Persona-Based Evaluation Framework for Pluralistic Alignment in Generative AI

DGX agent

arXiv:2605.31021v1 Announce Type: new Abstract: Current alignment paradigms for generative artificial intelligence rely predominantly on monolithic benchmarking frameworks that reduce the plurality of

safetyarxiv-cs-ai

1 Jun 2026

Research

'Intelegi Romaneste?'' A Recipe for Romanian Vision-Language Models

arXiv:2605.31401v1 Announce Type: new Abstract: Vision-Language Models (VLMs) largely follow the text-only LLM trajectory, excelling on English benchmarks but sharply degrading on low-resource languag

researcharxiv-cs-cl

1 Jun 2026

LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

arXiv:2605.30434v1 Announce Type: cross Abstract: Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability

1 Jun 2026

TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices

arXiv:2605.31113v1 Announce Type: new Abstract: Automatically detecting machine-generated text (MGT) is critical to maintaining the knowledge integrity of user-generated content (UGC) platforms such a

1 Jun 2026

BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval

model-releasesarxiv-cs-cv

arXiv:2605.30235v1 Announce Type: new Abstract: We present BullingerDB, a large-scale benchmark dataset for historical document analysis based on the correspondence of Heinrich Bullinger (1504-1575).

Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset

arXiv:2605.29365v1 Announce Type: new Abstract: Formality transfer is commonly framed as a symmetric bidirectional task between informal and formal registers. We argue that this framing conceals a sup

ExCAM: Explainable Cultural Awareness Metrics

arXiv:2605.29897v1 Announce Type: new Abstract: Evaluating the cultural awareness of large language models is crucial to ensure the fairness of generated text and the generalizability of applications

GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human

arXiv:2605.28882v1 Announce Type: cross Abstract: With the rapid advancement of large language models, evaluating human-likeness in open-ended conversation has become increasingly important. However,

Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

arXiv:2511.04934v3 Announce Type: replace Abstract: Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing

No More K-means:Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

arXiv:2605.30120v1 Announce Type: cross Abstract: Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-le

PhoneWorld: Scaling Phone-Use Agent Environments

arXiv:2605.29486v1 Announce Type: cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Ex

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

arXiv:2605.30284v1 Announce Type: new Abstract: Scientific discovery is an inherently creative and uncertain process, requiring reasoning beyond the recall of known knowledge. While many benchmarks ha

PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

arXiv:2508.15180v3 Announce Type: replace Abstract: High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language

Selection Hyper-heuristics Can Automatically Adjust the Learning Period to Optimally Solve Pseudo-Boolean Problems

arXiv:2605.29916v1 Announce Type: cross Abstract: The Random Gradient hyper-heuristic was recently shown to be able to learn the optimal neighbourhood size when optimizing the LeadingOnes benchmark vi

Striding Across Reynolds Numbers: Representation Geometry in Neural PDE Generalisation

arXiv:2605.30112v1 Announce Type: new Abstract: Cross-Reynolds generalisation in neural PDE solvers remains poorly characterised. On the canonical forced 2D Navier-Stokes benchmark, a trained Fourier

The Trust Paradox: How CS Researchers Engage LLM Leaderboards

arXiv:2605.28966v1 Announce Type: new Abstract: Large language model (LLM) leaderboards rank AI models using standardized benchmarks and have become highly visible across computer science, despite kno

Adaptive Reservoir Computing for Multi-Scenario Chaotic System Forecasting

arXiv:2605.28145v1 Announce Type: new Abstract: We present an adaptive reservoir computing framework for the CTF-4-Science Lorenz benchmark, which evaluates machine learning models across twelve disti

Disentangling Language Roles in Multilingual LLM Task Execution

arXiv:2605.27649v1 Announce Type: new Abstract: Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expa

Federated Learning for Multivariate Time Series Anomaly Detection in Industrial Automation

arXiv:2605.27486v1 Announce Type: new Abstract: Federated learning (FL) has broadened the horizon for multivariate time series anomaly detection (MTSAD). However, benchmarking such anomaly detection m

HardMTBench: Stress-Testing Chinese-English Translation on Knowledge-Intensive Domains

arXiv:2605.28315v1 Announce Type: new Abstract: General-purpose machine translation benchmarks such as FLORES-200 have reached a saturation regime on Chinese-English pairs, where modern large language

Models That Know How Evaluations Are Designed Score Safer

arXiv:2605.28591v1 Announce Type: cross Abstract: The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings. Prior work has identified tes

Research

ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generation

arXiv:2605.27709v1 Announce Type: new Abstract: Mathematical reasoning benchmarks are vital for evaluating large language models (LLMs), but many are static and repeatedly exposed through public evalu

researcharxiv-cs-cl

The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment

arXiv:2605.28464v1 Announce Type: cross Abstract: Legal Judgment Prediction (LJP) has become a core benchmark for evaluating AI in the criminal legal domain, but it only sees criminal cases that have

AI evaluation may bias perceptions: The importance of context in interpreting academic writing

arXiv:2605.26662v1 Announce Type: cross Abstract: This paper examines how estimates of AI use in scientific writing can be biased when evaluation methods ignore contextual differences across countries

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

arXiv:2603.03194v2 Announce Type: replace Abstract: Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software en

Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning

arXiv:2605.26789v1 Announce Type: new Abstract: Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answ

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

model-releasesarxiv-cs-cv

arXiv:2605.26244v1 Announce Type: new Abstract: Audio-visual generation is rapidly advancing from short clips to minute-long content, while existing evaluation protocols remain largely confined to sho

LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness

arXiv:2605.26438v1 Announce Type: cross Abstract: Large language models can recognize when they are being evaluated (evaluation awareness) and behave differently because of that, which undermines the

Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation

arXiv:2605.27210v1 Announce Type: cross Abstract: We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing f

Agents

AION: Next-Generation Tasks and Practical Harness for Time Series

arXiv:2605.25045v1 Announce Type: new Abstract: Time series research is moving beyond fixed forecasting benchmarks toward realistic tasks that combine prediction, contextual reasoning, tool use, and s

agentsarxiv-cs-ai

AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora

arXiv:2605.25382v1 Announce Type: new Abstract: Evidence construction systems--chunk retrieval, agent memory, knowledge-graph traversal, and thematic indexing--are evaluated on separate benchmarks wit

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

arXiv:2605.24305v1 Announce Type: cross Abstract: Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason

INDUCTION: Finite-Structure Concept Synthesis in First-Order Logic

arXiv:2602.18956v3 Announce Type: replace Abstract: We introduce INDUCTION, a benchmark for finite structure concept synthesis in first order logic. Given small finite relational worlds with extension

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

arXiv:2605.25240v1 Announce Type: cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative ju

Research

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

arXiv:2605.23965v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations r

researcharxiv-cs-ai

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

model-releasesarxiv-cs-ro

arXiv:2510.03827v2 Announce Type: replace-cross Abstract: LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluat

PDEInvBench: A Comprehensive Dataset and Design Space Exploration of Neural Networks for PDE Inverse Problems

arXiv:2605.25353v1 Announce Type: new Abstract: Inverse problems in partial differential equations (PDEs) involve estimating the physical parameters of a system from observed spatiotemporal solution f

TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning

arXiv:2605.25850v1 Announce Type: cross Abstract: This paper investigates large language model (LLM) abstention learning, specifically using ternary reward, which incentivize truthfulness in large lan

Applications

Towards a Universal Causal Reasoner

arXiv:2605.24873v1 Announce Type: cross Abstract: Despite the importance of causal reasoning, training LLMs to reason causally remains underexplored. Existing data efforts mostly focus on benchmarking

applicationsarxiv-cs-ai

VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenes

arXiv:2509.25339v3 Announce Type: replace-cross Abstract: Is basic visual understanding really solved in state-of-the-art VLMs? We present VisualOverload, a slightly different visual question answerin

When Reasoning Hurts: Source-Aware Evaluation of Frontier LLMs for Clinical SOAP Note Generation

arXiv:2605.24902v1 Announce Type: cross Abstract: Reasoning-enabled LLMs perform strongly on medical reasoning benchmarks, but it remains unclear whether these gains transfer to structured clinical do

Ax-Prover: A Deep Reasoning Agentic Framework for Theorem Proving in Mathematics and Quantum Physics

arXiv:2510.12787v4 Announce Type: replace Abstract: We present Ax-Prover, a multi-agent system for automated theorem proving in Lean that can solve problems across diverse scientific domains and opera

25 May 2026

Safety

Energy per Successful Goal: Goal-Level Energy Accounting for Agentic AI Systems

arXiv:2605.22883v1 Announce Type: new Abstract: Current AI energy benchmarks measure consumption at the granularity of a single model invocation or training run. For classical single-turn workloads th

safetyarxiv-cs-ai

25 May 2026

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

arXiv:2605.23238v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior i

25 May 2026

I-SAFE: Wasserstein Coherence Metrics for Structural Auditing of Scientific AI Models

arXiv:2605.21731v1 Announce Type: new Abstract: Deep learning models are increasingly used in scientific prediction tasks where strong benchmark performance is often interpreted as evidence of scienti

23 May 2026

Code Researcher: Deep Research Agent for Large Systems Code and Commit History

arXiv:2506.11060v2 Announce Type: replace-cross Abstract: Large Language Model (LLM)-based coding agents have shown promising results on coding benchmarks, but their effectiveness on systems code rema

22 May 2026

Learning Spatiotemporal Sensitivity in Video LLMs via Counterfactual Reinforcement Learning