Search: “benchmark”

FATE-VLA:Failue-aware test generation for vision-language-action models

arXiv:2606.02307v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks t

researcharxiv-cs-ro

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

FVSpec: Real-World Property-Based Tests as Lean Challenges

DGX agent

arXiv:2606.01008v1 Announce Type: cross Abstract: We present a benchmark for evaluating AI models and agents on real-world formal software verification tasks. We first scrape 11,039 property-based tes

model-releasesarxiv-cs-ai

2 Jun 2026

Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures

arXiv:2510.24081v2 Announce Type: replace Abstract: To date, there exist almost no culturally-specific evaluation benchmarks for large language models (LLMs) that cover a large number of languages and

Momento: Evaluating Persistent Memory and Reasoning with Multi-Session Agentic Conversations

arXiv:2606.00832v1 Announce Type: new Abstract: Recent advances in agentic AI have enabled agents to complete complex tasks through tool use, reasoning, and multi-step planning. Yet existing benchmark

Perception First: A Frontier Native-Video Model with Self-Consistency for Implicit Video Question Answering

arXiv:2606.01485v1 Announce Type: new Abstract: We describe our submission to the VRR Challenge @ CVPR 2026, built on the ImplicitQA / VRR-QA benchmark~ite{implicitqa}: multiple-choice video question

Pre-Deployment Robustness Stress Testing for CT Segmentation Systems Using Clinically Motivated Multi-Corruption Augmentation

arXiv:2606.00491v1 Announce Type: cross Abstract: Deep learning-based CT segmentation systems often achieve high accuracy on clean benchmark images, but their performance may degrade under heterogeneo

ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search

arXiv:2606.01825v1 Announce Type: new Abstract: Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based

The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialogue

arXiv:2606.01901v1 Announce Type: cross Abstract: We introduce the Image Reconstruction Game, a fully automated benchmark in which a vision-language model issues corrective instructions to an image ge

Safety

A Persona-Based Evaluation Framework for Pluralistic Alignment in Generative AI

arXiv:2605.31021v1 Announce Type: new Abstract: Current alignment paradigms for generative artificial intelligence rely predominantly on monolithic benchmarking frameworks that reduce the plurality of

safetyarxiv-cs-ai

'Intelegi Romaneste?'' A Recipe for Romanian Vision-Language Models

arXiv:2605.31401v1 Announce Type: new Abstract: Vision-Language Models (VLMs) largely follow the text-only LLM trajectory, excelling on English benchmarks but sharply degrading on low-resource languag

researcharxiv-cs-cl

LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

arXiv:2605.30434v1 Announce Type: cross Abstract: Real-world data analysis is inherently iterative, yet existing benchmarks mostly evaluate isolated or short interactive tasks, leaving agents' ability

TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices

arXiv:2605.31113v1 Announce Type: new Abstract: Automatically detecting machine-generated text (MGT) is critical to maintaining the knowledge integrity of user-generated content (UGC) platforms such a

BullingerDB: A Dataset for Handwritten Text Recognition and Writer Retrieval

arXiv:2605.30235v1 Announce Type: new Abstract: We present BullingerDB, a large-scale benchmark dataset for historical document analysis based on the correspondence of Heinrich Bullinger (1504-1575).

Casual as an Anchor: Resolving Supervision Misalignment in Formality Transfer Dataset

arXiv:2605.29365v1 Announce Type: new Abstract: Formality transfer is commonly framed as a symmetric bidirectional task between informal and formal registers. We argue that this framing conceals a sup

ExCAM: Explainable Cultural Awareness Metrics

arXiv:2605.29897v1 Announce Type: new Abstract: Evaluating the cultural awareness of large language models is crucial to ensure the fairness of generated text and the generalizability of applications

GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human

arXiv:2605.28882v1 Announce Type: cross Abstract: With the rapid advancement of large language models, evaluating human-likeness in open-ended conversation has become increasingly important. However,

Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding

arXiv:2511.04934v3 Announce Type: replace Abstract: Unlearning in large language models (LLMs) is critical for regulatory compliance and for building ethical generative AI systems that avoid producing

No More K-means:Single-Stage Sparse Coding for Efficient Multi-Vector Retrieval

arXiv:2605.30120v1 Announce Type: cross Abstract: Multi-vector retrieval (MVR) models, exemplified by ColBERT, have established new benchmarks in retrieval accuracy by preserving fine-grained token-le

PhoneWorld: Scaling Phone-Use Agent Environments

arXiv:2605.29486v1 Announce Type: cross Abstract: A central bottleneck for phone-use agents is that controllable, reproducible environments covering real mobile behavior are hard to build at scale. Ex

ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure

arXiv:2605.30284v1 Announce Type: new Abstract: Scientific discovery is an inherently creative and uncertain process, requiring reasoning beyond the recall of known knowledge. While many benchmarks ha

PuzzleClone: A DSL-Powered Framework for Synthesizing Verifiable Data

arXiv:2508.15180v3 Announce Type: replace Abstract: High-quality mathematical and logical datasets with verifiable answers are essential for strengthening the reasoning capabilities of large language

Selection Hyper-heuristics Can Automatically Adjust the Learning Period to Optimally Solve Pseudo-Boolean Problems

arXiv:2605.29916v1 Announce Type: cross Abstract: The Random Gradient hyper-heuristic was recently shown to be able to learn the optimal neighbourhood size when optimizing the LeadingOnes benchmark vi

Striding Across Reynolds Numbers: Representation Geometry in Neural PDE Generalisation

arXiv:2605.30112v1 Announce Type: new Abstract: Cross-Reynolds generalisation in neural PDE solvers remains poorly characterised. On the canonical forced 2D Navier-Stokes benchmark, a trained Fourier

The Trust Paradox: How CS Researchers Engage LLM Leaderboards

arXiv:2605.28966v1 Announce Type: new Abstract: Large language model (LLM) leaderboards rank AI models using standardized benchmarks and have become highly visible across computer science, despite kno

Adaptive Reservoir Computing for Multi-Scenario Chaotic System Forecasting

arXiv:2605.28145v1 Announce Type: new Abstract: We present an adaptive reservoir computing framework for the CTF-4-Science Lorenz benchmark, which evaluates machine learning models across twelve disti

Disentangling Language Roles in Multilingual LLM Task Execution

arXiv:2605.27649v1 Announce Type: new Abstract: Multilingual LLMs are increasingly used when instruction, source content, and required response languages do not coincide. Existing benchmarks have expa

Federated Learning for Multivariate Time Series Anomaly Detection in Industrial Automation

arXiv:2605.27486v1 Announce Type: new Abstract: Federated learning (FL) has broadened the horizon for multivariate time series anomaly detection (MTSAD). However, benchmarking such anomaly detection m

HardMTBench: Stress-Testing Chinese-English Translation on Knowledge-Intensive Domains

arXiv:2605.28315v1 Announce Type: new Abstract: General-purpose machine translation benchmarks such as FLORES-200 have reached a saturation regime on Chinese-English pairs, where modern large language

Models That Know How Evaluations Are Designed Score Safer

arXiv:2605.28591v1 Announce Type: cross Abstract: The validity of AI safety evaluations depends on models behaving consistently across controlled and deployment settings. Prior work has identified tes

ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generation

arXiv:2605.27709v1 Announce Type: new Abstract: Mathematical reasoning benchmarks are vital for evaluating large language models (LLMs), but many are static and repeatedly exposed through public evalu

researcharxiv-cs-cl

The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment

arXiv:2605.28464v1 Announce Type: cross Abstract: Legal Judgment Prediction (LJP) has become a core benchmark for evaluating AI in the criminal legal domain, but it only sees criminal cases that have

AI evaluation may bias perceptions: The importance of context in interpreting academic writing

arXiv:2605.26662v1 Announce Type: cross Abstract: This paper examines how estimates of AI use in scientific writing can be biased when evaluation methods ignore contextual differences across countries

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

arXiv:2603.03194v2 Announce Type: replace Abstract: Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software en

Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning

arXiv:2605.26789v1 Announce Type: new Abstract: Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answ

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

arXiv:2605.26244v1 Announce Type: new Abstract: Audio-visual generation is rapidly advancing from short clips to minute-long content, while existing evaluation protocols remain largely confined to sho

LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness

arXiv:2605.26438v1 Announce Type: cross Abstract: Large language models can recognize when they are being evaluated (evaluation awareness) and behave differently because of that, which undermines the

Hardware

NVIDIA Blackwell Sets STAC-AI Record for LLM Inference in Finance

NVIDIA's Blackwell GB200 NVL72 architecture achieved the fastest-ever results on the STAC-AI benchmark for financial LLM inference, delivering up to 3.2x single-GPU performance improvements over the p

hardwarenvidia-developer

Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation

arXiv:2605.27210v1 Announce Type: cross Abstract: We adapt Microsoft's QuantumKatas -- a well-established quantum computing curriculum -- from Q# to Qiskit, the most widely-adopted quantum computing f

Agents

AION: Next-Generation Tasks and Practical Harness for Time Series

arXiv:2605.25045v1 Announce Type: new Abstract: Time series research is moving beyond fixed forecasting benchmarks toward realistic tasks that combine prediction, contextual reasoning, tool use, and s

agentsarxiv-cs-ai

AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora

arXiv:2605.25382v1 Announce Type: new Abstract: Evidence construction systems--chunk retrieval, agent memory, knowledge-graph traversal, and thematic indexing--are evaluated on separate benchmarks wit

ChaosBench-Logic v2: Evaluating LLM Logical Reasoning over Dynamical Systems at Scale

arXiv:2605.24305v1 Announce Type: cross Abstract: Standard accuracy on binary reasoning benchmarks hides critical failure modes: prior collapse, inconsistency under paraphrase, and inability to reason

INDUCTION: Finite-Structure Concept Synthesis in First-Order Logic

arXiv:2602.18956v3 Announce Type: replace Abstract: We introduce INDUCTION, a benchmark for finite structure concept synthesis in first order logic. Given small finite relational worlds with extension

JudgmentBench: Comparing Rubric and Preference Evaluation for Quality Assessment

arXiv:2605.25240v1 Announce Type: cross Abstract: Two methodologies dominate current practices of benchmarking: rubric-based scoring evaluates items against predefined criteria, whereas comparative ju

LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs

arXiv:2605.23965v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance on logical reasoning benchmarks, yet their reliability remains uncertain. Existing evaluations r

researcharxiv-cs-ai

LIBERO-PRO: Towards Robust and Fair Evaluation of Vision-Language-Action Models Beyond Memorization

model-releasesarxiv-cs-ro

arXiv:2510.03827v2 Announce Type: replace-cross Abstract: LIBERO has emerged as a widely adopted benchmark for evaluating Vision-Language-Action (VLA) models; however, its current training and evaluat

NVIDIA Vera CPU Is ‘Packing a Heavy-Hitting Punch’ Against Competition

model-releasesnvidia-blog

The shift to agentic AI creates a new CPU requirement for the AI factory: fast cores, massive memory bandwidth and the ability to sustain high performance when all cores are active. Initial benchmark

PDEInvBench: A Comprehensive Dataset and Design Space Exploration of Neural Networks for PDE Inverse Problems

arXiv:2605.25353v1 Announce Type: new Abstract: Inverse problems in partial differential equations (PDEs) involve estimating the physical parameters of a system from observed spatiotemporal solution f

TIAR: Trajectory-Informed Advantage Reweighting for LLM Abstention Learning

arXiv:2605.25850v1 Announce Type: cross Abstract: This paper investigates large language model (LLM) abstention learning, specifically using ternary reward, which incentivize truthfulness in large lan