Search: “evaluation”

Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators

arXiv:2605.12748v1 Announce Type: cross Abstract: Large language models (LLMs) can fluently generate student-like responses, making them attractive as simulated students for training and evaluating AI

researcharxiv-cs-ai

14 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

SpikeProphecy: A Large-Scale Benchmark for Autoregressive Neural Population Forecasting

DGX agent

arXiv:2605.12992v1 Announce Type: cross Abstract: Neural population models, which predict the joint firing of many simultaneously recorded neurons forward in time, are typically evaluated by a single

model-releasesarxiv-cs-lg

14 May 2026

Stress-Testing the Reasoning Competence of LLMs With Proofs Under Minimal Formalism

arXiv:2605.12524v1 Announce Type: cross Abstract: We introduce ProofGrid, a benchmark suite for evaluating LLM reasoning through machine-checkable proofs rather than final answers alone. ProofGrid con

14 May 2026

ASD-Bench: A Four-Axis Comprehensive Benchmark of AI Models for Autism Spectrum Disorder

arXiv:2605.11091v1 Announce Type: new Abstract: Automated ASD screening tools remain limited by single-architecture evaluations, axis-restricted assessment, and near-exclusive focus on adult cohorts,

ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?

arXiv:2605.11086v1 Announce Type: cross Abstract: AI agents are rapidly gaining capabilities that could significantly reshape cybersecurity, making rigorous evaluation urgent. A critical capability is

gym-invmgmt: An Open Benchmarking Framework for Inventory Management Methods

arXiv:2605.11355v1 Announce Type: new Abstract: Inventory-policy comparisons are often difficult to interpret because performance depends on the evaluation contract as much as on the policy itself. Di

Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring

arXiv:2605.12398v1 Announce Type: new Abstract: Estimating question difficulty is a critical component in evaluating and improving large language models (LLMs) for question answering (QA). Existing ap

safetyarxiv-cs-cl

Targeted Tests for LLM Reasoning: An Audit-Constrained Protocol

arXiv:2605.11599v1 Announce Type: new Abstract: Fixed reasoning benchmarks evaluate canonical prompts, but semantically valid changes in presentation can still change model behavior. Studies of prompt

researcharxiv-cs-lg

Beyond Isolation: A Unified Benchmark for General-Purpose Navigation

model-releasesarxiv-cs-ro

arXiv:2605.09441v1 Announce Type: new Abstract: The pursuit of general-purpose embodied agents is hindered by fragmented evaluation protocols that isolate navigation skills and fixate on specific robo

Debugging the Debuggers: Failure-Anchored Structured Recovery for Software Engineering Agents

arXiv:2605.08717v1 Announce Type: cross Abstract: Software engineering agents are increasingly deployed in evaluable engineering environments, yet post-failure recovery remains costly, manual, and ad

safetyarxiv-cs-ai

MathConstraint: Automated Generation of Verified Combinatorial Reasoning Instances for LLMs

arXiv:2605.08498v1 Announce Type: cross Abstract: We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfa

Outlier detection for patient monitoring and alerting

arXiv:2605.08955v1 Announce Type: new Abstract: We develop and evaluate a data-driven approach for detecting unusual (anomalous) patient-management decisions using past patient cases stored in electro

researcharxiv-cs-lg

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage

arXiv:2604.01527v3 Announce Type: replace-cross Abstract: Production deployment of AI coding agents requires fast, reproducible evaluation signals. Existing industrial practices trade off speed and fi

Agents

Results and Retrospective Analysis of the CODS 2025 AssetOpsBench Challenge

arXiv:2605.08518v1 Announce Type: new Abstract: Competition retrospectives are useful when they explain what a leaderboard measured, how hidden evaluation changed conclusions, and which design pattern

agentsarxiv-cs-ai

RewardHarness: Self-Evolving Agentic Post-Training

arXiv:2605.08703v1 Announce Type: new Abstract: Evaluating instruction-guided image edits requires rewards that reflect subtle human preferences, yet current reward models typically depend on large-sc

SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution

arXiv:2605.08366v1 Announce Type: new Abstract: We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test W

APEX: Assumption-free Projection-based Embedding eXamination Metric for Image Quality Assessment

arXiv:2605.07786v1 Announce Type: cross Abstract: As generative models achieve unprecedented visual quality, the gold standard for image evaluation remains traditional feature-distribution metrics (e.

safetyarxiv-cs-ai

CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers

arXiv:2605.07905v1 Announce Type: cross Abstract: Despite the rapid development of AI reviewers, evaluating such systems remains challenging: metrics favor overlap with human reviews over correctness.

Fidel-TS: A High-Fidelity Multimodal Benchmark for Time Series Forecasting

arXiv:2509.24789v4 Announce Type: replace Abstract: The evaluation of time series forecasting models is hindered by a lack of high-quality benchmarks, leading to overestimated assessments of progress.

GTIG AI Threat Tracker: Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access

model-releasesgoogle-cloud-ai

Executive Summary Since our February 2026 report on AI-related threat activity, Google Threat Intelligence Group (GTIG) has continued to track a maturing transition from nascent AI-enabled operations

InterLV-Search: Benchmarking Interleaved Multimodal Agentic Search

arXiv:2605.07510v1 Announce Type: cross Abstract: Existing benchmarks for multimodal agentic search evaluate multimodal search and visual browsing, but visual evidence is either confined to the input

Stabilized neural Hamilton--Jacobi--Bellman solvers: Error analysis and applications in model-based reinforcement learning

arXiv:2605.07116v1 Announce Type: cross Abstract: Physics-informed neural solvers offer a promising route to model-based reinforcement learning in continuous time, where optimal feedback synthesis is

safetyarxiv-cs-ai

Conditional outlier detection for clinical alerting

arXiv:2605.05124v1 Announce Type: new Abstract: We develop and evaluate a data-driven approach for detecting unusual (anomalous) patient-management actions using past patient cases stored in an electr

researcharxiv-cs-lg

7 May 2026

DIAL: Direct Iterative Adversarial Learning for Realistic Multi-Turn Dialogue Simulation

arXiv:2512.20773v4 Announce Type: replace Abstract: Realistic user simulation is crucial for training and evaluating multi-turn dialogue systems, yet creating simulators that accurately replicate huma

researcharxiv-cs-cl

7 May 2026

DataClaw: A Process-Oriented Agent Benchmark for Exploratory Real-World Data Analysis

arXiv:2605.02503v1 Announce Type: new Abstract: Evaluating autonomous data analysis agents requires testing their ability to perform exploratory analysis in underexplored data environments. However, m

Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models

model-releasesarxiv-cs-cv

arXiv:2605.03547v1 Announce Type: new Abstract: Large Vision-Language Models (LVLMs), trained on web-scale data, risk memorizing and regenerating copyrighted visual content such as characters and logo

From Laboratory to Real-World Applications: Benchmarking Agentic Code Reasoning at the Repository Level

arXiv:2601.03731v3 Announce Type: replace-cross Abstract: As large language models (LLMs) evolve into autonomous agents, evaluating repository-level reasoning, the ability to maintain logical consiste

Tutorials

SCPRM: A Schema-aware Cumulative Process Reward Model for Knowledge Graph Question Answering

arXiv:2605.02819v1 Announce Type: new Abstract: Large language models excel at complex reasoning, yet evaluating their intermediate steps remains challenging. Although process reward models provide st

tutorialsarxiv-cs-ai

Seeking Information with RAG-Assistants: Does Model Size Matter in Human-AI Collaborations?

arXiv:2605.00964v1 Announce Type: cross Abstract: Much research on LLMs has focused on increasing benchmark performance. However, the evaluation of such models in real-world collaborative human-AI wor

WorldJen: An End-to-End Multi-Dimensional Benchmark for Generative Video Models

model-releasesarxiv-cs-cv

arXiv:2605.03475v1 Announce Type: new Abstract: Evaluating generative video models remains an open problem. Reference-based metrics such as Structural Similarity Index Measure (SSIM) and Peak Signal t

A multilingual hallucination benchmark: MultiWikiQHalluA

arXiv:2605.02504v1 Announce Type: new Abstract: Most hallucination evaluations focus on English, leaving it unclear whether findings transfer to lower-resource languages. We investigate faithfulness h

AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Models

model-releasesarxiv-cs-cv

arXiv:2506.09082v5 Announce Type: replace Abstract: The rise of vision foundation models (VFMs) calls for systematic evaluation. A common approach pairs VFMs with large language models (LLMs) as gener

Applications

Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting

arXiv:2605.02752v1 Announce Type: new Abstract: Open-world text-guided class-agnostic counting (CAC) has emerged as a flexible paradigm for counting arbitrary object classes by using natural language

applicationsarxiv-cs-cv

DurableUn: Quantization-Induced Recovery Attacks in Machine Unlearning

arXiv:2605.02196v1 Announce Type: new Abstract: Machine unlearning aims to remove specified training data to satisfy privacy regulations such as GDPR. However, existing evaluations assume identical pr

How Well Can We Decode Vowels from Auditory EEG -- A Rigorous Cross-Subject Benchmark with Honest Assessment

arXiv:2605.00865v1 Announce Type: cross Abstract: EEG based phoneme decoding is promising for brain computer interfaces, but many prior studies rely on within subject evaluation, small cohorts, or wea

Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks

arXiv:2605.01417v1 Announce Type: new Abstract: Evaluating large language models (LLMs) for medical applications remains challenging due to benchmark saturation, limited data accessibility, and insuff

PACE: Parameter Change for Unsupervised Environment Design

arXiv:2605.01358v1 Announce Type: new Abstract: Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training env

Agents

Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

arXiv:2603.03565v2 Announce Type: replace-cross Abstract: Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals tw

agentsarxiv-cs-cl

4 May 2026

Estimating LLM Grading Ability and Response Difficulty in Automatic Short Answer Grading via Item Response Theory

arXiv:2605.00238v1 Announce Type: new Abstract: Automated short answer grading (ASAG) with large language models (LLMs) is commonly evaluated with aggregate metrics such as macro-F1 and Cohen's kappa.

safetyarxiv-cs-cl

4 May 2026

The Algorithmic Gaze of Image Quality Assessment: An Audit and Trace Ethnography of the LAION-Aesthetics Predictor

arXiv:2601.09896v4 Announce Type: replace-cross Abstract: Visual generative AI models are trained using a one-size-fits-all measure of aesthetic appeal. However, what is deemed 'aesthetic' is inextric

researcharxiv-cs-cv

4 May 2026

AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR

arXiv:2604.27543v1 Announce Type: new Abstract: Evaluating English ASR systems for conversational AI applications remains difficult, as many publicly available corpora are either pre-segmented into sh

Do What I Say: A Spoken Prompt Dataset for Instruction-Following

arXiv:2603.09881v2 Announce Type: replace Abstract: Speech Large Language Models (SLLMs) have rapidly expanded, supporting a wide range of tasks. These models are typically evaluated using text prompt

Rethinking Pulmonary Embolism Segmentation: A Study of Current Approaches and Challenges with an Open Weight Model

arXiv:2509.18308v3 Announce Type: replace Abstract: Pulmonary Embolism (PE) is a life-threatening condition for which accurate and timely detection is critical to patient care. However, our systematic

researcharxiv-cs-cv

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

arXiv:2601.08611v2 Announce Type: replace-cross Abstract: The growing scale of online misinformation urgently demands Automated Fact-Checking (AFC). Existing benchmarks for evaluating AFC systems, how

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

arXiv:2603.06198v2 Announce Type: replace Abstract: Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving docum

30 Apr 2026

Stress Testing Factual Consistency Metrics for Long-Document Summarization

arXiv:2511.07689v2 Announce Type: replace-cross Abstract: Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where c

30 Apr 2026

Tools

AI evals are becoming the new compute bottleneck

As AI models grow larger and more capable, the computational cost and time required to evaluate them has become a significant limiting factor in development, potentially surpassing training compute as

toolshugging-face

29 Apr 2026

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

arXiv:2604.25231v1 Announce Type: cross Abstract: Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics