Search: “evaluation”

TRACER: Early Failure Detection for Task-Oriented Dialogue

arXiv:2607.03974v1 Announce Type: new Abstract: Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has al

researcharxiv-cs-cl

7 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking

DGX agent

arXiv:2410.15135v5 Announce Type: replace Abstract: With the surge of online misinformation, Large Language Models (LLMs) and Reasoning Large Language Models (RLMs) serving as Automatic Fact-Checking

model-releasesarxiv-cs-cl

7 Jul 2026

AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations

arXiv:2607.01934v1 Announce Type: cross Abstract: This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk asses

Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

arXiv:2607.01272v1 Announce Type: cross Abstract: Deploying 3D point cloud analysis in privacy-sensitive, resource-constrained settings faces two barriers: data cannot be centralized, and models must

MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation

arXiv:2508.16674v2 Announce Type: replace-cross Abstract: Medical report understanding from real-world document images is essential for generating patient-facing explanations and enabling structured i

Office Comprehension Benchmark

arXiv:2607.01245v1 Announce Type: cross Abstract: We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension

SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

arXiv:2607.02049v1 Announce Type: cross Abstract: Large Language Models are increasingly deployed in emotional-support contexts and crisis-related situations. Nevertheless, their cross-lingual abiliti

The Dual Nature of LLM Persona: Aggregated Tendencies and Frame-Dependent Geometry

arXiv:2607.02368v1 Announce Type: cross Abstract: Evaluations of LLM personas via psychometric questionnaires typically rely on aggregate scores, discarding within-instance correlation structure. We t

Aionoscope: Debugging Latent-State Accessibility in Time-Series Representations

arXiv:2607.00956v1 Announce Type: cross Abstract: Time-series models are often evaluated by what they can forecast or classify, but those scores do not show whether their representations preserve the

Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

arXiv:2607.00895v1 Announce Type: new Abstract: Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generatio

Tutorials

DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation

arXiv:2607.00183v1 Announce Type: new Abstract: Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended

tutorialsarxiv-cs-cv

EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

arXiv:2607.00547v1 Announce Type: cross Abstract: Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate ego

Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning

arXiv:2607.00452v1 Announce Type: cross Abstract: Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or va

Measuring the Gap Between Human and LLM Research Ideas

arXiv:2607.01233v1 Announce Type: cross Abstract: LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert pre

Path Planning in Physically Viable World Models

arXiv:2607.00673v1 Announce Type: new Abstract: Robots deployed in unstructured outdoor environments often plan from scene reconstructions collected before deployment because operators cannot remap la

researcharxiv-cs-ro

Self-Evolving Agents with Anytime-Valid Certificates

arXiv:2607.00871v1 Announce Type: new Abstract: Self-evolving agents violate the assumption behind most learning-theoretic guarantees: the data, evaluator, components, and hypothesis space are produce

Hardware

9/ ParallelKernelBench: Benchmarking LLMs on Multi-GPU Kernel Generation Paper: https://www.alphaxiv.org/abs/2606.parallel-kernel-bench

ParallelKernelBench is a benchmarking framework designed to evaluate large language models' ability to generate optimized GPU kernels for multi-GPU computing environments. The benchmark assesses LLMs

hardwaretogether-ai--x

An Executable Benchmarking Suite for Tool-Using Agents

arXiv:2605.11030v2 Announce Type: replace-cross Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often con

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

arXiv:2606.22723v2 Announce Type: replace Abstract: Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended,

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

arXiv:2606.31179v1 Announce Type: new Abstract: As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward

No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs

arXiv:2606.31933v1 Announce Type: new Abstract: We introduce VidPair-Halluc, a new benchmark for evaluating video hallucination in large video models (LVMs) under rigorous and controlled conditions. U

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

arXiv:2606.30686v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulatio

The Bidirectional Process Reward Model

arXiv:2508.01682v3 Announce Type: replace Abstract: Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promi

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action

arXiv:2606.31916v1 Announce Type: new Abstract: Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in inc

Who did it best? GLM-5.2 (left) | Fugu Ultra (middle) | Fable 5 (right) Same one-shot prompt. The last one is my favorite!

This post compares the outputs of three AI models—GLM-5.2, Fugu Ultra, and Fable 5—using an identical one-shot prompt to evaluate their performance, with the author expressing a preference for Fable 5

researchdair-ai--x

Xiaomi-GUI-0 Technical Report

arXiv:2606.31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions s

Bridging the Gap Between Image Restoration and Navigational Safety in Hazy Conditions: A New Visibility Estimation Metric for Maritime Surveillance

arXiv:2606.30049v1 Announce Type: new Abstract: Visibility distance is critical to maritime navigational safety because it determines the effective observation range of shipborne and shore-based monit

How Far Can Sharpness and Complexity Jointly Explain Generalization?

model-releasesarxiv-cs-lg

arXiv:2606.29043v1 Announce Type: new Abstract: Sharpness and complexity are two central factors in the generalization analysis of deep neural networks. Existing quantitative evaluations of generaliza

MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar

arXiv:2606.29580v1 Announce Type: new Abstract: Maternal and newborn mortality remain among the highest in sub-Saharan Africa, where midwifery care is often delivered by nurses who lack midwifery trai

Open Problems in Constitutional Preference Reconstruction

arXiv:2606.30116v1 Announce Type: new Abstract: Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a choice, not the rationale

Applications

ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

ScarfBench is a benchmarking framework designed to evaluate AI agents' capabilities in migrating enterprise Java applications to modern frameworks. The benchmark likely assesses how well AI systems ca

applicationshugging-face

Sequential Fairness Auditing with Limited Output Access

arXiv:2606.30338v1 Announce Type: new Abstract: External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited ac

SFBench: The SciFy Scientific Feasibility Benchmark

arXiv:2606.29630v1 Announce Type: new Abstract: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in material

Applications

SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation

arXiv:2606.30491v1 Announce Type: cross Abstract: Background. The widespread deployment of ambient digital scribes is driving large-scale capture of clinician-patient dialogues. Human coding of clinic

applicationsarxiv-cs-ai

SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models

arXiv:2606.29815v1 Announce Type: new Abstract: Evaluating code large language models (Code LLMs) requires reliable detection of data leakage, where benchmark performance is artificially inflated by e

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

arXiv:2606.29581v1 Announce Type: cross Abstract: Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

arXiv:2601.04693v2 Announce Type: replace Abstract: Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scar

VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On

arXiv:2603.11734v2 Announce Type: replace Abstract: As virtual try-on (VTON) continues to advance, a growing number of real-world scenarios have emerged, pushing beyond the ability of the existing spe

AirGroundBench: Probing Spatial Intelligence in Multimodal Large Models under Heterogeneous Multi-View Embodied Collaboration

arXiv:2606.28049v1 Announce Type: new Abstract: In recent years, multimodal large language models (MLLMs) have shown strong potential for embodied intelligence, yet their ability to maintain geometric

Differentiable design of the PIAA-ZWFS: a flexible wavefront sensor that approaches the fundamental limit

arXiv:2606.28136v1 Announce Type: cross Abstract: Extreme adaptive optics (AO) is necessary for high contrast astronomy at scales of the habitable zone of nearby systems. We seek to evaluate wavefront

researcharxiv-cs-cv

Estimation--Prediction Tradeoff in Causal Probabilistic Temporal Graphs

model-releasesarxiv-cs-lg

arXiv:2606.28225v1 Announce Type: new Abstract: Temporal link prediction is usually evaluated by predictive performance on unseen edges, but in probabilistic temporal graphs this criterion can conflat

Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs

arXiv:2606.27378v1 Announce Type: new Abstract: We introduce an axiomatic evaluation framework for latent thought representations in LLMs, comprising metrics that are independent of downstream benchma

Your AI Travel Agent Would Book You a Bullfight: An Agentic Benchmark for Implicit Animal Welfare in Frontier AI Models

arXiv:2606.18142v3 Announce Type: replace Abstract: AI agents are moving from advisors to actors, booking travel, planning menus, and running procurement on behalf of users. Existing benchmarks for AI

Applications

A Pipeline for Generating Longitudinal Synthetic Clinical Notes Using Large Language Models

arXiv:2606.26879v1 Announce Type: new Abstract: Synthetic data is increasingly used to enable the development and evaluation of AI systems in domains where access to real-world data is restricted. In

applicationsarxiv-cs-ai

Tutorials

AI code review benchmarks can seem a little mysterious. So @filip_hric breaks it down for us. Here's how to understand precision, recall, an…

tutorialsitamar-friedman--x

This post by Filip Hric explains how to interpret AI code review benchmarks, specifically breaking down key evaluation metrics like precision and recall that are commonly used to assess code review to

Assessing Post-Reform Changes in Risk Disclosure Quality with a Multidimensional Text Analysis Approach

arXiv:2606.26522v1 Announce Type: new Abstract: While corporate narrative disclosures provide crucial information to capital markets, comprehensively evaluating their qualitative changes over time rem

safetyarxiv-cs-cl

ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

arXiv:2606.26437v1 Announce Type: cross Abstract: Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to

Do Image Editing Models Understand Lighting?

arXiv:2606.26738v1 Announce Type: new Abstract: While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems p