Search: “evaluation”

arXiv:2601.04387v2 Announce Type: replace Abstract: Negotiation is a core component of social intelligence, requiring agents to balance strategic reasoning, cooperation, and social norms. Recent work

agentsarxiv-cs-ai

7 Jul 2026

Applications

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation

DGX agent

arXiv:2607.03487v1 Announce Type: cross Abstract: Mutual information (MI) estimation is a central problem in machine learning and statistics; however, existing benchmarks typically evaluate estimators

applicationsarxiv-cs-ai

7 Jul 2026

TRACER: Early Failure Detection for Task-Oriented Dialogue

arXiv:2607.03974v1 Announce Type: new Abstract: Task-oriented dialogue systems often fail before the final breakdown is obvious, but most evaluation only measures failure after the conversation has al

researcharxiv-cs-cl

7 Jul 2026

TrendFact: A Benchmark Towards Hotspot Perception in Automatic Fact-Checking

arXiv:2410.15135v5 Announce Type: replace Abstract: With the surge of online misinformation, Large Language Models (LLMs) and Reasoning Large Language Models (RLMs) serving as Automatic Fact-Checking

7 Jul 2026

AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations

arXiv:2607.01934v1 Announce Type: cross Abstract: This work introduces AIriskEval-edu-db2, a new dataset designed to train and evaluate auditors based on LLMs for an explainable pedagogical risk asses

Benchmarking Federated Learning and Knowledge Distillation for Point Cloud Classification

arXiv:2607.01272v1 Announce Type: cross Abstract: Deploying 3D point cloud analysis in privacy-sensitive, resource-constrained settings faces two barriers: data cannot be centralized, and models must

MedRepBench: A Comprehensive Benchmark for Medical Report Interpretation

arXiv:2508.16674v2 Announce Type: replace-cross Abstract: Medical report understanding from real-world document images is essential for generating patient-facing explanations and enabling structured i

Office Comprehension Benchmark

arXiv:2607.01245v1 Announce Type: cross Abstract: We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension

SPLIT: Cross-Lingual Empathy and Cultural Grounding in English and Ukrainian LLM Responses

arXiv:2607.02049v1 Announce Type: cross Abstract: Large Language Models are increasingly deployed in emotional-support contexts and crisis-related situations. Nevertheless, their cross-lingual abiliti

The Dual Nature of LLM Persona: Aggregated Tendencies and Frame-Dependent Geometry

arXiv:2607.02368v1 Announce Type: cross Abstract: Evaluations of LLM personas via psychometric questionnaires typically rely on aggregate scores, discarding within-instance correlation structure. We t

Aionoscope: Debugging Latent-State Accessibility in Time-Series Representations

arXiv:2607.00956v1 Announce Type: cross Abstract: Time-series models are often evaluated by what they can forecast or classify, but those scores do not show whether their representations preserve the

Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents

arXiv:2607.00895v1 Announce Type: new Abstract: Hallucination detection for retrieval-augmented generation (RAG) is usually evaluated on natural-language document evidence. However, grounded generatio

Tutorials

DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation

arXiv:2607.00183v1 Announce Type: new Abstract: Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended

tutorialsarxiv-cs-cv

EgoGapBench: Benchmarking Egocentric Action Selection in Multi-Agent Scenes

arXiv:2607.00547v1 Announce Type: cross Abstract: Existing egocentric benchmarks have primarily constructed the egocentric setting from first-person-view data, which makes it difficult to evaluate ego

Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning

arXiv:2607.00452v1 Announce Type: cross Abstract: Actor-critic methods depend on learned critics, but critic quality is often evaluated only indirectly through return, temporal-difference error, or va

Measuring the Gap Between Human and LLM Research Ideas

arXiv:2607.01233v1 Announce Type: cross Abstract: LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert pre

Path Planning in Physically Viable World Models

arXiv:2607.00673v1 Announce Type: new Abstract: Robots deployed in unstructured outdoor environments often plan from scene reconstructions collected before deployment because operators cannot remap la

researcharxiv-cs-ro

Self-Evolving Agents with Anytime-Valid Certificates

arXiv:2607.00871v1 Announce Type: new Abstract: Self-evolving agents violate the assumption behind most learning-theoretic guarantees: the data, evaluator, components, and hypothesis space are produce

An Executable Benchmarking Suite for Tool-Using Agents

arXiv:2605.11030v2 Announce Type: replace-cross Abstract: Closed-loop tool-using agents are increasingly evaluated in executable web, code, and micro-task environments, but benchmark reports often con

BLUEX v2: Benchmarking LLMs on Open-Ended Questions from Brazilian University Entrance Exams

arXiv:2606.22723v2 Announce Type: replace Abstract: Although Large Language Models (LLMs) excel in many tasks, their assessment in Portuguese has received less attention, particularly for open-ended,

HealthAgentBench: A Unified Benchmark Suite of Realistic Agentic Healthcare Environments for Challenging Frontier AI Agents

arXiv:2606.31179v1 Announce Type: new Abstract: As AI agents become increasingly capable of complex, long-horizon reasoning, rigorous and holistic evaluation is essential for measuring progress toward

No Place to Hide: Benchmarking Video Hallucination with Background-Controlled Pairs

arXiv:2606.31933v1 Announce Type: new Abstract: We introduce VidPair-Halluc, a new benchmark for evaluating video hallucination in large video models (LVMs) under rigorous and controlled conditions. U

Position: Vision-Language-Action Models Cannot Be Verified to Perform Physical Reasoning

arXiv:2606.30686v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) systems, built on pretrained vision-language models (VLMs), have shown rapidly improving performance on robot manipulatio

The Bidirectional Process Reward Model

arXiv:2508.01682v3 Announce Type: replace Abstract: Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promi

Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action

arXiv:2606.31916v1 Announce Type: new Abstract: Theory of Mind (ToM) benchmarks for Large Language Models (LLMs) typically rely on passive question-answering formats, but the deployment of LLMs in inc

Xiaomi-GUI-0 Technical Report

arXiv:2606.31410v1 Announce Type: new Abstract: Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions s

Bridging the Gap Between Image Restoration and Navigational Safety in Hazy Conditions: A New Visibility Estimation Metric for Maritime Surveillance

arXiv:2606.30049v1 Announce Type: new Abstract: Visibility distance is critical to maritime navigational safety because it determines the effective observation range of shipborne and shore-based monit

How Far Can Sharpness and Complexity Jointly Explain Generalization?

model-releasesarxiv-cs-lg

arXiv:2606.29043v1 Announce Type: new Abstract: Sharpness and complexity are two central factors in the generalization analysis of deep neural networks. Existing quantitative evaluations of generaliza

MAM-AI: An On-Device Medical Retrieval-Augmented Generation System for Nurses and Midwives in Zanzibar

arXiv:2606.29580v1 Announce Type: new Abstract: Maternal and newborn mortality remain among the highest in sub-Saharan Africa, where midwifery care is often delivered by nurses who lack midwifery trai

Open Problems in Constitutional Preference Reconstruction

arXiv:2606.30116v1 Announce Type: new Abstract: Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a choice, not the rationale

Sequential Fairness Auditing with Limited Output Access

arXiv:2606.30338v1 Announce Type: new Abstract: External evaluations are becoming increasingly central to the governance of AI systems. In practice, however, independent auditors often have limited ac

SFBench: The SciFy Scientific Feasibility Benchmark

arXiv:2606.29630v1 Announce Type: new Abstract: We present SFBench, a benchmark dataset for evaluating systems that assess the feasibility of scientific claims. SFBench includes 197 claims in material

Applications

SIMAX: A Scalable and Interpretable Framework for Multi-Fidelity and Annotated Clinician-Patient Dialogue Simulation

arXiv:2606.30491v1 Announce Type: cross Abstract: Background. The widespread deployment of ambient digital scribes is driving large-scale capture of clinician-patient dialogues. Human coding of clinic

applicationsarxiv-cs-ai

SrDetection: A Self-Referential Framework for Data Leakage Detection in Code Large Language Models

arXiv:2606.29815v1 Announce Type: new Abstract: Evaluating code large language models (Code LLMs) requires reliable detection of data leakage, where benchmark performance is artificially inflated by e

The Joint Effect of Quantization and Sampling Temperature on LLM Safety Alignment: A Factorial Analysis

arXiv:2606.29581v1 Announce Type: cross Abstract: Modern LLM deployments routinely compress models and raise sampling temperature to reduce cost, latency, or repetition, yet safety evaluations usually

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding

arXiv:2601.04693v2 Announce Type: replace Abstract: Although negation is known to challenge large language models (LLMs), benchmarks for evaluating negation understanding-especially in Korean-are scar

VTEdit-Bench: A Comprehensive Benchmark for Multi-Reference Image Editing Models in Virtual Try-On

arXiv:2603.11734v2 Announce Type: replace Abstract: As virtual try-on (VTON) continues to advance, a growing number of real-world scenarios have emerged, pushing beyond the ability of the existing spe

AirGroundBench: Probing Spatial Intelligence in Multimodal Large Models under Heterogeneous Multi-View Embodied Collaboration

arXiv:2606.28049v1 Announce Type: new Abstract: In recent years, multimodal large language models (MLLMs) have shown strong potential for embodied intelligence, yet their ability to maintain geometric

Differentiable design of the PIAA-ZWFS: a flexible wavefront sensor that approaches the fundamental limit

arXiv:2606.28136v1 Announce Type: cross Abstract: Extreme adaptive optics (AO) is necessary for high contrast astronomy at scales of the habitable zone of nearby systems. We seek to evaluate wavefront

researcharxiv-cs-cv

Estimation--Prediction Tradeoff in Causal Probabilistic Temporal Graphs

model-releasesarxiv-cs-lg

arXiv:2606.28225v1 Announce Type: new Abstract: Temporal link prediction is usually evaluated by predictive performance on unseen edges, but in probabilistic temporal graphs this criterion can conflat

Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs

arXiv:2606.27378v1 Announce Type: new Abstract: We introduce an axiomatic evaluation framework for latent thought representations in LLMs, comprising metrics that are independent of downstream benchma

Your AI Travel Agent Would Book You a Bullfight: An Agentic Benchmark for Implicit Animal Welfare in Frontier AI Models

arXiv:2606.18142v3 Announce Type: replace Abstract: AI agents are moving from advisors to actors, booking travel, planning menus, and running procurement on behalf of users. Existing benchmarks for AI

Applications

A Pipeline for Generating Longitudinal Synthetic Clinical Notes Using Large Language Models

arXiv:2606.26879v1 Announce Type: new Abstract: Synthetic data is increasingly used to enable the development and evaluation of AI systems in domains where access to real-world data is restricted. In

applicationsarxiv-cs-ai

Assessing Post-Reform Changes in Risk Disclosure Quality with a Multidimensional Text Analysis Approach

arXiv:2606.26522v1 Announce Type: new Abstract: While corporate narrative disclosures provide crucial information to capital markets, comprehensively evaluating their qualitative changes over time rem

safetyarxiv-cs-cl

ConflictScore: Identifying and Measuring How Language Models Handle Conflicting Evidence

arXiv:2606.26437v1 Announce Type: cross Abstract: Existing metrics for factuality and faithfulness evaluate whether an answer is supported or contradicted by its grounding documents, but they fail to

Do Image Editing Models Understand Lighting?

arXiv:2606.26738v1 Announce Type: new Abstract: While recent advancements in generative image editing models have achieved stunning visual fidelity, it remains an open question whether these systems p

Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform

model-releasesarxiv-cs-lg

arXiv:2606.26590v1 Announce Type: new Abstract: Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used

From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

arXiv:2606.26535v1 Announce Type: cross Abstract: Current VLM evaluations often conflate language priors with genuine spatial reasoning. To address this, we introduce CRISP, a novel structural-diagnos