Search: “evaluation”

Stress Testing Factual Consistency Metrics for Long-Document Summarization

arXiv:2511.07689v2 Announce Type: replace-cross Abstract: Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where c

30 Apr 2026

Tools

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

AI evals are becoming the new compute bottleneck

DGX agent

As AI models grow larger and more capable, the computational cost and time required to evaluate them has become a significant limiting factor in development, potentially surpassing training compute as

toolshugging-face

29 Apr 2026

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

arXiv:2604.25231v1 Announce Type: cross Abstract: Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics

Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?

arXiv:2507.15707v2 Announce Type: replace Abstract: Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study

researcharxiv-cs-cl

M^3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

model-releasesarxiv-cs-cv

arXiv:2604.25122v1 Announce Type: new Abstract: We present M^3-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (ML

Measuring the stability and plasticity of recommender systems

arXiv:2508.03941v3 Announce Type: replace-cross Abstract: The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of th

Monitoring exposure-length variations in submarine power cables using distributed fiber-optic sensing

arXiv:2604.24880v1 Announce Type: cross Abstract: This study proposes an anomaly-detection framework for monitoring exposure-length variations in submarine free-span cables using Distributed Acoustic

AI Safety Training Can be Clinically Harmful

arXiv:2604.23445v1 Announce Type: cross Abstract: Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigo

An Information-Geometric Framework for Stability Analysis of Large Language Models under Entropic Stress

arXiv:2604.24076v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed in high-stakes and operational settings, evaluation strategies based solely on aggregate accur

Agents

Dr. RTL: Autonomous Agentic RTL Optimization through Tool-Grounded Self-Improvement

arXiv:2604.14989v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have sparked growing interest in automatic RTL optimization for better performance, power, and area

agentsarxiv-cs-ai

EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs

arXiv:2604.23348v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and generation, and are increasingly used in

Robust and Clinically Reliable EEG Biomarkers: A Cross Population Framework for Generalizable Parkinson's Disease Detection

arXiv:2604.23933v1 Announce Type: new Abstract: Developing robust and clinically reliable EEG biomarkers requires evaluation frameworks that explicitly address cross population generalization in multi

Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers

arXiv:2601.07698v2 Announce Type: replace Abstract: As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged:

TSAssistant: A Human-in-the-Loop Agentic Framework for Automated Target Safety Assessment

arXiv:2604.23938v1 Announce Type: new Abstract: Target Safety Assessment (TSA) requires systematic integration of heterogeneous evidence, including genetic, transcriptomic, target homology, pharmacolo

safetyarxiv-cs-cl

Value Alignment Tax: Measuring Value Trade-offs in LLM Alignment

arXiv:2602.12134v2 Announce Type: replace Abstract: Existing work on value alignment typically characterizes value relations statically, ignoring how alignment interventions, such as prompting, fine-t

Applications

When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Models

arXiv:2510.00626v3 Announce Type: replace-cross Abstract: Large audio-language models (LALMs) unify speech and text processing, but their robustness in noisy real-world settings remains underexplored.

applicationsarxiv-cs-cl

BLAST: Benchmarking LLMs with ASP-based Structured Testing

arXiv:2604.22306v1 Announce Type: cross Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dial

researcharxiv-cs-ai

Detecting Cognitive Signatures in Typing Behavior for Non-Intrusive Authorship Verification

arXiv:2603.00177v2 Announce Type: replace-cross Abstract: The proliferation of AI-generated text has intensified the need for reliable authorship verification, yet current output-based methods are inc

On Benchmark Hacking in ML Contests: Modeling, Insights and Design

model-releasesarxiv-cs-lg

arXiv:2604.22230v1 Announce Type: cross Abstract: Benchmark hacking refers to tuning a machine learning model to score highly on certain evaluation criteria without improving true generalization or fa

PoLO: Proof-of-Learning and Proof-of-Ownership at Once with Chained Watermarking

arXiv:2505.12296v2 Announce Type: replace-cross Abstract: Our evaluation shows that PoLO achieves extbf{99%} watermark detection accuracy for ownership verification, while preserving data privacy and

researcharxiv-cs-ai

TS-Arena -- A Live Forecast Pre-Registration Platform

arXiv:2512.20761v3 Announce Type: replace-cross Abstract: Time Series Foundation Models (TSFMs) are transforming the field of forecasting. However, evaluating them on historical data is increasingly d

Empirical Comparison of Agent Communication Protocols for Task Orchestration

arXiv:2603.22823v3 Announce Type: replace Abstract: Context. The problem of comparative evaluation of communication protocols for task orchestration by large language model (LLM) agents is considered.

Finding Meaning in Embeddings: Concept Separation Curves

arXiv:2604.21555v1 Announce Type: new Abstract: Sentence embedding techniques aim to encode key concepts of a sentence's meaning in a vector space. However, the majority of evaluation approaches for s

researcharxiv-cs-cl

Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization

arXiv:2603.28342v2 Announce Type: replace Abstract: We present Kernel-Smith, a framework for high-performance GPU kernel and operator generation that combines a stable evaluation-driven evolutionary a

Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms

arXiv:2604.21882v1 Announce Type: new Abstract: Understanding what kinds of factual knowledge large language models (LLMs) memorize is essential for evaluating their reliability and limitations. Entit

researcharxiv-cs-cl

Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry

arXiv:2604.20983v1 Announce Type: cross Abstract: Vision evaluations are typically done through multi-step processes. In most contemporary fields, experts analyze images using structured, evidence-bas

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text

arXiv:2604.20051v1 Announce Type: new Abstract: Self-play has recently emerged as a promising paradigm to train Large Language Models (LLMs). In self-play, the target LLM creates the task input (e.g.,

Day 2 at Google Cloud Next: A marathon developer keynote

model-releasesgoogle-cloud-ai

At Google Cloud, every day is Developer Day, but none so much as day 2 of Google Cloud Next, when we hold the developer keynote. This year’s topic? An in-depth look at Gemini Enterprise Agent Platform

Explainable Speech Emotion Recognition: Weighted Attribute Fairness to Model Demographic Contributions to Social Bias

arXiv:2604.19763v1 Announce Type: cross Abstract: Speech Emotion Recognition (SER) systems have growing applications in sensitive domains such as mental health and education, where biased predictions

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

arXiv:2604.20006v1 Announce Type: new Abstract: Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. Ho

Large language models perceive cities through a culturally uneven baseline

arXiv:2604.20048v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to describe, evaluate and interpret places, yet it remains unclear whether they do so from a cultural

safetyarxiv-cs-cl

Local Ai

LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoning

arXiv:2603.06870v2 Announce Type: replace Abstract: Long-horizon execution in Large Language Models (LLMs) remains unstable even when high-level strategies are provided. Evaluating on controlled algor

local-aiarxiv-cs-ai

MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models

arXiv:2604.19809v1 Announce Type: new Abstract: We introduce MIRROR, a benchmark comprising eight experiments across four metacognitive levels that evaluates whether large language models can use self

Self-Awareness before Action: Mitigating Logical Inertia via Proactive Cognitive Awareness

arXiv:2604.20413v1 Announce Type: new Abstract: Large language models perform well on many reasoning tasks, yet they often lack awareness of whether their current knowledge or reasoning state is compl

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

arXiv:2604.20087v1 Announce Type: new Abstract: Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how

X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis

model-releasesarxiv-cs-cv

arXiv:2604.20350v1 Announce Type: new Abstract: Despite significant progress in Multi-modal Large Language Models (MLLMs), their clinical reasoning capacity for multi-modal diagnosis remains largely u

Analytical Extraction of Conditional Sobol' Indices via Basis Decomposition of Polynomial Chaos Expansions

model-releasesarxiv-cs-lg

arXiv:2604.19165v1 Announce Type: cross Abstract: In uncertainty quantification, evaluating sensitivity measures under specific conditions (i.e., conditional Sobol' indices) is essential for systems w

Benchmarking Misuse Mitigation Against Covert Adversaries

arXiv:2506.06414v2 Announce Type: replace-cross Abstract: Existing language model safety evaluations focus on overt attacks and low-stakes tasks. In reality, an attacker can easily subvert existing sa

Compile to Compress: Boosting Formal Theorem Provers by Compiler Outputs

arXiv:2604.18587v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated significant potential in formal theorem proving, yet state-of-the-art performance often necessitates pr

Introducing Gemini Enterprise Agent Platform, powering the next wave of agents

model-releasesgoogle-cloud-ai

In the early days of generative AI, building safe and reliable business tools took massive engineering effort and a high tolerance for trial and error. We helped solve that with Vertex AI, our trusted

Rank-Turbulence Delta and Interpretable Approaches to Stylometric Delta Metrics

arXiv:2604.19499v1 Announce Type: new Abstract: This article introduces two new measures for authorship attribution - Rank-Turbulence Delta and Jensen-Shannon Delta - which generalise Burrows's classi

Time Series Augmented Generation for Financial Applications

arXiv:2604.19633v1 Announce Type: new Abstract: Evaluating the reasoning capabilities of Large Language Models (LLMs) for complex, quantitative financial tasks is a critical and unsolved challenge. St

Towards Optimal Agentic Architectures for Offensive Security Tasks

arXiv:2604.18718v1 Announce Type: cross Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear w

CGCMA: Conditionally-Gated Cross-Modal Attention for Event-Conditioned Asynchronous Fusion

arXiv:2604.16411v1 Announce Type: new Abstract: We study asynchronous alignment, a first-class multimodal learning setting in which a dense primary stream must be fused with sporadic external context

safetyarxiv-cs-lg

Chain Of Interaction Benchmark (COIN): When Reasoning meets Embodied Interaction

model-releasesarxiv-cs-ro

arXiv:2604.16886v1 Announce Type: new Abstract: Generalist embodied agents must perform interactive, causally-dependent reasoning, continually interacting with the environment, acquiring information,

EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions

arXiv:2604.16456v1 Announce Type: new Abstract: Real-time voice assistants must revise task state when users interrupt mid-response, but existing spoken-dialog benchmarks largely evaluate turn-based i

From User Recognition to Activity Counting: An Identity-Agnostic Approach to Multi-User WiFi Sensing

arXiv:2604.16572v1 Announce Type: new Abstract: Wi-Fi Channel State Information (CSI) enables device-free human activity recognition, but existing multi-user approaches assume a fixed set of known use

Heterogeneous Self-Play for Realistic Highway Traffic Simulation

arXiv:2604.16406v1 Announce Type: cross Abstract: Realistic highway simulation is critical for scalable safety evaluation of autonomous vehicles, particularly for interactions that are too rare to stu

safetyarxiv-cs-lg