Search: “evaluation”

Stress Testing Factual Consistency Metrics for Long-Document Summarization

arXiv:2511.07689v2 Announce Type: replace-cross Abstract: Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where c

30 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams

DGX agent

arXiv:2604.25231v1 Announce Type: cross Abstract: Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics

model-releasesarxiv-cs-cl

29 Apr 2026

Is Large Language Model Performance on Reasoning Tasks Impacted by Different Ways Questions Are Asked?

arXiv:2507.15707v2 Announce Type: replace Abstract: Large Language Models (LLMs) have been evaluated using diverse question types, e.g., multiple-choice, true/false, and short/long answers. This study

researcharxiv-cs-cl

M^3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

model-releasesarxiv-cs-cv

arXiv:2604.25122v1 Announce Type: new Abstract: We present M^3-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (ML

Measuring the stability and plasticity of recommender systems

arXiv:2508.03941v3 Announce Type: replace-cross Abstract: The typical offline protocol to evaluate recommendation algorithms is to collect a dataset of user-item interactions and then use a part of th

Monitoring exposure-length variations in submarine power cables using distributed fiber-optic sensing

arXiv:2604.24880v1 Announce Type: cross Abstract: This study proposes an anomaly-detection framework for monitoring exposure-length variations in submarine free-span cables using Distributed Acoustic

AI Safety Training Can be Clinically Harmful

arXiv:2604.23445v1 Announce Type: cross Abstract: Large language models are being deployed as mental health support agents at scale, yet only 16% of LLM-based chatbot interventions have undergone rigo

An Information-Geometric Framework for Stability Analysis of Large Language Models under Entropic Stress

arXiv:2604.24076v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed in high-stakes and operational settings, evaluation strategies based solely on aggregate accur

Agents

Dr. RTL: Autonomous Agentic RTL Optimization through Tool-Grounded Self-Improvement

arXiv:2604.14989v2 Announce Type: replace Abstract: Recent advances in large language models (LLMs) have sparked growing interest in automatic RTL optimization for better performance, power, and area

agentsarxiv-cs-ai

EmoTrans: A Benchmark for Understanding, Reasoning, and Predicting Emotion Transitions in Multimodal LLMs

arXiv:2604.23348v1 Announce Type: cross Abstract: Recent multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and generation, and are increasingly used in

Robust and Clinically Reliable EEG Biomarkers: A Cross Population Framework for Generalizable Parkinson's Disease Detection

arXiv:2604.23933v1 Announce Type: new Abstract: Developing robust and clinically reliable EEG biomarkers requires evaluation frameworks that explicitly address cross population generalization in multi

Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers

arXiv:2601.07698v2 Announce Type: replace Abstract: As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged:

TSAssistant: A Human-in-the-Loop Agentic Framework for Automated Target Safety Assessment

arXiv:2604.23938v1 Announce Type: new Abstract: Target Safety Assessment (TSA) requires systematic integration of heterogeneous evidence, including genetic, transcriptomic, target homology, pharmacolo

safetyarxiv-cs-cl

Value Alignment Tax: Measuring Value Trade-offs in LLM Alignment

arXiv:2602.12134v2 Announce Type: replace Abstract: Existing work on value alignment typically characterizes value relations statically, ignoring how alignment interventions, such as prompting, fine-t

Applications

When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Models

arXiv:2510.00626v3 Announce Type: replace-cross Abstract: Large audio-language models (LALMs) unify speech and text processing, but their robustness in noisy real-world settings remains underexplored.

applicationsarxiv-cs-cl

BLAST: Benchmarking LLMs with ASP-based Structured Testing

arXiv:2604.22306v1 Announce Type: cross Abstract: Large Language Models (LLMs) have demonstrated remarkable performance across a broad spectrum of tasks, including natural language understanding, dial

researcharxiv-cs-ai

Detecting Cognitive Signatures in Typing Behavior for Non-Intrusive Authorship Verification

arXiv:2603.00177v2 Announce Type: replace-cross Abstract: The proliferation of AI-generated text has intensified the need for reliable authorship verification, yet current output-based methods are inc

On Benchmark Hacking in ML Contests: Modeling, Insights and Design

model-releasesarxiv-cs-lg

arXiv:2604.22230v1 Announce Type: cross Abstract: Benchmark hacking refers to tuning a machine learning model to score highly on certain evaluation criteria without improving true generalization or fa

PoLO: Proof-of-Learning and Proof-of-Ownership at Once with Chained Watermarking

arXiv:2505.12296v2 Announce Type: replace-cross Abstract: Our evaluation shows that PoLO achieves extbf{99%} watermark detection accuracy for ownership verification, while preserving data privacy and

researcharxiv-cs-ai

TS-Arena -- A Live Forecast Pre-Registration Platform

arXiv:2512.20761v3 Announce Type: replace-cross Abstract: Time Series Foundation Models (TSFMs) are transforming the field of forecasting. However, evaluating them on historical data is increasingly d

Empirical Comparison of Agent Communication Protocols for Task Orchestration

arXiv:2603.22823v3 Announce Type: replace Abstract: Context. The problem of comparative evaluation of communication protocols for task orchestration by large language model (LLM) agents is considered.

Finding Meaning in Embeddings: Concept Separation Curves

arXiv:2604.21555v1 Announce Type: new Abstract: Sentence embedding techniques aim to encode key concepts of a sentence's meaning in a vector space. However, the majority of evaluation approaches for s

researcharxiv-cs-cl

Kernel-Smith: A Unified Recipe for Evolutionary Kernel Optimization

arXiv:2603.28342v2 Announce Type: replace Abstract: We present Kernel-Smith, a framework for high-performance GPU kernel and operator generation that combines a stable evaluation-driven evolutionary a

Revisiting Non-Verbatim Memorization in Large Language Models: The Role of Entity Surface Forms

arXiv:2604.21882v1 Announce Type: new Abstract: Understanding what kinds of factual knowledge large language models (LLMs) memorize is essential for evaluating their reliability and limitations. Entit

researcharxiv-cs-cl

Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry

arXiv:2604.20983v1 Announce Type: cross Abstract: Vision evaluations are typically done through multi-step processes. In most contemporary fields, experts analyze images using structured, evidence-bas

Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Text

arXiv:2604.20051v1 Announce Type: new Abstract: Self-play has recently emerged as a promising paradigm to train Large Language Models (LLMs). In self-play, the target LLM creates the task input (e.g.,

Explainable Speech Emotion Recognition: Weighted Attribute Fairness to Model Demographic Contributions to Social Bias

arXiv:2604.19763v1 Announce Type: cross Abstract: Speech Emotion Recognition (SER) systems have growing applications in sensitive domains such as mental health and education, where biased predictions

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

arXiv:2604.20006v1 Announce Type: new Abstract: Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. Ho

Large language models perceive cities through a culturally uneven baseline

arXiv:2604.20048v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to describe, evaluate and interpret places, yet it remains unclear whether they do so from a cultural

safetyarxiv-cs-cl

Local Ai

LEAD: Breaking the No-Recovery Bottleneck in Long-Horizon Reasoning

arXiv:2603.06870v2 Announce Type: replace Abstract: Long-horizon execution in Large Language Models (LLMs) remains unstable even when high-level strategies are provided. Evaluating on controlled algor

local-aiarxiv-cs-ai

MIRROR: A Hierarchical Benchmark for Metacognitive Calibration in Large Language Models

arXiv:2604.19809v1 Announce Type: new Abstract: We introduce MIRROR, a benchmark comprising eight experiments across four metacognitive levels that evaluates whether large language models can use self

Self-Awareness before Action: Mitigating Logical Inertia via Proactive Cognitive Awareness

arXiv:2604.20413v1 Announce Type: new Abstract: Large language models perform well on many reasoning tasks, yet they often lack awareness of whether their current knowledge or reasoning state is compl

SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks

arXiv:2604.20087v1 Announce Type: new Abstract: Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how

X-PCR: A Benchmark for Cross-modality Progressive Clinical Reasoning in Ophthalmic Diagnosis

model-releasesarxiv-cs-cv

arXiv:2604.20350v1 Announce Type: new Abstract: Despite significant progress in Multi-modal Large Language Models (MLLMs), their clinical reasoning capacity for multi-modal diagnosis remains largely u

Analytical Extraction of Conditional Sobol' Indices via Basis Decomposition of Polynomial Chaos Expansions

model-releasesarxiv-cs-lg

arXiv:2604.19165v1 Announce Type: cross Abstract: In uncertainty quantification, evaluating sensitivity measures under specific conditions (i.e., conditional Sobol' indices) is essential for systems w

Benchmarking Misuse Mitigation Against Covert Adversaries

arXiv:2506.06414v2 Announce Type: replace-cross Abstract: Existing language model safety evaluations focus on overt attacks and low-stakes tasks. In reality, an attacker can easily subvert existing sa

Compile to Compress: Boosting Formal Theorem Provers by Compiler Outputs

arXiv:2604.18587v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated significant potential in formal theorem proving, yet state-of-the-art performance often necessitates pr

Rank-Turbulence Delta and Interpretable Approaches to Stylometric Delta Metrics

arXiv:2604.19499v1 Announce Type: new Abstract: This article introduces two new measures for authorship attribution - Rank-Turbulence Delta and Jensen-Shannon Delta - which generalise Burrows's classi

Time Series Augmented Generation for Financial Applications

arXiv:2604.19633v1 Announce Type: new Abstract: Evaluating the reasoning capabilities of Large Language Models (LLMs) for complex, quantitative financial tasks is a critical and unsolved challenge. St

Towards Optimal Agentic Architectures for Offensive Security Tasks

arXiv:2604.18718v1 Announce Type: cross Abstract: Agentic security systems increasingly audit live targets with tool-using LLMs, but prior systems fix a single coordination topology, leaving unclear w

CGCMA: Conditionally-Gated Cross-Modal Attention for Event-Conditioned Asynchronous Fusion

arXiv:2604.16411v1 Announce Type: new Abstract: We study asynchronous alignment, a first-class multimodal learning setting in which a dense primary stream must be fused with sporadic external context

safetyarxiv-cs-lg

Chain Of Interaction Benchmark (COIN): When Reasoning meets Embodied Interaction

model-releasesarxiv-cs-ro

arXiv:2604.16886v1 Announce Type: new Abstract: Generalist embodied agents must perform interactive, causally-dependent reasoning, continually interacting with the environment, acquiring information,

EchoChain: A Full-Duplex Benchmark for State-Update Reasoning Under Interruptions

arXiv:2604.16456v1 Announce Type: new Abstract: Real-time voice assistants must revise task state when users interrupt mid-response, but existing spoken-dialog benchmarks largely evaluate turn-based i

From User Recognition to Activity Counting: An Identity-Agnostic Approach to Multi-User WiFi Sensing

arXiv:2604.16572v1 Announce Type: new Abstract: Wi-Fi Channel State Information (CSI) enables device-free human activity recognition, but existing multi-user approaches assume a fixed set of known use

Heterogeneous Self-Play for Realistic Highway Traffic Simulation

arXiv:2604.16406v1 Announce Type: cross Abstract: Realistic highway simulation is critical for scalable safety evaluation of autonomous vehicles, particularly for interactions that are too rare to stu

safetyarxiv-cs-lg

LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

arXiv:2601.16504v3 Announce Type: replace Abstract: Commonsense reasoning often involves evaluating multiple plausible interpretations rather than selecting a single atomic answer, yet most benchmarks

Measuring Representation Robustness in Large Language Models for Geometry

arXiv:2604.16421v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated on mathematical reasoning, yet their robustness to equivalent problem representations remains po

Applications

Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measure

arXiv:2412.15176v3 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly employed in real-world applications, driving the need to evaluate the trustworthiness of their generat

applicationsarxiv-cs-lg