Search: “evaluation”

SCAN: Structured Capability Assessment and Navigation for LLMs

arXiv:2505.06698v4 Announce Type: replace Abstract: Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives t

4 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototype

DGX agent

arXiv:2604.27275v1 Announce Type: cross Abstract: Large language model (LLM) reading assistants are increasingly used in settings that require interpretation rather than simple retrieval. In these con

researcharxiv-cs-ai

1 May 2026

Evaluating TabPFN for Mild Cognitive Impairment to Alzheimer's Disease Conversion in Data Limited Settings

arXiv:2604.27195v1 Announce Type: new Abstract: Accurate prediction of conversion from Mild Cognitive Impairment (MCI) to Alzheimers Diseases (AD) is essential for early intervention, however, develop

researcharxiv-cs-ai

1 May 2026

Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations

arXiv:2604.26148v1 Announce Type: cross Abstract: AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality

30 Apr 2026

Evaluating Strategic Reasoning in Forecasting Agents

arXiv:2604.26106v1 Announce Type: new Abstract: Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to t

safetyarxiv-cs-ai

30 Apr 2026

Talent or Luck? Evaluating Attribution Bias in Large Language Models

arXiv:2505.22910v2 Announce Type: replace Abstract: When a student fails an exam, do we tend to blame their effort or the test's difficulty? Attribution, defined as how reasons are assigned to event o

safetyarxiv-cs-cl

30 Apr 2026

Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts

arXiv:2410.06723v2 Announce Type: replace-cross Abstract: Pathology foundation models (PFMs) have emerged as powerful pretrained encoders for computational pathology, but their robustness under clinic

29 Apr 2026

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

arXiv:2510.16340v2 Announce Type: replace Abstract: Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensi

safetyarxiv-cs-cl

29 Apr 2026

Evaluating CUDA Tile for AI Workloads on Hopper and Blackwell GPUs

arXiv:2604.23466v1 Announce Type: cross Abstract: NVIDIA's CUDA Tile (CuTile) introduces a Python-based, tile-centric abstraction for GPU kernel development that aims to simplify programming while ret

Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards

arXiv:2604.23341v1 Announce Type: cross Abstract: The deployment of Large Language Models (LLMs) as assistants in electric grid operations promises to streamline compliance and decision-making but exp

Game-Time: Evaluating Temporal Dynamics in Spoken Language Models

arXiv:2509.26388v3 Announce Type: replace-cross Abstract: Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of

Tutorials

Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms

arXiv:2604.23842v1 Announce Type: cross Abstract: Neologisms and emerging slang are central to daily conversation, yet challenging for non-native speakers (NNS) to interpret and use appropriately in c

tutorialsarxiv-cs-ai

Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines

arXiv:2604.22661v1 Announce Type: cross Abstract: Large Language Models (LLMs) have made query reformulation ubiquitous in modern retrieval and Retrieval-Augmented Generation (RAG) pipelines, enabling

27 Apr 2026

Agents

Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents

arXiv:2604.22452v1 Announce Type: new Abstract: Collective intelligence refers to the ability of a group to achieve outcomes beyond what any individual member can accomplish alone. As large language m

agentsarxiv-cs-ai

27 Apr 2026

EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

arXiv:2604.21229v1 Announce Type: cross Abstract: Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce Engrama

Evaluating Post-hoc Explanations of the Transformer-based Genome Language Model DNABERT-2

arXiv:2604.21690v1 Announce Type: new Abstract: Explaining deep neural network predictions on genome sequences enables biological insight and hypothesis generation-often of greater interest than predi

researcharxiv-cs-lg

From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media

arXiv:2604.21786v1 Announce Type: new Abstract: Social media platforms have become primary arenas for climate communication, generating millions of images and posts that - if systematically analysed -

Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores

arXiv:2511.20697v4 Announce Type: replace-cross Abstract: Understanding complete musical scores entails integrated reasoning over pitch, rhythm, harmony, and large-scale structure, yet the ability of

Evaluating Black-Box Vulnerabilities with Wasserstein-Constrained Data Perturbations

arXiv:2603.15867v2 Announce Type: replace Abstract: The growing use of Machine Learning (ML) tools comes with critical challenges, such as limited model explainability. We propose a global explainabil

safetyarxiv-cs-lg

23 Apr 2026

What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review

arXiv:2604.19998v1 Announce Type: new Abstract: Evaluating AI-generated reviews by verdict agreement is widely recognized as insufficient, yet current alternatives rarely audit which concerns a system

safetyarxiv-cs-ai

23 Apr 2026

Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning

arXiv:2604.19459v1 Announce Type: new Abstract: Formal verification guarantees proof validity but not formalization faithfulness. For natural-language logical reasoning, where models construct axiom s

22 Apr 2026

MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

arXiv:2604.18914v1 Announce Type: cross Abstract: While multilingual large language models (LLMs) perform well on high-level tasks like translation and question answering, their ability to handle gram

22 Apr 2026

Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

arXiv:2601.11886v2 Announce Type: replace Abstract: In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the

safetyarxiv-cs-cl

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

arXiv:2604.18490v1 Announce Type: new Abstract: Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely

ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

arXiv:2505.19897v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdi

Tape: A Cellular Automata Benchmark for Evaluating Rule-Shift Generalization in Reinforcement Learning

model-releasesarxiv-cs-lg

arXiv:2601.04695v2 Announce Type: replace-cross Abstract: Out-of-distribution generalization in reinforcement learning is hard to diagnose when benchmark shifts mix dynamics, observations, goals, and

Evaluating LLMs as Human Surrogates in Controlled Experiments

arXiv:2604.15329v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used to simulate human responses in behavioral research, yet it remains unclear when LLM-generated data

researcharxiv-cs-ai

20 Apr 2026

RoleConflictBench: A Benchmark of Role Conflict Scenarios for Evaluating LLMs' Contextual Sensitivity

arXiv:2509.25897v2 Announce Type: replace-cross Abstract: People often encounter role conflicts -- social dilemmas where the expectations of multiple roles clash and cannot be simultaneously fulfilled

20 Apr 2026

Growing Pains: Extensible and Efficient LLM Benchmarking Via Fixed Parameter Calibration

arXiv:2604.12843v1 Announce Type: new Abstract: The rapid release of both language models and benchmarks makes it increasingly costly to evaluate every model on every dataset. In practice, models are

Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems

arXiv:2603.01045v2 Announce Type: replace-cross Abstract: Large language models are increasingly deployed in multi-agent systems to overcome context limitations by distributing information across agen

Tutorials

Why Did Apple Fall: Evaluating Curiosity in Large Language Models

arXiv:2510.20635v2 Announce Type: replace-cross Abstract: Curiosity serves as a pivotal conduit for human beings to discover and learn new knowledge. Recent advancements of large language models (LLMs

tutorialsarxiv-cs-ai

WiseOWL: A Methodology for Evaluating Ontological Descriptiveness and Semantic Correctness for Ontology Reuse and Ontology Recommendations

arXiv:2604.12025v1 Announce Type: new Abstract: The Semantic Web standardizes concept meaning for humans and machines, enabling machine-operable content and consistent interpretation that improves adv

safetyarxiv-cs-ai

Do Thought Streams Matter? Evaluating Reasoning in Gemini Vision-Language Models for Video Scene Understanding

arXiv:2604.11177v1 Announce Type: new Abstract: We benchmark how internal reasoning traces, which we call thought streams, affect video scene understanding in vision-language models. Using four config

Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

arXiv:2604.09585v1 Announce Type: cross Abstract: Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applyi

researcharxiv-cs-ai

GazeVaLM: A Multi-Observer Eye-Tracking Benchmark for Evaluating Clinical Realism in AI-Generated X-Rays

arXiv:2604.11653v1 Announce Type: new Abstract: We introduce GazeVaLM, a public eye-tracking dataset for studying clinical perception during chest radiograph authenticity assessment. The dataset compr

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation

arXiv:2604.11290v1 Announce Type: new Abstract: Synthesizing supervised finetuning (SFT) data from language models (LMs) to teach smaller models multilingual tasks has become increasingly common. Howe

Who Wrote This Line? Evaluating the Detection of LLM-Generated Classical Chinese Poetry

arXiv:2604.10101v1 Announce Type: new Abstract: The rapid development of large language models (LLMs) has extended text generation tasks into the literary domain. However, AI-generated literary creati

Applications

Why Don't You Know? Evaluating the Impact of Uncertainty Sources on Uncertainty Quantification in LLMs

arXiv:2604.10495v1 Announce Type: new Abstract: As Large Language Models (LLMs) are increasingly deployed in real-world applications, reliable uncertainty quantification (UQ) becomes critical for safe

applicationsarxiv-cs-cl

Evaluating Low-Light Image Enhancement Across Multiple Intensity Levels

arXiv:2511.15496v2 Announce Type: replace Abstract: Imaging in low-light environments is challenging due to reduced scene radiance, which leads to elevated sensor noise and reduced color saturation. M

10 Apr 2026

From Individuals to Interactions: Benchmarking Gender Bias in Multimodal Large Language Models from the Lens of Social Relationship

arXiv:2506.23101v2 Announce Type: replace-cross Abstract: Ensuring the trustworthiness of Multimodal Large Language Models (MLLMs) requires rigorous evaluation of gender bias, particularly in socially

15 Oct 2026

Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

arXiv:2610.18610v1 Announce Type: cross Abstract: Dialogue and collaborative assistants repeatedly judge what a partner knows, believes, or intends. Typed decision models offer a potential alternative

15 Oct 2026

Specification-Based Testing of Large Language Models

arXiv:2610.14443v1 Announce Type: cross Abstract: Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined bench

15 Oct 2026

AMemOps: Evaluating Memory Updates in Long Audio Streams

model-releasesarxiv-cs-lg

arXiv:2610.18652v1 Announce Type: cross Abstract: Audio memory must recognise recurring sources across acoustic changes while keeping their attributes up to date. Overall accuracy does not reveal how

ChangeT^3: Evaluating Vision Test-Time Training for Remote Sensing Change Detection

arXiv:2610.15813v1 Announce Type: new Abstract: Bitemporal remote sensing change detection requires representations that preserve spatial detail while capturing contextual information relevant to chan

Tutorials

Deploy and evaluate models with Amazon SageMaker JumpStart

Learn how to deploy an available model from Amazon SageMaker JumpStart, run a lightweight and repeatable performance test using the SageMaker Python SDK, and delete the endpoint when testing is comple

tutorialsaws-ml-blog

EviScene: Beyond 'Is It AI?' for Agentic Image Factuality Assessment

arXiv:2610.14609v1 Announce Type: new Abstract: Determining whether an image is AI-generated does not establish whether the situation it depicts is factually supported. We formulate Image Factuality A

Agents

Predict Before You Modify: Learning Improvement Dynamics for Self-Evolving Agents

arXiv:2610.13991v1 Announce Type: new Abstract: Self-evolving agents improve through repeated cycles of modification and evaluation. However, existing methods receive feedback on modifications only af

agentsarxiv-cs-lg

REEL: Learning to Retrieve, Evaluate, and Evolve Visual Memory for Any Long-Horizon Multi-Shot Video Generation

arXiv:2610.16632v1 Announce Type: new Abstract: Memory-based long-video generation aims to combine user control with long-range visual consistency. We organize this process into three functional compo

safetyarxiv-cs-cv