Search: “evaluation”

Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke

arXiv:2605.09236v1 Announce Type: cross Abstract: While digitized corpora have transformed the study of intellectual transmission, current methods rely heavily on lexical text reuse detection, capturi

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models

DGX agent

arXiv:2605.10002v1 Announce Type: new Abstract: Large vision-language models (VLMs) demonstrate strong performance in medical image understanding, but frequently generate clinically plausible yet inco

model-releasesarxiv-cs-cv

12 May 2026

Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimuli

arXiv:2509.21671v2 Announce Type: replace Abstract: High-resolution neural datasets enable foundation models for the next generation of brain-computer interfaces and neurological treatments. The commu

researcharxiv-cs-lg

OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

arXiv:2605.08904v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and tool use. However, the fundamental cognitive faculties essential

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605.09883v1 Announce Type: cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores gen

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

arXiv:2605.07247v1 Announce Type: new Abstract: Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments

11 May 2026

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

arXiv:2605.07073v1 Announce Type: new Abstract: Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls.

11 May 2026

KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

model-releasesarxiv-cs-lg

arXiv:2605.04956v1 Announce Type: new Abstract: LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capabili

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

arXiv:2605.04313v1 Announce Type: new Abstract: Causal reasoning in natural language requires identifying relevant variables, understanding their interactions, and reasoning about effects and interven

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

arXiv:2605.03361v2 Announce Type: new Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organizati

StoryAlign: Evaluating and Training Reward Models for Story Generation

arXiv:2605.04831v1 Announce Type: new Abstract: Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly a

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

arXiv:2605.01986v1 Announce Type: new Abstract: What if the twelve jurors of Sidney Lumet's 12 Angry Men (1957) were not men, but large language models? Would the one juror who disagrees still be able

Local Ai

BIT.UA-AAUBS at ArchEHR-QA 2026: Evaluating Open-Source and Proprietary LLMs via Prompting in Low-Resource QA

arXiv:2605.03618v1 Announce Type: new Abstract: This paper presents the joint participation of the BIT.UA and AAUBS groups in the ArchEHR-QA 2026 shared task, which focuses on clinical question answer

local-aiarxiv-cs-cl

Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior

arXiv:2605.03855v1 Announce Type: new Abstract: Human-AI collaboration requires AI agents to understand human behavior for effective coordination. While advances in foundation models show promising ca

researcharxiv-cs-ro

Evaluating Reasoning Models for Queries with Presuppositions

arXiv:2605.03050v1 Announce Type: new Abstract: Millions of users turn to AI models for their information needs. It is conceivable that a large number of user queries contain assumptions that may be f

Repurposing and Evaluating the (In)Feasibility of Dataset Poisoning enabled Watermarking for Contrastive Learning

arXiv:2605.01834v1 Announce Type: cross Abstract: Contrastive learning (CL) reduces annotation cost via auto-derived supervisory signals. Since large-scale in-house CL datasets are infeasible, relianc

Creating and Evaluating Figurative Language Dataset for Sindhi

arXiv:2605.01323v1 Announce Type: new Abstract: In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various b

Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages

arXiv:2605.02608v1 Announce Type: new Abstract: Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-

From Concept to Capability: Evaluating 3D Gaussian Splatting for Synthetic Scene Editing in Autonomous Driving

arXiv:2605.01995v1 Announce Type: new Abstract: The perception of an Autonomous Driving System (ADS) critically depends on relevant, comprehensive, and diverse datasets to ensure its safety while oper

safetyarxiv-cs-cv

GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models

arXiv:2605.01203v1 Announce Type: cross Abstract: Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly genera

Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search

model-releasesarxiv-cs-lg

arXiv:2605.01936v1 Announce Type: new Abstract: In sequential search, alternatives are tested until the true class is found. Standard proper scoring rules like log loss are local, ignoring the ranking

FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

arXiv:2605.00706v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal

Image Score: Learning and Evaluating Human Preferences for Mercari Search

arXiv:2408.11349v2 Announce Type: replace Abstract: Mercari is the largest C2C e-commerce marketplace in Japan, having more than 20 million active monthly users. Search being the fundamental way to di

researcharxiv-cs-cv

Linking Behaviour and Perception to Evaluate Meaningful Human Control over Partially Automated Driving

arXiv:2605.00556v1 Announce Type: cross Abstract: Partial driving automation creates a tension: drivers remain legally responsible for vehicle behaviour, yet their active control is significantly redu

safetyarxiv-cs-ro

SCAN: Structured Capability Assessment and Navigation for LLMs

arXiv:2505.06698v4 Announce Type: replace Abstract: Evaluating Large Language Models (LLMs) has become increasingly important, with automatic evaluation benchmarks gaining prominence as alternatives t

Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototype

arXiv:2604.27275v1 Announce Type: cross Abstract: Large language model (LLM) reading assistants are increasingly used in settings that require interpretation rather than simple retrieval. In these con

1 May 2026

Evaluating TabPFN for Mild Cognitive Impairment to Alzheimer's Disease Conversion in Data Limited Settings

arXiv:2604.27195v1 Announce Type: new Abstract: Accurate prediction of conversion from Mild Cognitive Impairment (MCI) to Alzheimers Diseases (AD) is essential for early intervention, however, develop

1 May 2026

Beyond Screenshots: Evaluating VLMs' Understanding of UI Animations

arXiv:2604.26148v1 Announce Type: cross Abstract: AI agents operating on user interfaces must understand how interfaces communicate state and feedback to act reliably. As a core communicative modality

30 Apr 2026

Evaluating Strategic Reasoning in Forecasting Agents

arXiv:2604.26106v1 Announce Type: new Abstract: Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to t

safetyarxiv-cs-ai

30 Apr 2026

Talent or Luck? Evaluating Attribution Bias in Large Language Models

arXiv:2505.22910v2 Announce Type: replace Abstract: When a student fails an exam, do we tend to blame their effort or the test's difficulty? Attribution, defined as how reasons are assigned to event o

safetyarxiv-cs-cl

30 Apr 2026

Evaluating Computational Pathology Foundation Models for Prostate Cancer Grading under Distribution Shifts

model-releasesarxiv-cs-cv

arXiv:2410.06723v2 Announce Type: replace-cross Abstract: Pathology foundation models (PFMs) have emerged as powerful pretrained encoders for computational pathology, but their robustness under clinic

29 Apr 2026

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

arXiv:2510.16340v2 Announce Type: replace Abstract: Recent advances in post-training techniques have endowed Large Language Models (LLMs) with enhanced capabilities for tackling complex, logic-intensi

safetyarxiv-cs-cl

29 Apr 2026

Evaluating CUDA Tile for AI Workloads on Hopper and Blackwell GPUs

arXiv:2604.23466v1 Announce Type: cross Abstract: NVIDIA's CUDA Tile (CuTile) introduces a Python-based, tile-centric abstraction for GPU kernel development that aims to simplify programming while ret

Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards

arXiv:2604.23341v1 Announce Type: cross Abstract: The deployment of Large Language Models (LLMs) as assistants in electric grid operations promises to streamline compliance and decision-making but exp

Game-Time: Evaluating Temporal Dynamics in Spoken Language Models

arXiv:2509.26388v3 Announce Type: replace-cross Abstract: Conversational Spoken Language Models (SLMs) are emerging as a promising paradigm for real-time speech interaction. However, their capacity of

Tutorials

Reheat Nachos for Dinner? Evaluating AI Support for Cross-Cultural Communication of Neologisms

arXiv:2604.23842v1 Announce Type: cross Abstract: Neologisms and emerging slang are central to daily conversation, yet challenging for non-native speakers (NNS) to interpret and use appropriately in c

tutorialsarxiv-cs-ai

Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelines

arXiv:2604.22661v1 Announce Type: cross Abstract: Large Language Models (LLMs) have made query reformulation ubiquitous in modern retrieval and Retrieval-Augmented Generation (RAG) pipelines, enabling

27 Apr 2026

Agents

Superminds Test: Actively Evaluating Collective Intelligence of Agent Society via Probing Agents

arXiv:2604.22452v1 Announce Type: new Abstract: Collective intelligence refers to the ability of a group to achieve outcomes beyond what any individual member can accomplish alone. As large language m

agentsarxiv-cs-ai

27 Apr 2026

EngramaBench: Evaluating Long-Term Conversational Memory with Structured Graph Retrieval

arXiv:2604.21229v1 Announce Type: cross Abstract: Large language model assistants are increasingly expected to retain and reason over information accumulated across many sessions. We introduce Engrama

Evaluating Post-hoc Explanations of the Transformer-based Genome Language Model DNABERT-2

arXiv:2604.21690v1 Announce Type: new Abstract: Explaining deep neural network predictions on genome sequences enables biological insight and hypothesis generation-often of greater interest than predi

researcharxiv-cs-lg

From Codebooks to VLMs: Evaluating Automated Visual Discourse Analysis for Climate Change on Social Media

model-releasesarxiv-cs-cv

arXiv:2604.21786v1 Announce Type: new Abstract: Social media platforms have become primary arenas for climate communication, generating millions of images and posts that - if systematically analysed -

Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores

arXiv:2511.20697v4 Announce Type: replace-cross Abstract: Understanding complete musical scores entails integrated reasoning over pitch, rhythm, harmony, and large-scale structure, yet the ability of

Evaluating Black-Box Vulnerabilities with Wasserstein-Constrained Data Perturbations

arXiv:2603.15867v2 Announce Type: replace Abstract: The growing use of Machine Learning (ML) tools comes with critical challenges, such as limited model explainability. We propose a global explainabil

safetyarxiv-cs-lg

23 Apr 2026

What Makes a Good AI Review? Concern-Level Diagnostics for AI Peer Review

arXiv:2604.19998v1 Announce Type: new Abstract: Evaluating AI-generated reviews by verdict agreement is widely recognized as insufficient, yet current alternatives rarely audit which concerns a system

safetyarxiv-cs-ai

23 Apr 2026

Do LLMs Game Formalization? Evaluating Faithfulness in Logical Reasoning

arXiv:2604.19459v1 Announce Type: new Abstract: Formal verification guarantees proof validity but not formalization faithfulness. For natural-language logical reasoning, where models construct axiom s

22 Apr 2026

MORPHOGEN: A Multilingual Benchmark for Evaluating Gender-Aware Morphological Generation

arXiv:2604.18914v1 Announce Type: cross Abstract: While multilingual large language models (LLMs) perform well on high-level tasks like translation and question answering, their ability to handle gram

22 Apr 2026

Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

arXiv:2601.11886v2 Announce Type: replace Abstract: In high-stakes domains like medicine, it may be generally desirable for models to faithfully adhere to the context provided. But what happens if the

safetyarxiv-cs-cl

21 Apr 2026

LQM: Linguistically Motivated Multidimensional Quality Metrics for Machine Translation

arXiv:2604.18490v1 Announce Type: new Abstract: Existing MT evaluation frameworks, including automatic metrics and human evaluation schemes such as Multidimensional Quality Metrics (MQM), are largely