Search: “evaluation”

arXiv:2606.11235v1 Announce Type: new Abstract: A key step in knowledge discovery is the evaluation of data mining results. In several applications, including pattern mining, graph analysis, and other

applicationsarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Geometric Metrics and LLMs: What They Measure and When They Work

DGX agent

arXiv:2509.25359v2 Announce Type: replace-cross Abstract: We present a systematic stress-test of geometric metrics for LLM evaluation. Rank-based geometric properties of internal representations have

researcharxiv-cs-ai

11 Jun 2026

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology

arXiv:2606.12407v1 Announce Type: new Abstract: General-purpose large language models (LLMs) are routinely used as baselines when evaluating specialized pathology models on whole-slide images (WSIs).

IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization

arXiv:2606.12086v1 Announce Type: new Abstract: Contextualized assessment offers high ecological validity for evaluating creativity but introduces a critical challenge: observed performance may be con

Measuring Semantic Progress in Multi-turn Dialogue via Information Gain

arXiv:2606.12332v1 Announce Type: new Abstract: Evaluating multi-turn dialogue is challenging because quality emerges across turns rather than within individual responses. We focus on a key dimension

safetyarxiv-cs-cl

MedCTA: A Benchmark for Clinical Tool Agents

arXiv:2606.11702v1 Announce Type: cross Abstract: To make clinically grounded decisions, medical AI agents are expected to go beyond simple recognition and be capable of tool retrieval, evidence acqui

enterprise-aisamsungworkforce

Applications

Samsung Electronics deploys multiple AI tools for employees

Human

Samsung's own announcement says its DX division selected ChatGPT, Gemini and Claude after a 2,500-person evaluation; no measured outcome is reported.

Semantic Grading of Written Answers in Low-Resource Language Bangla Using a Fine-Tuned Lightweight Language Model

arXiv:2606.11931v1 Announce Type: new Abstract: Bangla is among the world's most widely spoken languages, yet it remains underserved in educational NLP research. In many remote and rural regions, acce

researcharxiv-cs-cl

Agents

3D-CoS: A New 3D Reconstruction Paradigm Based on VLM Code Synthesis

arXiv:2606.10478v1 Announce Type: new Abstract: Most recent 3D reconstruction and editing systems operate on implicit and explicit representations such as NeRF, point clouds, or meshes. While these re

agentsarxiv-cs-cv

Anomaly Detection and Root Cause Analysis for Microservice Systems

arXiv:2606.09942v1 Announce Type: cross Abstract: Microservice systems are widely used to build cloud applications, yet their complexity makes failures inevitable, degrading user experience and causin

ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark

arXiv:2505.23851v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly applied to symbolic mathematics, yet existing evaluations often conflate pattern memorization wi

Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations

model-releasesarxiv-cs-cl

arXiv:2606.10281v1 Announce Type: cross Abstract: This paper presents AuditBench, a new benchmark dataset for evaluating the capabilities of LLMs at investigating security-related system audit logs. W

Divide-and-Conquer Modeling for the CTF-4-Science Lorenz Benchmark

arXiv:2606.10084v1 Announce Type: cross Abstract: This work presents a divide-and-conquer modeling strategy for the CTF-4-Science Lorenz benchmark, which evaluates chaotic-system prediction across twe

Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate

arXiv:2606.10307v1 Announce Type: new Abstract: Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whethe

safetyarxiv-cs-cl

Ethical and Technical Limits of Deepfake Speech Datasets

arXiv:2606.10911v1 Announce Type: cross Abstract: Claims about the robustness and fairness of deepfake speech detectors are only as credible as the datasets used to train and evaluate those systems. W

Frontier Coding Agents Use Metaprogramming to Adapt to Unfamiliar Programming Languages

arXiv:2606.10933v1 Announce Type: new Abstract: LLM-based coding agents are usually evaluated in familiar software settings: mainstream languages, common libraries, and public repositories. These benc

Generative Explainability for Next-Generation Networks: LLM-Augmented XAI with Mutual Feature Interactions

arXiv:2606.10942v1 Announce Type: cross Abstract: As artificial intelligence and machine learning (AI/ML) models become integral to network operations, their lack of transparency poses a significant b

researcharxiv-cs-ai

How can we assess human-agent interactions? Case studies in software agent design

arXiv:2510.09801v3 Announce Type: replace Abstract: While benchmarks measure the accuracy of LLM-powered agents, they mostly assume full automation, failing to represent the collaborative nature of re

P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning

arXiv:2606.11152v1 Announce Type: new Abstract: Multimodal large language models can write code to produce complex programs as well as use programs to do 3D modeling, which opens up a new avenue for 3

ProbeLLM: Automating Principled Diagnosis of LLM Failures

model-releasesarxiv-cs-cl

arXiv:2602.12966v2 Announce Type: replace Abstract: Understanding how and why large language models (LLMs) fail is becoming a central challenge as models rapidly evolve and static evaluations fall beh

Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Tasks

arXiv:2606.10967v1 Announce Type: new Abstract: Visual in-context learning has been proposed as a pathway towards dynamic models that can generate predictions based on a provided context and thereby c

Agents

The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

arXiv:2606.10296v1 Announce Type: cross Abstract: Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning

agentsarxiv-cs-ai

When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models

arXiv:2606.10740v1 Announce Type: new Abstract: Failures in multi-turn reasoning models are largely invisible to terminal-score evaluation. A model can lock onto an unsafe stance early in a long dialo

WorldOlympiad: Can Your World Model Survive a Triathlon?

arXiv:2606.11129v1 Announce Type: new Abstract: We introduce WorldOlympiad, a benchmark for diagnosing video-based world models across physical faithfulness, geometric consistency, and interaction fid

A Comparison of SSL-Based Feature Extractors and Back-End Classifiers for Spoofing Detection: A Multi-Corpus Training and Cross-Linguistic Analysis

model-releasesarxiv-cs-lg

arXiv:2606.08669v1 Announce Type: cross Abstract: Voice biometric systems face growing threats from spoofing attacks, yet the evaluation of detection models remains inconsistent across datasets. To in

Tools

[AINews] FrontierCode: Benchmarking for Code Quality over Slop

FrontierCode is a benchmarking framework designed to evaluate code quality and move beyond 'slop'—low-quality or poorly generated code—in AI-generated outputs. The framework likely provides metrics an

toolslatent-space

Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?

arXiv:2606.08894v1 Announce Type: new Abstract: Reasoning Vision-Language Models (VLMs) achieve strong performance on complex multimodal tasks, but reliable real-world application requires handling vi

C3VD-DEFCOL: A Deformable Colonoscopy Dataset with Time-Resolved 3D Ground Truth and Realistic Appearance

arXiv:2606.07891v1 Announce Type: new Abstract: 3D reconstruction could improve colonoscopy by estimating mucosal coverage and alerting clinicians to missed regions during screening. However, algorith

ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China

arXiv:2606.08959v1 Announce Type: new Abstract: We introduce ChinaHeritaQA, a multimodal benchmark dataset for evaluating the cultural reasoning abilities of vision-language models (VLMs) on UNESCO Wo

DHAuDS: A Dynamic and Heterogeneous Audio Benchmark for Test-Time Adaptation

model-releasesarxiv-cs-lg

arXiv:2511.18421v2 Announce Type: replace-cross Abstract: Existing Test-time Adaptation (TTA) studies rely heavily on static and homogeneous corruption protocols, such as ImageNet-C and CIFAR-10-C/100

Emergent alignment and the projectability of ethical personas

arXiv:2606.09475v1 Announce Type: new Abstract: Work on `emergent misalignment' shows that finetuning LLMs on narrow tasks can induce broadly misaligned behavior. This supports the `persona selection'

HARP: Efficient Data Selection for Finetuning Large Language Models

arXiv:2606.07690v1 Announce Type: cross Abstract: Finetuning data selection requires balancing two competing goals: selecting examples that improve the downstream objective, and doing so without repea

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

arXiv:2606.09169v1 Announce Type: new Abstract: In recent years, unified multimodal models (UMMs) have emerged to support both understanding and generation within a single framework. Mastering dynamic

Learning to Solve Generative ODEs Beyond the Linear Span

arXiv:2606.08672v1 Announce Type: new Abstract: Diffusion and flow generative models sample by integrating a learned ODE, but high quality still requires many sequential model evaluations. Solver lear

researcharxiv-cs-cv

Mesh Graph Neural Network Framework for Accelerating Finite Element Simulation for Arbitrary Geometries

arXiv:2606.08287v1 Announce Type: new Abstract: Finite element analysis (FEA) is essential for structural design but remains computationally expensive, particularly when evaluating multiple design ite

researcharxiv-cs-lg

No Modality Left Behind: Adapting to Missing Modalities via Knowledge Distillation for Brain Tumor Segmentation

arXiv:2509.15017v2 Announce Type: replace Abstract: Accurate brain tumor segmentation is essential for preoperative evaluation and personalized treatment. Multi-modal MRI is widely used due to its abi

safetyarxiv-cs-cv

Offline Reinforcement Learning for Plasma Control in Nuclear Fusion: Codebase and Benchmark

arXiv:2606.07550v1 Announce Type: cross Abstract: Offline reinforcement learning (RL) offers a promising route for developing plasma controllers from historical tokamak data, since online trial-and-er

Local Ai

PAI: Preserving Amplitude Information in Representation-Based Time-Series Anomaly Detection

arXiv:2606.08935v1 Announce Type: cross Abstract: Representation-based time-series anomaly detection algorithms significantly outperform other methods on diverse anomaly detection tasks. However, we n

local-aiarxiv-cs-ai

Post-Rejection Follow-up Sampling: A Methodology for Counterfactual Outcome Measurement in Algorithmic DEX Trading

arXiv:2606.08228v1 Announce Type: cross Abstract: Algorithmic trading systems on decentralised exchanges (DEXs) reject most candidate tokens they evaluate. The counterfactual outcome of rejected candi

researcharxiv-cs-lg

PRISM: PRior-guided Imagination Sampling in world Models

arXiv:2606.07974v1 Announce Type: cross Abstract: A learned world model provides a powerful physical intuition for evaluating future states. But its effectiveness in continuous control also depends cr

SEF-CLGC at SemEval-2026 Task 11: Logical Notation Impact on Language Model Performance

arXiv:2606.09157v1 Announce Type: cross Abstract: This paper revisits our pipeline called Syllogistic Evaluation Framework-Common Logic Grammar Construction (SEF-CLGC). We combine formal logical notat

Agents

SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History

arXiv:2606.08671v1 Announce Type: new Abstract: Agent skills extend language-model agents with task-specific procedures, scripts, and references, but the tasks and environments they target continually

agentsarxiv-cs-lg

SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Tasks

arXiv:2606.09669v1 Announce Type: new Abstract: Spatial reasoning is a foundational capability for multimodal large language models (MLLMs) to perceive and operate within the physical world. However,

Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning

arXiv:2606.08735v1 Announce Type: new Abstract: Quality-diversity reinforcement learning (QD-RL) aims to construct policy repertoires that contain both high-performing and behaviorally diverse policie

TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking

arXiv:2602.03224v2 Announce Type: replace Abstract: Test-time evolution of agent memory represents a pivotal paradigm for advancing AGI, as it strengthens complex reasoning through experience accumula

The AI Epistemic Deference Index: A Continuous Measure of Sycophancy

arXiv:2606.07897v1 Announce Type: new Abstract: Current AI models frequently exhibit epistemic sycophancy, endorsing claims to agree with a user. Existing evaluations typically measure this either by

When Do Local Score Models Extrapolate Across Size? A Diagnostic Theory and Benchmark

model-releasesarxiv-cs-lg

arXiv:2606.09705v1 Announce Type: new Abstract: Scientific generative modeling often requires size transfer, where models trained on small systems are evaluated on larger ones. While translation-invar

Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Driving

arXiv:2606.09644v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) achieve strong results on visual reasoning benchmarks, but answer accuracy alone does not indicate whether a