Search: “evaluation”

What to Forget in Unlearning? Forget Set Curation for Language Models

arXiv:2608.14855v1 Announce Type: new Abstract: Machine unlearning aims to remove targeted data or behaviors from a trained model without retraining from scratch. Yet most evaluations assume that the

18 Aug 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

AI Research Preference Models

DGX agent

arXiv:2608.13940v1 Announce Type: new Abstract: AI research agents (AIRA) can now propose, implement, and evaluate their own machine learning experiments, but progress on frontier tasks is throttled b

model-releasesarxiv-cs-ai

17 Aug 2026

AlignFace: Human-Aligned Face Similarity Metric with Interpretable Concept Relations

arXiv:2608.14130v1 Announce Type: cross Abstract: Computer vision models for generated facial content, such as face editing and privacy protection, increasingly affect people, requiring similarity met

CPI-Bench: A Comprehensive,Practical and Intelligent Benchmark for Real-World Image Editing

model-releasesarxiv-cs-cv

arXiv:2608.14546v1 Announce Type: new Abstract: With the rapid advancement of image editing models and their widespread application across various domains, there is an increasingly urgent need to depl

LLM-Advisor: An LLM Advisor for Cost-efficient Path Planning across Multiple Terrains

arXiv:2503.01236v3 Announce Type: replace-cross Abstract: This paper addresses fixed-graph terrain-aware path refinement, in which a global planner is restricted to a predefined route space and may re

Measuring Cross-Task Behavioral Consistency in Language Model Agents

arXiv:2608.13598v1 Announce Type: new Abstract: Agent evaluation relies almost entirely on outcome metrics such as success rate, which capture whether an agent succeeds but not how consistently it beh

Measuring Fairness in Large Audio Language Models via Semantic-Aware Bias Estimation

arXiv:2608.13624v1 Announce Type: cross Abstract: Large Audio Language Models (LALMs) have seen increasing use for audio understanding tasks such as speech recognition and audio question answering, ra

PACE-Bench: Benchmarking Physics Adaptation via Code Evolution in Dynamic Environments

arXiv:2608.14441v1 Announce Type: new Abstract: Self-evolving agents improve future behavior from interaction experience, yet existing evaluations typically optimize under fixed execution conditions a

Recent Advances in Deep Learning-Based Drug-Target Binding Affinity Prediction

model-releasesarxiv-cs-lg

arXiv:2608.13797v1 Announce Type: new Abstract: Computational approaches to drug discovery involve multiple sub-problems, and among them, drug-target binding affinity prediction plays an important rol

A Browser-Native Digital Test Range for Benchmarking 4D Ocean-Glider Planning Algorithms

arXiv:2608.13511v1 Announce Type: new Abstract: Repeated in-situ evaluation of ocean-glider planners requires scarce vehicles, operators, deployment and recovery resources, and ocean conditions that c

safetyarxiv-cs-ro

Don't Want Your LLM to Recommend Nuclear Strike? Try Asking It in Japanese

arXiv:2608.12373v1 Announce Type: new Abstract: Large language models are increasingly used in strategic and advisory contexts, yet their safety alignment is typically evaluated in English only. We te

Applications

H-VAEP and H-xT: Valuing Offensive On-the-Ball Actions in Handball by Estimating Probabilities

arXiv:2608.12926v1 Announce Type: cross Abstract: Traditional player evaluation in professional handball relies on basic box-score metrics or heuristic indices, which fail to credit the multi-player b

applicationsarxiv-cs-ai

How Good are Foundation Models in Longitudinal MRI Disease Progression Reasoning?

model-releasesarxiv-cs-cv

arXiv:2608.13309v1 Announce Type: new Abstract: Magnetic Resonance Imaging (MRI) interpretation is fundamental to clinical decision-making, requiring radiologists to integrate multi-view anatomical pl

Large Language Models Pass the History Exam But Miss the <<History>>: A Polish High School Exit Exam Matura Benchmark

arXiv:2608.12343v1 Announce Type: new Abstract: AI chatbots are widely used by students as knowledge sources, yet LLM benchmarks rarely assess interpretative historical reasoning. We evaluate eight le

Learning Under Treatment-Induced Label Indeterminacy with Expert Annotations of Counterfactual Outcomes: A Case Study in Neurological Prognostication

arXiv:2608.12477v1 Announce Type: new Abstract: Clinical prediction models are often developed as if the outcome of interest were cleanly observed for every patient. This assumption fails when treatme

safetyarxiv-cs-lg

LigBench: A Unified and Human-Aligned Benchmark for LLM-based Research Idea Generation

arXiv:2608.13136v1 Announce Type: cross Abstract: With the rapid advancement of large language models (LLMs), research idea generation has attracted increasing attention. Existing approaches enable LL

NARU: A Benchmark for NARrative Evolution and Cultural Nuance Understanding in Japanese Extreme Long Video

arXiv:2608.13210v1 Announce Type: cross Abstract: Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting s

Applications

On Measuring Semantic Preservation in Legal Ontology Learning

arXiv:2608.12326v1 Announce Type: new Abstract: Ontology learning transforms unstructured text into structured representations for automated reasoning. Yet structuring information risks losing it, and

applicationsarxiv-cs-cl

SLAyiNG: A Diverse and Community-validated Dataset of Queer Slang

arXiv:2509.17449v2 Announce Type: replace Abstract: Queer vernacular is rarely studied in NLP, despite advancements in resources and evaluation for other sociolects and informal language. Because of t

safetyarxiv-cs-cl

StarEmbed: Benchmarking Time Series Foundation Models on Astronomical Observations of Variable Stars

arXiv:2510.06200v4 Announce Type: replace-cross Abstract: Current time series foundation model (TSFM) training corpora largely omit data with certain complexities like irregular temporal sampling. Ast

The 'Knowledge-Behavior Gap' in Cultural Taboo Safety of Large Language Models

arXiv:2608.12341v1 Announce Type: new Abstract: Cultural taboo safety is essential for deploying large language models (LLMs), as culturally insensitive outputs may cause offense or even social harm.

TsuGO: Probing Search Efficiency in LLM Reasoning via Go Life-and-Death Problems

arXiv:2608.13221v1 Announce Type: new Abstract: The evaluation of LLM reasoning is moving from final-answer accuracy to process-level assessment, yet existing methods still fail to capture how models

Vero: Can AI Agents Build Formally Verified Software Repositories?

arXiv:2608.13522v1 Announce Type: cross Abstract: AI agents are increasingly used for programming, but do not provide any guarantee on the correctness of generated code. Verified code generation, in w

A method for tissue-mask supported whole-body image registration in the UK Biobank

arXiv:2512.02702v3 Announce Type: replace Abstract: The UK Biobank is a large-scale study collecting imaging and non-imaging health data. Robust and accurate inter-subject image registration of these

safetyarxiv-cs-cv

Backtrader-Bench: Benchmarking LLM Agents on Algorithmic Trading with Self-Generated MCQs

arXiv:2608.11232v1 Announce Type: cross Abstract: Evaluating LLM coding agents in algorithmic trading is difficult because static benchmarks risk data contamination and numerical backtest outputs requ

Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents

arXiv:2608.11552v1 Announce Type: cross Abstract: Uncertainty quantification (UQ) methods for language models are typically evaluated on single-turn outputs, where uncertainty is attached to one gener

Applications

Trust Region Constrained Bayesian Optimization with Penalized Constraint Handling

arXiv:2603.24567v2 Announce Type: replace-cross Abstract: Constrained optimization in high-dimensional black-box settings is difficult due to expensive evaluations, the lack of gradient information, a

applicationsarxiv-cs-lg

Unmasking Toxic Mimicry in Medical Offline Reinforcement Learning for ICU Sepsis Management via Counterfactual Clinical Audits

arXiv:2608.11410v1 Announce Type: new Abstract: Offline reinforcement learning (RL) offers considerable promise for optimizing ICU treatment decisions, yet standard evaluation metrics Mean Squared Err

safetyarxiv-cs-lg

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

arXiv:2608.12246v1 Announce Type: cross Abstract: Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first intr

Who Would You Vote For? Auditing Political Alignment in LLMs: An Italian Case-Study

arXiv:2608.11649v1 Announce Type: new Abstract: As users increasingly turn to Large Language Models (LLMs) for information and advice on political matters, particularly during election periods, the po

safetyarxiv-cs-cl

A HamNoSys-Guided Dataset and Baselines for Fine-Grained Isolated Handshape Recognition in Sign Language

arXiv:2608.10588v1 Announce Type: cross Abstract: Purpose: Fine-grained handshape recognition supports computational sign-language transcription, recognition, and translation, but broad, phonetically

Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

arXiv:2608.11110v1 Announce Type: new Abstract: When a tool-using agent is given the same task in a different language, does it still take the same steps? Multilingual evaluation rarely asks: it compa

safetyarxiv-cs-cl

DashArena: Benchmarking LLMs on Interactive Analytic Dashboard Generation

arXiv:2608.10567v1 Announce Type: new Abstract: Analytic dashboards combine coordinated views and interactions for data exploration and decision-making. Recent models can generate them from data and n

Research

Decomposition-Induced Context-Memory Conflict: When Fact-Checking Pipelines Contradict Their Own Source Text

arXiv:2608.10627v1 Announce Type: new Abstract: Decompose-then-verify pipelines, including FActScore-style fact-checkers and long-form factuality evaluators, first split a passage into atomic claims b

researcharxiv-cs-cl

DuplexWorld: Can voice agents help you get through the day?

arXiv:2608.10716v1 Announce Type: cross Abstract: Speech-to-speech (S2S) voice agents are increasingly being incorporated into enterprise for customer care and as daily companions for consumers owing

Hardware

From Detection to Understanding: TAR and TAR-Bench for Multi-Task Traffic Anomaly Reasoning

arXiv:2608.10317v1 Announce Type: new Abstract: We present TAR (Traffic Anomaly Reasoning) and TAR-Bench datasets, resources for training and evaluating video-language models beyond anomaly detection.

hardwarearxiv-cs-cv

INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators

arXiv:2608.10492v1 Announce Type: new Abstract: Large Language Model (LLM)-based simulators often reproduce observable actions but fail to capture the underlying reasoning behind them. In education, w

Research

Patients With Personality: Realistic Patient Simulation through Controlled Diversity and Selective Disclosure

arXiv:2606.17441v2 Announce Type: replace-cross Abstract: Simulating realistic patient interactions is a key requirement to testing clinical applications of LLMs at scale without time-consuming and ex

researcharxiv-cs-ai

Status Association Does Not Reliably Predict Decision Leakage

arXiv:2608.10089v1 Announce Type: cross Abstract: Bias evaluations often move too quickly from evidence that a model encodes a social association to claims that the same association will alter consequ

TAF-MED: Multi-Turn Safety Refusal Collapse in LLMs Under Declared Self-Treatment Intent

arXiv:2608.10258v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly provide conversational health information that may influence treatment decisions, yet existing benchmarks do

The Multilingual Quantization Tax: Structural Collapse and Typological Fragility in Edge SLMs

arXiv:2608.09941v1 Announce Type: new Abstract: While 4-bit weight quantization is critical for deploying Small Language Models (SLMs) on edge devices, evaluations of the resulting performance degrada

VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

arXiv:2608.10875v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained re

360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

arXiv:2608.08814v1 Announce Type: cross Abstract: We present 360CityArena, a benchmark for evaluating the urban exploration capabilities of embodied agents within a photorealistic environment construc

A Unified Issue Resolution Benchmark for Requirement Clarification, Planning, and Code Generation for Coding Agents

arXiv:2608.09072v1 Announce Type: cross Abstract: Large language model-powered coding agents are increasingly used to modify existing code repositories, for example, by adding features or fixing bugs.

Cultivar: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

arXiv:2608.09766v1 Announce Type: cross Abstract: Multilingual translation benchmarks are typically sourced in English and translated into other languages, treating language pairs as the unit of evalu

Research

DS@GT ARC at Touche: Large Language Models for Retrieval-Augmented Debate

arXiv:2608.08143v1 Announce Type: cross Abstract: We extend the DS@GT ARC working-note submission to the Touche 2025 Retrieval-Augmented Debate task. The task has two subtasks: generating the next utt

researcharxiv-cs-cl

Lingjing: A Simulation Testbed for Multi-Agent Embodied Tasks in Open-Ended Cities

arXiv:2608.08045v1 Announce Type: new Abstract: Urban embodied intelligence requires coordination among heterogeneous agents (e.g., UAVs, ground robots, and autonomous vehicles) in dynamic cities. Sim

MetaSpace: Metamorphic Testing for Spatial Cognition in Embodied Agents

arXiv:2608.07533v1 Announce Type: new Abstract: An embodied agent is an intelligent entity that interacts with its environment through a physical body. Currently, the evaluation of embodied agents pri