Search: “evaluation”

HarmReduction: Benchmarking LLMs in Harm Reduction Information Provision to Support People Who Use Drugs

arXiv:2507.21815v2 Announce Type: replace Abstract: Millions of individuals' well-being are challenged by the harms of substance use. Harm reduction as a public health strategy provides non-judgementa

3 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

PragAlign: Feedback-Guided Pragmatic Alignment for Controlled Synthetic Dialogue Generation

DGX agent

arXiv:2609.02480v1 Announce Type: new Abstract: Synthetic dialogue generation can support research in privacy-restricted service settings, but generated conversations must preserve communicative inten

safetyarxiv-cs-cl

3 Sept 2026

UTP-Bench: Uncertainty-aware Travel Planning Benchmark

arXiv:2609.02421v1 Announce Type: new Abstract: Large Language Models (LLMs) have recently demonstrated strong capabilities in automated travel itinerary generation. However, real- world travel planni

3 Sept 2026

A Benchmark for Vehicle Attribute Classification in Cross-Domain Surveillance Scenarios

arXiv:2609.01584v1 Announce Type: new Abstract: Vehicle attribute analysis is a key component of Intelligent Transportation Systems (ITS), supporting applications such as vehicle identification, traff

Agentic Empirical Asset Pricing: Methodological Foundations

arXiv:2609.00731v1 Announce Type: new Abstract: Recent advances in LLM agents enable a new paradigm for asset pricing, which we call Agentic Empirical Asset Pricing (AEAP): systems that autonomously c

agentsarxiv-cs-ai

Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

arXiv:2609.00482v1 Announce Type: new Abstract: Small leaderboard gaps are often interpreted as evidence that one language model is better than another, but their sign may depend on which benchmark it

Conditional Flow Matching for ML-Based Inverse Design Problems

arXiv:2609.00863v1 Announce Type: new Abstract: Engineering inverse design is often limited by the high computational cost of iterative solvers for optimization problems constrained by partial differe

researcharxiv-cs-lg

CopyShield: A Cross-Level Benchmark of Copyright Defenses in LLMs

model-releasesarxiv-cs-lg

arXiv:2609.01161v1 Announce Type: new Abstract: Large language models can reproduce memorized text verbatim, yet copyright defenses are usually evaluated under incompatible protocols. We introduce Cop

GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments

arXiv:2609.00048v1 Announce Type: cross Abstract: GUI world models are increasingly evaluated as one-step next-screen predictors, yet their intended use is often as multi-step environments for GUI age

Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulations

arXiv:2609.01257v1 Announce Type: new Abstract: As LLM-based human simulators are increasingly used for policy, evaluation, and training, they must faithfully reproduce real behavioral patterns. While

safetyarxiv-cs-ai

NSIDDx: A Design Framework for Neuro-Symbolic, Practitioner-First Differential Diagnosis in Low-Resource Settings

arXiv:2609.00256v1 Announce Type: new Abstract: LLM-based diagnostic systems achieve high semantic accuracy on benchmarks, but open-ended evaluation on clinically uncommon presentations reveals a syst

agentsarxiv-cs-cl

Optimizing Byzantine Node Placement in Decentralized Federated Learning

arXiv:2609.01495v1 Announce Type: cross Abstract: Security evaluations of decentralized federated learning (DFL) typically focus on how Byzantine participants behave, while largely overlooking which p

ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues

arXiv:2606.18237v2 Announce Type: replace-cross Abstract: Reproducing research results from papers and released code is central to scientific progress. Existing works have introduced benchmarks to eva

Scale-based Approach for Active Wildfire Segmentation on Satellite Imagery

arXiv:2609.01392v1 Announce Type: new Abstract: Active wildfire mapping from satellite imagery is challenging due to the sparse and highly imbalanced nature of fire pixels, especially in early-stage o

researcharxiv-cs-cv

Local Ai

The Answer Is Not the Argument

arXiv:2609.00264v1 Announce Type: new Abstract: Chain-of-thought monitoring is proposed for AI oversight, yet evaluations often provide monitors with a trusted reference answer. We ask whether answer

local-aiarxiv-cs-ai

Toward Workflow-Aware Benchmarking for Healthcare NLP Agents

arXiv:2609.00296v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging,

agentsarxiv-cs-cl

Visual Framing for News Stance Detection via Image Generation

arXiv:2609.00685v1 Announce Type: cross Abstract: Article-level news stance detection aims to identify the perspective of news articles toward social issues. Despite advances in stance detection and i

AlgoWorlds: Benchmarking Tool Use for Global Optimization in Algorithmic Worlds

arXiv:2608.29397v1 Announce Type: new Abstract: Tool-use benchmarks generally evaluate whether an agent completes a workflow using appropriate tools and valid arguments. However, feasibility alone is

Arkios: An Open Bilingual English-Nepali Language Model Trained From Scratch, with a Devanagari-Aware Tokenizer

arXiv:2608.30092v1 Announce Type: cross Abstract: We present Arkios, a 1.04B-parameter dense transformer pretrained from scratch on 150B tokens of bilingual English-Nepali text, using a custom single-

Beyond Helpfulness: A Teaching-over-Solving Diagnostic for Measuring Educational Impact in LLM Tutors

arXiv:2606.16206v2 Announce Type: replace Abstract: Large language models are increasingly proposed as educational tutors, yet stronger task-solving ability does not necessarily imply stronger learnin

DERELAB: Probing Defeasible Reasoning and Confirmation Bias in LLMs with a Generative Benchmark

arXiv:2608.30413v1 Announce Type: new Abstract: Defeasible reasoning is a type of reasoning where inferences are drawn from plausible current evidence, but can be retracted upon the introduction of ne

Error Detection for PET/CT Radiology Reports: Domain-Specific vs Large Language Models

arXiv:2608.30021v1 Announce Type: cross Abstract: Errors in radiology reports can adversely affect patient treatment, yet automated report quality assurance remains challenging because errors are ofte

From Analytics to Tumor Boards: An Evidence-Linked Multi-Agent Workflow for Oncology Feature Extraction

arXiv:2608.28974v1 Announce Type: new Abstract: Clinically relevant oncology information is distributed across heterogeneous, longitudinal documentation, creating substantial abstraction burden and re

agentsarxiv-cs-ai

Generating Clinical Vignettes that Preserve Cognitive Formulations

arXiv:2608.29995v1 Announce Type: cross Abstract: Large language models can generate fluent clinical case vignettes, but fluency alone does not ensure fidelity to a specifiable clinical structure. We

GPAgentBench-2K: Benchmarking Large Language Model Agents in Complex Clinical Action Space

arXiv:2608.30188v1 Announce Type: new Abstract: Large Language Models (LLMs) show great potential as clinical agents, yet existing benchmarks reduce clinical workflows to static predictions or unconst

Learning Compositional Spatio-Temporal Video Grounding with Synthetic Curriculum

arXiv:2608.30584v1 Announce Type: new Abstract: Despite the impressive progress of recent MLLMs on spatio-temporal video grounding (STVG), existing evaluations and training data focus primarily on sim

Motus2: A Self-Evolving General World Model for Dexterous Manipulation

arXiv:2608.30237v1 Announce Type: cross Abstract: General embodied agents should perceive, predict, act, evaluate, and improve within a unified system. World models have shown great promise in buildin

safetyarxiv-cs-ai

N_0-Foundation: Towards the Age of Tactile Intelligence

arXiv:2608.29601v1 Announce Type: cross Abstract: We present N_0-Foundation, a paradigm for tactile-enabled embodied manipulation, which integrates tactile sensing hardware, large-scale multimodal dat

Redesigning and Auditing Deep Research Writing for Faithful Reports

arXiv:2608.28643v1 Announce Type: cross Abstract: Rubric-based evaluations of deep-research (DR) systems often obscure fine-grained factual failures in generated reports. We introduce CLAIMPROBE, a cl

SurgSkill-Bench: A Benchmark for Multimodal Surgical Skill Assessment

arXiv:2608.30872v1 Announce Type: new Abstract: Objective assessment of surgical technical skill is important for surgical training and structured feedback, but current workflows remain dependent on l

Applications

Uncertainty-Aware Trajectory Forecasting from Imperfect Tracking

arXiv:2608.30899v1 Announce Type: new Abstract: Most trajectory forecasting models are trained on clean annotated histories, and are often evaluated under the same idealized assumption, although pract

applicationsarxiv-cs-cv

VIBE: Video Instruction-aligned Background music gEneration

arXiv:2608.30125v1 Announce Type: cross Abstract: Current video-to-music (V2M) models lack semantic control and fail to penalize instruction violations, largely due to their reliance on reconstruction

safetyarxiv-cs-ai

Local Ai

XQDT: eXplainable and Quantitative Data-Text Alignment Metric with Feedback Signals

arXiv:2608.29948v1 Announce Type: new Abstract: Evaluating data-text alignment remains challenging: existing metrics often provide limited explanations for the scores, while prompt-based LLM-as-Judge

local-aiarxiv-cs-cl

Denoising-Aware Temporal Point Cloud Completion for 3D Crop Architecture Recovery and Phenotypic Trait Extraction

arXiv:2608.28343v1 Announce Type: new Abstract: High-throughput phenotyping depends on accurate 3D reconstruction of plants across growth stages, yet the development and evaluation of temporal complet

31 Aug 2026

Explainable Diabetic Retinopathy Classification Using Vision Foundation Models

arXiv:2608.28207v1 Announce Type: new Abstract: Diabetic retinopathy (DR) is a major cause of preventable blindness, creating a need for accurate and trustworthy automated screening. This study invest

31 Aug 2026

From weeks to minutes: The new agentic era of data pipelines

model-releasesgoogle-cloud-ai

Data pipelines are the backbone of the modern enterprise, yet a barrier to entry exists for orchestrating them, making this critical capability unavailable to many data professionals. Following our an

31 Aug 2026

A Multi-Framework Comparison of Outline Stages in Long-Form Generation with LLMs

arXiv:2608.26177v1 Announce Type: cross Abstract: Long-form generation exposes fundamental limitations of large language models. Even 70B-parameter models exhibit length collapse at 16k-token outputs,

AfriSwitch: A Benchmark for In-the-Wild African Code-Switched Speech Recognition

arXiv:2608.26434v1 Announce Type: new Abstract: Code-switching is pervasive in bilingual African conversation, yet most ASR systems assume monolingual input and are evaluated on curated monolingual be

Barrier Function Conformal Safety Clearance Certification with CVaR for Driving Trajectory Selection

arXiv:2608.26533v1 Announce Type: cross Abstract: Autonomous driving motion planners generate and select candidate trajectories while accounting for interactions with surrounding agents. However, thes

safetyarxiv-cs-ro

Beyond Classification: Task-Dependent Learnability under Privacy-Motivated Image Transformations

arXiv:2608.27066v1 Announce Type: cross Abstract: Privacy-Enhancing Technologies (PETs) in computer vision often rely on noise or image perturbations to protect visual data while securely processing i

CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answering

arXiv:2608.26114v1 Announce Type: new Abstract: Calculation-intensive financial question answering requires exact reasoning over structured rates, temporal conditions, numerical formulas, and rule-bas

Applications

Exploring the Role of LLMs in HPC Programming: A Survey

arXiv:2608.26110v1 Announce Type: cross Abstract: Large Language Models (LLMs) are emerging as promising assistants in High-Performance Computing (HPC), where programming remains complex and expertise

applicationsarxiv-cs-ai

GSM8K-V: Can Vision Language Models Solve Grade School Math Word Problems in Visual Contexts

arXiv:2509.25160v2 Announce Type: replace-cross Abstract: Mathematical reasoning is a key capability for vision-language models (VLMs), yet current benchmarks mainly evaluate text-based or explicitly

How Unlikely Is 'Unlikely'? Assessing Verbal Probability Perception Across Large Language Models

arXiv:2608.26327v1 Announce Type: cross Abstract: Large language models increasingly produce and interpret verbal probability expressions, yet whether these expressions carry consistent meaning across

Rethinking Image Processing for the Age of AI: A Problem-First Framework for Scientific Progress

arXiv:2608.26833v1 Announce Type: new Abstract: Modern AI has greatly expanded the capabilities of image processing. However, the ready availability of powerful models, public datasets, and benchmark

Translation Asymmetry in LLMs as a Data Augmentation Factor: A Case Study for 6 Romansh Language Varieties

arXiv:2603.25489v2 Announce Type: replace Abstract: Recent strategies for low-resource machine translation rely on LLMs to generate synthetic data based on text in higher-resource languages. We revisi

VIPER: An Expert-Curated Benchmark for Vision-Language Models in Veterinary Pathology

arXiv:2608.26382v1 Announce Type: new Abstract: Pathology vision-language models are advancing rapidly, yet existing benchmarks remain focused on human tissue, particularly oncology, leaving non-human

LibriBrain100: One Hundred Hours of Broad and Deep MEG Data for Neural Speech Decoding at Scale

arXiv:2608.25204v1 Announce Type: cross Abstract: We introduce LibriBrain100, a large-scale MEG dataset for speech decoding designed from the ground up for reproducible, standardised evaluation. Libri