Search: “evaluation”

Training with synthetic data for drone detection in thermal imagery

arXiv:2608.17799v1 Announce Type: cross Abstract: Ground-to-Air (G2A) drone detection in medium- and long-wave infrared (MWIR/LWIR) imagery is challenging due to reduced texture information, sensor no

19 Aug 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

A Human-Centred Approach to Benchmarking LLMs for Parenting Advice

DGX agent

arXiv:2608.14622v1 Announce Type: new Abstract: People are increasingly using large language models (LLMs) to seek advice, including for parenting. Parenting is a critical and socially sensitive domai

researcharxiv-cs-ai

18 Aug 2026

A validity-guided workflow for robust large language model research in psychology

arXiv:2507.04491v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are rapidly being integrated into psychological and behavioral research as research tools, evaluation targets, hu

Augmenting Text to Increase Translation Difficulty

arXiv:2608.15932v1 Announce Type: new Abstract: As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models

Clause Encounters of the Third Kind: Can LLMs Replace Language Teachers?

arXiv:2608.16286v1 Announce Type: new Abstract: While various organizations now actively encourage LLM use in classrooms, we still lack rigorous, systematic evaluations of how well these models actual

researcharxiv-cs-cl

Decoupling Parcellation from Classification: Systematic Benchmark of Fast Brain Segmentation Methods for Alzheimer's Disease Detection

arXiv:2608.16039v1 Announce Type: cross Abstract: Brain parcellation and classification are typically evaluated in isolation, yet downstream AD detection performance depends on their interaction. We d

I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering

arXiv:2604.03904v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often produce confident but incorrect answers, in part because standard evaluation incentives reward guessing ove

Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search

arXiv:2608.15669v1 Announce Type: new Abstract: Scientific discovery often involves optimising expensive-to-evaluate objectives over vast, structured, and open-ended hypothesis spaces, such as molecul

researcharxiv-cs-lg

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

arXiv:2608.14552v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated and used in medicine, but clinical usefulness depends on answer accuracy and whether confidence

LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

arXiv:2608.15242v1 Announce Type: new Abstract: When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the traject

Macroeconomic Forecasting with Large Language Models

arXiv:2407.00890v5 Announce Type: replace-cross Abstract: This paper presents a comparative analysis evaluating the accuracy of Large Language Models (LLMs) against traditional macro time series forec

applicationsarxiv-cs-cl

Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to …

model-releasesharrison-chase--x

Online evals are difficult to set up correctly. Even if you know what you want to look for, it's an optimization question of which model to choose, how to tune the prompt, and how to think about the c

Platform Adaptation Under Governance Interventions: Actor Best-Response Modeling and an External Public-Case Benchmark

arXiv:2608.15131v1 Announce Type: new Abstract: Digital platforms govern by changing rules: rankings, monetization thresholds, moderation standards, verification systems, disclosure requirements, appe

Local Ai

TRACE-CASH: Trial-History-Conditioned Reinforcement Learning for Adaptive Configuration Exploration in Time-Series CASH

arXiv:2608.16410v1 Announce Type: new Abstract: Combined algorithm selection and hyperparameter optimization (CASH) searches a conditional space in which the selected model determines which hyperparam

local-aiarxiv-cs-lg

TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents

arXiv:2608.16156v1 Announce Type: new Abstract: Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi

Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values

arXiv:2607.14345v4 Announce Type: replace-cross Abstract: People use language models for practical questions whose answers are difficult to verify. We show that models exhibit covert value leakage: th

Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents

arXiv:2608.16045v1 Announce Type: cross Abstract: LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded fil

What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

arXiv:2608.16370v1 Announce Type: new Abstract: Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by

When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding

arXiv:2608.16801v1 Announce Type: new Abstract: We study how teams of AI coding agents coordinate while solving programming tasks. Current evaluations usually report whether the agents complete the ta

Wiktionary as a Crowdsourced Lexicon for English Dialects

arXiv:2608.15641v1 Announce Type: new Abstract: This paper evaluates Wiktionary as an ethically crowdsourced lexicon for English dialects. We took a two-phase approach, providing an in-depth descripti

safetyarxiv-cs-cl

Exponential-Family Membership Inference: From LiRA and RMIA to BaVarIA

model-releasesarxiv-cs-lg

arXiv:2603.11799v2 Announce Type: replace Abstract: Membership inference attacks (MIAs) are becoming standard tools for auditing the privacy of machine learning models. The leading attacks -- LiRA (Ca

Geometric Filtering of LLM-Generated Samples for Few-Shot Text Classification

arXiv:2608.13866v1 Announce Type: cross Abstract: Large language models (LLMs) can generate synthetic training data for text classification, but the quality of generated samples is heterogeneous: some

researcharxiv-cs-cl

Investigating individual writing style as a contributor to gender gaps in science and technology

arXiv:2204.13805v5 Announce Type: replace-cross Abstract: Gender gaps in how scientific work is evaluated are well documented, but their sources remain debated. We ask whether an overlooked factor---t

safetyarxiv-cs-cl

On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

model-releasesarxiv-cs-cv

arXiv:2608.14262v1 Announce Type: new Abstract: Temporal vision-language models (TVLMs) offer a reusable, prompt-based interface for surgical video understanding, yet, their robustness under clinicall

Optimal Scheduling of Road Maintenance Jobs Considering Impact on Traffic Flows

arXiv:2608.14491v1 Announce Type: cross Abstract: Network-level maintenance planning requires repeated evaluations of equilibrium traffic flows under road capacity reductions. While equilibrium traffi

applicationsarxiv-cs-ai

Reflex: Enabling Fast and Predictive Vision-Language-Action Models for Reaction-Critical Manipulation

arXiv:2608.14379v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have recently achieved promising performance in robotic manipulation. However, existing benchmarks mainly evaluate

Local Ai

Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

arXiv:2608.14157v1 Announce Type: new Abstract: Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. W

local-aiarxiv-cs-ai

SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

arXiv:2608.13612v1 Announce Type: new Abstract: Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queri

TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection

arXiv:2608.13711v1 Announce Type: cross Abstract: Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss rates, yet reliable real-world deployment remains elusive. Thi

applicationsarxiv-cs-cv

JUST IN: OpenAI shuts down team responsible for assessing whether its AI models could pose 'catastrophic risks,' FT reports.

OpenAI has shut down the internal team that evaluated whether its AI models could pose “catastrophic risks,” a decision reported by the Financial Times on August 16, 2026. The move was announced via a

safetygary-marcus--x

16 Aug 2026

Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments

arXiv:2608.12368v1 Announce Type: new Abstract: Agreement with human judgments is a common proxy for evaluating the alignment of large language models (LLMs). Yet agreement in final labels does not sh

AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

arXiv:2608.13560v1 Announce Type: cross Abstract: Transforming multimodal sources into condensed and structured media outputs can be fundamentally conceptualized as a long-horizon agentic process cent

Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ

model-releasesarxiv-cs-cv

arXiv:2608.13441v1 Announce Type: new Abstract: Although multimodal large language models (MLLMs) have shown substantial potential in visual understanding and graphic code generation, editing scientif

Exemplar-based objective classification of gust-induced loads across multiple flight conditions

arXiv:2608.12448v1 Announce Type: new Abstract: Is it possible to find an objective classification criterion that organizes the complexity of gust-induced loads across many flight conditions? And one

researcharxiv-cs-lg

FIRE-VLA: Failure-Informed Self-Evolution for Vision-Language-Action Models in Autonomous Driving

model-releasesarxiv-cs-ro

arXiv:2608.13395v1 Announce Type: new Abstract: Reinforcement learning improves autonomous-driving vision-language-action (VLA) models by evaluating trajectories sampled from the current policy. Group

GeoCache: Training-Free Acceleration of Multi-View Texture Diffusion via Geometric Delta Transport

arXiv:2608.13255v1 Announce Type: cross Abstract: Geometry-conditioned multi-view diffusion enables high-quality 3D texture generation, but its repeated per-view denoiser evaluations introduce substan

HumanTracker: Towards Comprehensive and Human-Aligned Motion Tracking Benchmark

arXiv:2608.13555v1 Announce Type: cross Abstract: Humanoid motion tracking is central to teleoperation and whole-body imitation, yet evaluation often disagrees with what people perceive in videos. Kin

Jagged Judges: Epistemic Stability Under Silence, Pressure, and Persistence

arXiv:2608.12645v1 Announce Type: new Abstract: LLM judges have become central infrastructure for model evaluations, online grading, and reward modeling. Judges are typically validated by accuracy on

Large Language Models Persuade Without Planning Theory of Mind

arXiv:2602.17045v2 Announce Type: replace Abstract: A growing body of work attempts to evaluate the theory of mind (ToM) abilities of humans and large language models (LLMs) using static, non-interact

safetyarxiv-cs-cl

PatientAct: Theory-Grounded Mental Health Client Simulation

arXiv:2608.12750v1 Announce Type: cross Abstract: LLM-based simulated clients are increasingly used to train novice counselors, evaluate LLM therapists, and generate synthetic data. However, current s

Personalized Scorer Modeling: A Learning-Based Framework for Deriving Robust Sleep Stage Labels from Multiple Experts

arXiv:2608.12446v1 Announce Type: cross Abstract: Sleep stage classification is important for the diagnosis and management of sleep disorders, yet most automatic staging studies evaluate models agains

PhysMaster: Building an Autonomous AI Physicist for Theoretical and Computational Physics Research

arXiv:2512.19799v2 Announce Type: replace Abstract: Advances in LLM reasoning and tool use have enabled agentic science, yet frontier theoretical and computational physics remains challenging because

Privacy-Preserving RAG by Concealing Sensitive Information from External LLMs

arXiv:2608.12675v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) is widely used to improve the performance of Large Language Models (LLMs) in answering user queries. Existing priva

Robust Dempster-Shafer Evidence Fusion with Chaos-Conflict Measurement and Historical-Experience Weighting

arXiv:2608.13108v1 Announce Type: new Abstract: Multi-source evidence fusion under Dempster-Shafer theory faces two persistent challenges: existing conflict measures assess inter-evidence inconsistenc

Semantic Radiance Fields as Simulators for Spatial Reasoning in Real-World Scenes

arXiv:2608.13095v1 Announce Type: cross Abstract: Training and evaluating spatial reasoning in embodied agents requires diverse environments that are both geometrically faithful and semantically query

applicationsarxiv-cs-cv

Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

arXiv:2608.11947v1 Announce Type: cross Abstract: Multiple-choice benchmarks are widely used to evaluate large language models, but MCQ scores conflate knowledge with sensitivity to option order, whic

13 Aug 2026

Automated binary classification of hazelnut X-ray images: A deep-learning benchmark for quality assessment

model-releasesarxiv-cs-cv

arXiv:2608.11759v1 Announce Type: new Abstract: Non-destructive X-ray imaging can reveal internal hazelnut defects that are difficult to detect by external inspection alone; however, automated interpr

13 Aug 2026

Commonsense on Demand: Generating and Selectively Integrating Commonsense Knowledge for Natural Language Inference

arXiv:2507.15100v3 Announce Type: replace-cross Abstract: Natural Language Inference (NLI) determines whether a premise entails, contradicts, or is neutral with respect to a hypothesis. The task is of