Search: “evaluation”

Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs

arXiv:2608.18131v1 Announce Type: new Abstract: Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages,

20 Aug 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

DGX agent

arXiv:2608.18701v1 Announce Type: new Abstract: Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the ta

model-releasesarxiv-cs-ro

20 Aug 2026

What Makes Software Issue Resolution Tasks Difficult for Agents?

arXiv:2608.18280v1 Announce Type: cross Abstract: Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scor

20 Aug 2026

Benchmarking Automated Security Patch Backporting: How Far Are We?

arXiv:2608.17671v1 Announce Type: cross Abstract: Automated security patch backporting is critical for mitigating N-day vulnerabilities. Recent tools report success rates above 80% on their respective

Chain-of-Experience for Continual LLM Improvement

model-releasesarxiv-cs-cl

arXiv:2608.18027v1 Announce Type: new Abstract: Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inf

Deep Learning for Cross-Border Electricity Price Forecasting: A Comparative Study

arXiv:2608.17091v1 Announce Type: new Abstract: While publicly available electricity market data presents a valuable resource for forecasting research, the field lacks established benchmark datasets f

Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery

arXiv:2608.17522v1 Announce Type: cross Abstract: Persistent shortages in the surgical workforce and inherent limitations of traditional training methods highlight the necessity of automated, data-dri

applicationsarxiv-cs-ai

HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

arXiv:2608.17597v1 Announce Type: cross Abstract: Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external act

How to make the most of your masked language model for protein engineering

arXiv:2603.10302v3 Announce Type: replace Abstract: A plethora of protein language models have been released in recent years. Yet comparatively little work has addressed how to best sample from them t

LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2608.17600v1 Announce Type: new Abstract: Visual cues are increasingly adopted to guide robot learning, but whether Vision-Language-Action (VLA) models can reliably follow authorized cues while

MITRE-SAGE: A Multi-Agent Cybersecurity Question-Answering Model

arXiv:2608.16921v2 Announce Type: cross Abstract: Effective cybersecurity operations require timely and accurate analysis of large-scale heterogeneous security information; however, analysts increasin

Nonlocal Transition Kernel for Efficient Learning of Restricted Boltzmann Machines

arXiv:2608.17450v1 Announce Type: cross Abstract: Learning restricted Boltzmann machines (RBMs) is computationally challenging because it requires expectations whose exact evaluation is generally intr

researcharxiv-cs-lg

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

model-releasesarxiv-cs-cv

arXiv:2608.17110v1 Announce Type: new Abstract: Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image cat

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

arXiv:2608.17426v1 Announce Type: cross Abstract: We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achie

Structural Plan-to-Model Conversion with Deterministic Geometry and Guarded Agentic Vision-Language Refinement

arXiv:2608.17237v1 Announce Type: cross Abstract: Converting structural framing plans into editable finite-element model drafts remains labor-intensive and prone to transcription error. Existing drawi

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

arXiv:2608.17795v1 Announce Type: new Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inferen

safetyarxiv-cs-cl

Training with synthetic data for drone detection in thermal imagery

arXiv:2608.17799v1 Announce Type: cross Abstract: Ground-to-Air (G2A) drone detection in medium- and long-wave infrared (MWIR/LWIR) imagery is challenging due to reduced texture information, sensor no

safetyarxiv-cs-ai

A Human-Centred Approach to Benchmarking LLMs for Parenting Advice

arXiv:2608.14622v1 Announce Type: new Abstract: People are increasingly using large language models (LLMs) to seek advice, including for parenting. Parenting is a critical and socially sensitive domai

A validity-guided workflow for robust large language model research in psychology

arXiv:2507.04491v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are rapidly being integrated into psychological and behavioral research as research tools, evaluation targets, hu

Augmenting Text to Increase Translation Difficulty

arXiv:2608.15932v1 Announce Type: new Abstract: As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models

Clause Encounters of the Third Kind: Can LLMs Replace Language Teachers?

arXiv:2608.16286v1 Announce Type: new Abstract: While various organizations now actively encourage LLM use in classrooms, we still lack rigorous, systematic evaluations of how well these models actual

researcharxiv-cs-cl

Decoupling Parcellation from Classification: Systematic Benchmark of Fast Brain Segmentation Methods for Alzheimer's Disease Detection

arXiv:2608.16039v1 Announce Type: cross Abstract: Brain parcellation and classification are typically evaluated in isolation, yet downstream AD detection performance depends on their interaction. We d

I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering

arXiv:2604.03904v2 Announce Type: replace-cross Abstract: Large language models (LLMs) often produce confident but incorrect answers, in part because standard evaluation incentives reward guessing ove

Large Discovery Models: Empirically-grounded Model-Based Open-Ended Search

arXiv:2608.15669v1 Announce Type: new Abstract: Scientific discovery often involves optimising expensive-to-evaluate objectives over vast, structured, and open-ended hypothesis spaces, such as molecul

researcharxiv-cs-lg

Large Language Models Show Metacognitive Sensitivity in Medical Reasoning

arXiv:2608.14552v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly evaluated and used in medicine, but clinical usefulness depends on answer accuracy and whether confidence

LongRCA Bench: Diagnosing Responsible Roles and Root Causes in Long-Horizon Agent Failures

arXiv:2608.15242v1 Announce Type: new Abstract: When a long-horizon agent execution fails, outcome-level evaluation reveals the unsuccessful result but not where the decisive error entered the traject

Macroeconomic Forecasting with Large Language Models

arXiv:2407.00890v5 Announce Type: replace-cross Abstract: This paper presents a comparative analysis evaluating the accuracy of Large Language Models (LLMs) against traditional macro time series forec

applicationsarxiv-cs-cl

Platform Adaptation Under Governance Interventions: Actor Best-Response Modeling and an External Public-Case Benchmark

arXiv:2608.15131v1 Announce Type: new Abstract: Digital platforms govern by changing rules: rankings, monetization thresholds, moderation standards, verification systems, disclosure requirements, appe

Local Ai

TRACE-CASH: Trial-History-Conditioned Reinforcement Learning for Adaptive Configuration Exploration in Time-Series CASH

arXiv:2608.16410v1 Announce Type: new Abstract: Combined algorithm selection and hyperparameter optimization (CASH) searches a conditional space in which the selected model determines which hyperparam

local-aiarxiv-cs-lg

TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents

arXiv:2608.16156v1 Announce Type: new Abstract: Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi

safetyarxiv-cs-ai

Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values

arXiv:2607.14345v4 Announce Type: replace-cross Abstract: People use language models for practical questions whose answers are difficult to verify. We show that models exhibit covert value leakage: th

Walk Before You Run: The Importance of Data Exploration for Data Analysis Agents

arXiv:2608.16045v1 Announce Type: cross Abstract: LLM-based data-analysis tools are increasingly used to help users analyze messy spreadsheets and workbooks, from answering questions over uploaded fil

What Does Context Compression Cost an Agent? Interaction Costs Unrevealed by Task-Completion Metrics

arXiv:2608.16370v1 Announce Type: new Abstract: Task completion is the standard metric for evaluating context compression, yet it is incomplete: compression can increase an agent's interaction cost by

When Agents Coordinate: Measuring Coordination in Multi-Agent AI Coding

arXiv:2608.16801v1 Announce Type: new Abstract: We study how teams of AI coding agents coordinate while solving programming tasks. Current evaluations usually report whether the agents complete the ta

safetyarxiv-cs-ai

Wiktionary as a Crowdsourced Lexicon for English Dialects

arXiv:2608.15641v1 Announce Type: new Abstract: This paper evaluates Wiktionary as an ethically crowdsourced lexicon for English dialects. We took a two-phase approach, providing an in-depth descripti

safetyarxiv-cs-cl

Exponential-Family Membership Inference: From LiRA and RMIA to BaVarIA

arXiv:2603.11799v2 Announce Type: replace Abstract: Membership inference attacks (MIAs) are becoming standard tools for auditing the privacy of machine learning models. The leading attacks -- LiRA (Ca

Geometric Filtering of LLM-Generated Samples for Few-Shot Text Classification

arXiv:2608.13866v1 Announce Type: cross Abstract: Large language models (LLMs) can generate synthetic training data for text classification, but the quality of generated samples is heterogeneous: some

researcharxiv-cs-cl

Investigating individual writing style as a contributor to gender gaps in science and technology

arXiv:2204.13805v5 Announce Type: replace-cross Abstract: Gender gaps in how scientific work is evaluated are well documented, but their sources remain debated. We ask whether an overlooked factor---t

safetyarxiv-cs-cl

On the Robustness of Temporal Vision-Language Models for Surgical Endoscopy Videos

model-releasesarxiv-cs-cv

arXiv:2608.14262v1 Announce Type: new Abstract: Temporal vision-language models (TVLMs) offer a reusable, prompt-based interface for surgical video understanding, yet, their robustness under clinicall

Optimal Scheduling of Road Maintenance Jobs Considering Impact on Traffic Flows

arXiv:2608.14491v1 Announce Type: cross Abstract: Network-level maintenance planning requires repeated evaluations of equilibrium traffic flows under road capacity reductions. While equilibrium traffi

applicationsarxiv-cs-ai

Reflex: Enabling Fast and Predictive Vision-Language-Action Models for Reaction-Critical Manipulation

arXiv:2608.14379v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have recently achieved promising performance in robotic manipulation. However, existing benchmarks mainly evaluate

Local Ai

Removing Temporal Note Redundancy Improves Multimodal Reinforcement Learning for Medicine

arXiv:2608.14157v1 Announce Type: new Abstract: Mechanical ventilation is a critical life-support intervention, requiring dynamic adjustments to ventilator settings as a patient's condition evolves. W

local-aiarxiv-cs-ai

SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data

arXiv:2608.13612v1 Announce Type: new Abstract: Natural-language interfaces to enterprise data must translate underspecified requests into governed, executable behavior while controlling invalid queri

TRUE-Colon: Exposing a Consistent Transfer Asymmetry in Real-Time Polyp Detection

arXiv:2608.13711v1 Announce Type: cross Abstract: Computer-aided detection (CADe) systems for colonoscopy promise to reduce clinical miss rates, yet reliable real-world deployment remains elusive. Thi

applicationsarxiv-cs-cv

Agreement Is Not Alignment: Divergent Moral Grounds in Human and LLM Ethical Judgments

arXiv:2608.12368v1 Announce Type: new Abstract: Agreement with human judgments is a common proxy for evaluating the alignment of large language models (LLMs). Yet agreement in final labels does not sh

AutoDesign: Meta-Harness Optimization for Long-Horizon Agentic Design

arXiv:2608.13560v1 Announce Type: cross Abstract: Transforming multimodal sources into condensed and structured media outputs can be fundamentally conceptualized as a long-horizon agentic process cent

Edit2TikZ: A Comprehensive and Challenging Benchmark for Scientific Figure Editing with TikZ

model-releasesarxiv-cs-cv

arXiv:2608.13441v1 Announce Type: new Abstract: Although multimodal large language models (MLLMs) have shown substantial potential in visual understanding and graphic code generation, editing scientif

Exemplar-based objective classification of gust-induced loads across multiple flight conditions

arXiv:2608.12448v1 Announce Type: new Abstract: Is it possible to find an objective classification criterion that organizes the complexity of gust-induced loads across many flight conditions? And one

researcharxiv-cs-lg