Search: “evaluation”

Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA

arXiv:2608.22363v1 Announce Type: new Abstract: Medical visual question answering (VQA) is a crucial task in clinical AI, yet its evaluation has so far centered almost exclusively on English, limiting

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

DGX agent

arXiv:2608.21693v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance,

model-releasesarxiv-cs-lg

25 Aug 2026

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

model-releasesarxiv-cs-cl

arXiv:2608.21827v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks, yet their ability to und

Hardware

Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification

arXiv:2608.22506v1 Announce Type: new Abstract: Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of eac

hardwarearxiv-cs-cl

Mode-Dependent Rectification for Stable PPO Training

arXiv:2602.05619v2 Announce Type: replace-cross Abstract: Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

arXiv:2608.23179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poo

PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

model-releasesarxiv-cs-cl

arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understan

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

arXiv:2608.22432v1 Announce Type: cross Abstract: Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmar

Stress Testing Unlearning Algorithms

arXiv:2608.22527v1 Announce Type: new Abstract: Recently, machine unlearning, the removal of specific training data influence from a model, has gained increasing attention. In large language models (L

Agents

A skill is just an agent. So measure your changes.

A skill is just another AI agent: a prompt plus a harness that runs it. That means you can trace it, eval it, and prove a change made it better instead of shipping on vibes. Here's how we quantify ski

agentsarize-ai

C-Score: Beyond Accuracy for Robustness Assessment in Semi-Supervised Learning under Open-World Unlabeled Contamination

arXiv:2608.20667v1 Announce Type: cross Abstract: Pseudo-label-based semi-supervised learning has achieved strong performance due to its simplicity and scalability. However, it is typically developed

ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents

arXiv:2608.21101v1 Announce Type: cross Abstract: As large language model (LLM) agents move from conversation to executing code, reading local files, and orchestrating external tools, a single agent h

Detecting Functional Memorization in Code Language Models

arXiv:2606.12764v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may

researcharxiv-cs-cl

Agents

Edge-Based Agentic Retrieval-Augmented Generation for Autonomous FHWA Bridge Inspection Compliance

arXiv:2608.20372v1 Announce Type: cross Abstract: The Federal Highway Administration (FHWA) mandates that over 600,000 bridges in the United States be evaluated against the Recording and Coding Guide

agentsarxiv-cs-ai

Agents

ExpertIVS: Sociological Expert Driven Individual Value Simulation in Large Language Models

arXiv:2608.20355v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have demonstrated considerable potential for social simulation, yet struggle to accurately model individual value sy

agentsarxiv-cs-ai

Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda

arXiv:2608.21107v1 Announce Type: new Abstract: Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and partic

When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generation

arXiv:2601.20476v2 Announce Type: replace Abstract: GenAI is widespread in educational applications; however, it is known to generate content with intrinsic and extrinsic hallucination. We introduce a

researcharxiv-cs-cl

A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries

arXiv:2608.19875v1 Announce Type: cross Abstract: Patients often submit short, underspecified queries to healthcare chatbots that lack the patient-specific information needed to determine an appropria

A Virtual Member of a Community of Practice for the Society of Petroleum Engineers: From Prototype to Deployment

arXiv:2608.19199v1 Announce Type: cross Abstract: We describe the evolution of a virtual assistant, called ATHENA, designed to support the capture, retrieval, and dissemination of knowledge for member

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

arXiv:2608.19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves re

researcharxiv-cs-cl

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

arXiv:2608.19901v1 Announce Type: cross Abstract: Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a

Stopping and Routing LLM Judge Panels

arXiv:2608.19802v1 Announce Type: new Abstract: LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and t

safetyarxiv-cs-cl

Time-Series Retrieval for Grounding Multimodal Language Models in Remaining Useful Life

arXiv:2608.19218v1 Announce Type: cross Abstract: Large language models (LLMs) and agentic AI systems are increasingly being explored for domain-specific maintenance and prognostics tasks, raising the

Hardware

TorchDCM: A Unified PyTorch-Native Package for Discrete Choice Modeling

arXiv:2608.19231v1 Announce Type: cross Abstract: Estimating large and simulation-intensive discrete choice models (DCMs) requires repeated evaluation of utilities, probabilities, derivatives, and sim

hardwarearxiv-cs-lg

VGI-BENCH: Probing Visual Intelligence in Video Generation Models

arXiv:2608.19583v1 Announce Type: cross Abstract: Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable eva

CamWorldQA: Perceptual Quality Assessment of Camera-Controlled World Video Generation

model-releasesarxiv-cs-cv

arXiv:2608.18710v1 Announce Type: new Abstract: Recent advances in generative video models have enabled camera-controlled world video generation, allowing models to synthesize videos under user-define

Institutional Prestige as Geographic Bias in Large Language Models: Evidence from Three Factorial Experiments with Bootstrap Confidence Intervals

arXiv:2608.18107v1 Announce Type: cross Abstract: We investigate whether large language models (LLMs) systematically discriminate in candidate evaluations based on applicant name ethnicity and/or inst

MemFuse: Multi-Source Memory Fusion from Fragmented Observations

arXiv:2608.18704v1 Announce Type: cross Abstract: Long-term memory is essential for agents that operate across extended interactions, yet existing memory systems and benchmarks predominantly focus on

Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs

arXiv:2608.18131v1 Announce Type: new Abstract: Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages,

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

model-releasesarxiv-cs-ro

arXiv:2608.18701v1 Announce Type: new Abstract: Physical interaction quality is central to deformable-object manipulation, yet most benchmarks evaluate task success alone. A policy may complete the ta

What Makes Software Issue Resolution Tasks Difficult for Agents?

arXiv:2608.18280v1 Announce Type: cross Abstract: Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scor

Benchmarking Automated Security Patch Backporting: How Far Are We?

arXiv:2608.17671v1 Announce Type: cross Abstract: Automated security patch backporting is critical for mitigating N-day vulnerabilities. Recent tools report success rates above 80% on their respective

Chain-of-Experience for Continual LLM Improvement

model-releasesarxiv-cs-cl

arXiv:2608.18027v1 Announce Type: new Abstract: Humans continuously learn from experience, whereas conventional large language model (LLM) evaluations ignore the models' ability to improve through inf

Deep Learning for Cross-Border Electricity Price Forecasting: A Comparative Study

arXiv:2608.17091v1 Announce Type: new Abstract: While publicly available electricity market data presents a valuable resource for forecasting research, the field lacks established benchmark datasets f

Applications

Explainable AI-Powered Framework for Video-Based Skill Assessment in Cataract Surgery

arXiv:2608.17522v1 Announce Type: cross Abstract: Persistent shortages in the surgical workforce and inherent limitations of traditional training methods highlight the necessity of automated, data-dri

applicationsarxiv-cs-ai

HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety

arXiv:2608.17597v1 Announce Type: cross Abstract: Large language models are increasingly deployed through agent harnesses that manage tools, extensions, persistent state, permissions, and external act

How to make the most of your masked language model for protein engineering

arXiv:2603.10302v3 Announce Type: replace Abstract: A plethora of protein language models have been released in recent years. Yet comparatively little work has addressed how to best sample from them t

LIBERO-VIFO: Benchmarking the Capability and Safety of Visual Cue Following in Vision-Language-Action Models

model-releasesarxiv-cs-ro

arXiv:2608.17600v1 Announce Type: new Abstract: Visual cues are increasingly adopted to guide robot learning, but whether Vision-Language-Action (VLA) models can reliably follow authorized cues while

MITRE-SAGE: A Multi-Agent Cybersecurity Question-Answering Model

arXiv:2608.16921v2 Announce Type: cross Abstract: Effective cybersecurity operations require timely and accurate analysis of large-scale heterogeneous security information; however, analysts increasin

Nonlocal Transition Kernel for Efficient Learning of Restricted Boltzmann Machines

arXiv:2608.17450v1 Announce Type: cross Abstract: Learning restricted Boltzmann machines (RBMs) is computationally challenging because it requires expectations whose exact evaluation is generally intr

researcharxiv-cs-lg

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

model-releasesarxiv-cs-cv

arXiv:2608.17110v1 Announce Type: new Abstract: Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image cat

SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation

arXiv:2608.17426v1 Announce Type: cross Abstract: We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achie

Structural Plan-to-Model Conversion with Deterministic Geometry and Guarded Agentic Vision-Language Refinement

arXiv:2608.17237v1 Announce Type: cross Abstract: Converting structural framing plans into editable finite-element model drafts remains labor-intensive and prone to transcription error. Existing drawi

TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

arXiv:2608.17795v1 Announce Type: new Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inferen

safetyarxiv-cs-cl

Training with synthetic data for drone detection in thermal imagery

arXiv:2608.17799v1 Announce Type: cross Abstract: Ground-to-Air (G2A) drone detection in medium- and long-wave infrared (MWIR/LWIR) imagery is challenging due to reduced texture information, sensor no

A Human-Centred Approach to Benchmarking LLMs for Parenting Advice

arXiv:2608.14622v1 Announce Type: new Abstract: People are increasingly using large language models (LLMs) to seek advice, including for parenting. Parenting is a critical and socially sensitive domai

18 Aug 2026

A validity-guided workflow for robust large language model research in psychology

arXiv:2507.04491v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are rapidly being integrated into psychological and behavioral research as research tools, evaluation targets, hu

18 Aug 2026

Augmenting Text to Increase Translation Difficulty

arXiv:2608.15932v1 Announce Type: new Abstract: As state-of-the-art machine translation models saturate standard benchmarks, the field needs more challenging evaluations to distinguish between models