Search: “evaluation”

arXiv:2510.12899v3 Announce Type: replace Abstract: Recently, several multi-turn dialogue benchmarks have been proposed to evaluate the conversational abilities of large language models (LLMs). As LLM

applicationsarxiv-cs-cl

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

DGX agent

arXiv:2608.25071v1 Announce Type: new Abstract: General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, maki

researcharxiv-cs-cl

27 Aug 2026

Longitudinal Robot Learning from Demonstration with Care Providers in a Home Environment

arXiv:2608.25196v1 Announce Type: new Abstract: Learning from demonstration (LfD) methods enable non-expert end users to teach robots novel skills without explicit programming. However most evaluation

researcharxiv-cs-ro

Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agent

arXiv:2601.04879v2 Announce Type: replace Abstract: Synthesizing informative commercial reports from massive and noisy web sources is critical for high-stakes business decisions. Although recent deep

Skill Issue: Are Skills Language-Invariant in LLMs?

arXiv:2608.25832v1 Announce Type: new Abstract: Large language models access knowledge inconsistently across languages, but to what extent do they differ in their skill sets when interacting with diff

ADVERSA: Measuring Multi-Turn Guardrail Degradation and Judge Reliability in Large Language Models

arXiv:2603.10068v2 Announce Type: replace-cross Abstract: Most adversarial evaluations of large language model (LLM) safety assess single prompts and report binary pass/fail outcomes, which fails to c

Diverse by Reasoning: Harnessing the Wisdom of LLM Crowds for Future Prediction

arXiv:2608.24001v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for future prediction, motivating the use of multiple models as a wisdom-of-the-crowd mechanism. Howe

DoublesEval: Diagnosing Multi-Agent Tactical Reasoning in Vision-Language Models via Professional Doubles Badminton

arXiv:2608.24439v1 Announce Type: new Abstract: Visual Language Models (VLMs) excel at describing visible scene content but struggle to reason about dynamic multi-agent interactions, where action sema

Agents

EgoErrorVQA: Assess Egocentric Comprehension Capabilities through Procedural Errors for Ego-Agentic AI

arXiv:2608.24134v1 Announce Type: new Abstract: The majority of our everyday activities are procedural and consist of sequences of interdependent steps. However, existing benchmarks for Visual Agents

agentsarxiv-cs-cv

Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail

arXiv:2608.23651v1 Announce Type: cross Abstract: Agent harnesses record a failed tool call and its error message in the transcript and ask the model to continue, on the assumption that the error is c

From Seeing to Acting: Smart Glasses as First-Person Intelligence Platforms

arXiv:2608.24877v1 Announce Type: new Abstract: Smart glasses are evolving from capture and display accessories into first-person intelligence platforms that connect human perception, persistent conte

researcharxiv-cs-cv

Housing Potential Common Data Model and City Digital Twin

arXiv:2605.05535v2 Announce Type: replace Abstract: The evaluation of housing potential requires consideration of a location from multiple perspectives, ranging from zoning and land use to population

Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition

arXiv:2602.19001v2 Announce Type: replace Abstract: As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people

MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs

model-releasesarxiv-cs-lg

arXiv:2602.06268v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly integrated into clinical workflows. However, pr

PatientHub: A Unified Framework for Patient Simulation

arXiv:2602.11684v2 Announce Type: replace-cross Abstract: As Large Language Models increasingly power role-playing applications, simulating patients has become a valuable tool for training counselors

Syn2RealTrack: Bridging the Gap Between Synthetic and Real-World Datasets for Online Multi-View Multi-Target Tracking

arXiv:2608.24130v1 Announce Type: cross Abstract: Multi-camera 3D perception systems for warehouse scenes are trained largely on synthetic data and evaluated on physically captured environments. The r

Taming foundation model with invariance-oriented pre-training for broad-spectrum EEG analysis across signal-level, brain-state, and brain-health tasks

arXiv:2608.24597v1 Announce Type: cross Abstract: Electroencephalography (EEG) is a widely used window into human brain function, but most EEG models remain tied to a one-dataset-one-model supervised

Towards Reliable AI-Based Histological Staining: A Systematic Study of Scaling and Uncertainty in Unpaired Generative Models

arXiv:2608.24626v1 Announce Type: new Abstract: Liver fibrosis, the principal predictor of long-term outcome in chronic liver disease, is staged from histological estimates of collagen content. Sirius

Applications

TW-LegalBench: Measuring Taiwanese Legal Understanding

arXiv:2606.18699v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown impressive capabilities across diverse tasks, yet their performance on jurisdiction-specific legal rea

applicationsarxiv-cs-ai

Applications

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

arXiv:2608.23978v1 Announce Type: new Abstract: Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a cen

applicationsarxiv-cs-ai

A Multidimensional Data-Driven Hybrid Transformer Framework for Non-invasive Continuous Blood Pressure Prediction

arXiv:2608.23276v1 Announce Type: new Abstract: Objective. To develop and evaluate a cuffless continuous blood pressure (BP) estimator using temporal physiological and demographic features. We propose

researcharxiv-cs-lg

Analyzing and Mitigating Cross-Lingual Degradation in Multilingual Medical VQA

arXiv:2608.22363v1 Announce Type: new Abstract: Medical visual question answering (VQA) is a crucial task in clinical AI, yet its evaluation has so far centered almost exclusively on English, limiting

Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs

model-releasesarxiv-cs-lg

arXiv:2608.21693v1 Announce Type: new Abstract: Mixture-of-Experts (MoE) LLMs scale model capacity efficiently through sparse activation, but their large expert parameter footprint, routing imbalance,

Do Large Language Models Perform Well on Comprehending Poetic Logic in Modern Chinese Poetry?

arXiv:2608.21827v1 Announce Type: new Abstract: Large Language Models (LLMs) have achieved significant progress across a wide range of natural language processing (NLP) tasks, yet their ability to und

Hardware

Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification

arXiv:2608.22506v1 Announce Type: new Abstract: Claim-level Uncertainty Quantification (UQ) aims to mitigate the lack of reliability of Large Language Models (LLMs) by evaluating the factuality of eac

hardwarearxiv-cs-cl

Mode-Dependent Rectification for Stable PPO Training

arXiv:2602.05619v2 Announce Type: replace-cross Abstract: Mode-dependent architectural components (layers that behave differently during training and evaluation, such as Batch Normalization or dropout

safetyarxiv-cs-ai

NetConfArena: An Executable Benchmark for LLM Agents in Closed-Loop Network Configuration

arXiv:2608.23179v1 Announce Type: cross Abstract: Large language model (LLM) agents are increasingly attractive for automating network configuration, yet their reliability and failure patterns are poo

PUMA: A Polish Benchmark for Culturally Grounded Multimodal Understanding

arXiv:2608.21853v1 Announce Type: new Abstract: Large language models are increasingly moving beyond text processing, adding support for other modalities such as images and audio. While text understan

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

arXiv:2608.22432v1 Announce Type: cross Abstract: Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmar

Stress Testing Unlearning Algorithms

model-releasesarxiv-cs-lg

arXiv:2608.22527v1 Announce Type: new Abstract: Recently, machine unlearning, the removal of specific training data influence from a model, has gained increasing attention. In large language models (L

C-Score: Beyond Accuracy for Robustness Assessment in Semi-Supervised Learning under Open-World Unlabeled Contamination

arXiv:2608.20667v1 Announce Type: cross Abstract: Pseudo-label-based semi-supervised learning has achieved strong performance due to its simplicity and scalability. However, it is typically developed

ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents

arXiv:2608.21101v1 Announce Type: cross Abstract: As large language model (LLM) agents move from conversation to executing code, reading local files, and orchestrating external tools, a single agent h

Detecting Functional Memorization in Code Language Models

arXiv:2606.12764v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may

researcharxiv-cs-cl

Agents

Edge-Based Agentic Retrieval-Augmented Generation for Autonomous FHWA Bridge Inspection Compliance

arXiv:2608.20372v1 Announce Type: cross Abstract: The Federal Highway Administration (FHWA) mandates that over 600,000 bridges in the United States be evaluated against the Recording and Coding Guide

agentsarxiv-cs-ai

Agents

ExpertIVS: Sociological Expert Driven Individual Value Simulation in Large Language Models

arXiv:2608.20355v1 Announce Type: cross Abstract: Large Language Model (LLM) agents have demonstrated considerable potential for social simulation, yet struggle to accurately model individual value sy

agentsarxiv-cs-ai

Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agenda

arXiv:2608.21107v1 Announce Type: new Abstract: Large Language Models (LLMs) are moving from code completion toward repository-scale agents that retrieve context, edit files, execute tools, and partic

When Looks Do Not Lie: Discourse Structure Guided In-Context Learning for Faithful Diagram Generation

arXiv:2601.20476v2 Announce Type: replace Abstract: GenAI is widespread in educational applications; however, it is known to generate content with intrinsic and extrinsic hallucination. We introduce a

researcharxiv-cs-cl

A knowledge-guided agentic framework for mitigating patient-context ambiguity in health queries

arXiv:2608.19875v1 Announce Type: cross Abstract: Patients often submit short, underspecified queries to healthcare chatbots that lack the patient-specific information needed to determine an appropria

safetyarxiv-cs-ai

A Virtual Member of a Community of Practice for the Society of Petroleum Engineers: From Prototype to Deployment

arXiv:2608.19199v1 Announce Type: cross Abstract: We describe the evolution of a virtual assistant, called ATHENA, designed to support the capture, retrieval, and dissemination of knowledge for member

Hear2Act: Benchmarking When Prosody Should Change What an Assistant Does

arXiv:2608.19515v1 Announce Type: new Abstract: Prosodic cues can convey task-relevant information that alters the trajectory and outcome of a task-oriented dialogue, even when the words themselves re

researcharxiv-cs-cl

MaliciousSkillBench: A Comprehensive Benchmark for Malicious Agent Skill Detection

arXiv:2608.19901v1 Announce Type: cross Abstract: Agent Skills extend LLM agents with reusable instruction packages that may also include scripts, resources, and service configuration. This creates a

Stopping and Routing LLM Judge Panels

arXiv:2608.19802v1 Announce Type: new Abstract: LLM evaluation pipelines often have many candidate judges: general LLM-as-a-judge prompts, reward models, safety classifiers, confidence variants, and t

safetyarxiv-cs-cl

Time-Series Retrieval for Grounding Multimodal Language Models in Remaining Useful Life

arXiv:2608.19218v1 Announce Type: cross Abstract: Large language models (LLMs) and agentic AI systems are increasingly being explored for domain-specific maintenance and prognostics tasks, raising the

Hardware

TorchDCM: A Unified PyTorch-Native Package for Discrete Choice Modeling

arXiv:2608.19231v1 Announce Type: cross Abstract: Estimating large and simulation-intensive discrete choice models (DCMs) requires repeated evaluation of utilities, probabilities, derivatives, and sim

hardwarearxiv-cs-lg

VGI-BENCH: Probing Visual Intelligence in Video Generation Models

arXiv:2608.19583v1 Announce Type: cross Abstract: Recent studies suggest that video generation models can exhibit certain forms of zero-shot visual reasoning through generated frames. Yet reliable eva

CamWorldQA: Perceptual Quality Assessment of Camera-Controlled World Video Generation

arXiv:2608.18710v1 Announce Type: new Abstract: Recent advances in generative video models have enabled camera-controlled world video generation, allowing models to synthesize videos under user-define

20 Aug 2026

Institutional Prestige as Geographic Bias in Large Language Models: Evidence from Three Factorial Experiments with Bootstrap Confidence Intervals

arXiv:2608.18107v1 Announce Type: cross Abstract: We investigate whether large language models (LLMs) systematically discriminate in candidate evaluations based on applicant name ethnicity and/or inst

safetyarxiv-cs-ai

20 Aug 2026

MemFuse: Multi-Source Memory Fusion from Fragmented Observations

arXiv:2608.18704v1 Announce Type: cross Abstract: Long-term memory is essential for agents that operate across extended interactions, yet existing memory systems and benchmarks predominantly focus on