Search: “evaluation”

Threat Modelling using Domain-Adapted Language Models: Empirical Evaluation and Insights

arXiv:2605.10808v1 Announce Type: cross Abstract: Large Language Models(LLMs) are increasingly explored for cybersecurity applications such as vulnerability detection. In the domain of threat modellin

researcharxiv-cs-ai

12 May 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Multi-Dimensional Evaluation of LLMs for Grammatical Error Correction

DGX agent

arXiv:2605.07635v1 Announce Type: new Abstract: Automated assistants for Grammatical Error Correction are now embedded in educational platforms serving millions of learners, yet three critical gaps re

researcharxiv-cs-cl

11 May 2026

Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

arXiv:2605.06474v2 Announce Type: replace-cross Abstract: We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one f

Robustness of Refugee-Matching Gains to Off-Policy Evaluation Choices

arXiv:2605.06686v1 Announce Type: new Abstract: Previous research has investigated the potential of refugee matching for boosting refugee outcomes, first considered by Bansak et al. (2018). This paper

safetyarxiv-cs-lg

Sources: the White House's Office of the National Cyber Director and Commerce Department's CAISI are fighting over which agency should lead AI model evaluations (Washington Post)

Washington Post: Sources: the White House's Office of the National Cyber Director and Commerce Department's CAISI are fighting over which agency should lead AI model evaluations — As the White House g

safetytechmeme

Unsolvability Ceiling in Multi-LLM Routing: An Empirical Study of Evaluation Artifacts

arXiv:2605.07395v1 Announce Type: cross Abstract: Efficient routing across multiple LLMs enables cost-quality tradeoffs by directing queries to the cheapest capable model. Prior work attributes routin

VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents

arXiv:2506.21582v5 Announce Type: replace-cross Abstract: Text analytics has traditionally required specialized knowledge in Natural Language Processing (NLP) or text analysis, which presents a barrie

agentsarxiv-cs-ai

Hardware

Zero-Shot Neural Network Evaluation with Sample-Wise Activation Patterns

arXiv:2605.07378v1 Announce Type: new Abstract: Zero-shot proxies, also known as training-free metrics, are widely adopted to reduce the computational overhead in neural network evaluation for scenari

hardwarearxiv-cs-lg

Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols

arXiv:2605.04727v1 Announce Type: new Abstract: Programming Knowledge Tracing (PKT) has recently advanced through hybrid approaches that integrate attention-based feature modeling for code representat

researcharxiv-cs-lg

7 May 2026

Unintended Negative Impacts of Promotional Language in Patent Evaluation

arXiv:2605.04926v1 Announce Type: new Abstract: Promotional language has been increasingly used to aid the communication of innovative ideas in science. Yet, less is known about its role in the contex

7 May 2026

Sim-to-Real Transfer and Robustness Evaluation of Reinforcement Learning Control with Integrated Perception on an ASV for Floating Waste Capture

arXiv:2605.02529v1 Announce Type: new Abstract: Autonomous surface vessels for floating-waste removal operate under varying hydrodynamics, external disturbances, and challenging water-surface percepti

agentsarxiv-cs-ro

5 May 2026

Banger paper from Meta FAIR. They introduce Autodata, an agentic data scientist that builds high-quality training and evaluation data autono…

Banger paper from Meta FAIR. They introduce Autodata, an agentic data scientist that builds high-quality training and evaluation data autonomously. The headline result: on a CS research QA task, an Ag

agentsdair-ai--x

4 May 2026

Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation

arXiv:2602.00665v3 Announce Type: replace Abstract: Customer-service question answering (QA) systems increasingly rely on conversational language understanding. While Large Language Models (LLMs) achi

4 May 2026

Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment

arXiv:2605.00022v1 Announce Type: new Abstract: The rapid proliferation of large audio models (LAMs) demands efficient approaches for model comparison, yet comprehensive benchmarks are costly. To fill

4 May 2026

Instruction Complexity Induces Positional Collapse in Adversarial LLM Evaluation

arXiv:2604.27249v1 Announce Type: cross Abstract: When instructed to underperform on multiple-choice evaluations, do language models engage with question content or fall back on positional shortcuts?

1 May 2026

Qualitative Evaluation of Language Model Rescoring in Automatic Speech Recognition

arXiv:2604.27533v1 Announce Type: new Abstract: Evaluating automatic speech recognition (ASR) systems is a classical but difficult and still open problem, which often boils down to focusing only on th

1 May 2026

From Intent to Evidence: A Categorical Approach for Structural Evaluation of Deep Research Agents

model-releasesarxiv-cs-lg

arXiv:2603.25342v2 Announce Type: replace Abstract: Deep Research Agents (DRAs) aim to answer complex questions by searching the web, checking evidence, and synthesizing conclusions across heterogeneo

30 Apr 2026

Learning-Based Automated Adversarial Red-Teaming for Robustness Evaluation of Large Language Models

arXiv:2512.20677v4 Announce Type: replace-cross Abstract: The increasing deployment of large language models (LLMs) in safety-critical applications raises fundamental challenges in systematically eval

29 Apr 2026

PSI-Bench: Towards Clinically Grounded and Interpretable Evaluation of Depression Patient Simulators

arXiv:2604.25840v1 Announce Type: new Abstract: Patient simulators are gaining traction in mental health training by providing scalable exposure to complex and sensitive patient interactions. Simulati

29 Apr 2026

VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluation

arXiv:2604.25235v1 Announce Type: cross Abstract: Vision-language models (VLMs) are increasingly used as automated judges for multimodal systems, yet their scores provide no indication of reliability.

29 Apr 2026

An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code

arXiv:2604.23361v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysi

An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness

arXiv:2604.23954v1 Announce Type: new Abstract: Artificial Intelligence and Machine Learning (AI/ML) models used in clinical settings are increasingly deployed to support clinical decision-making. How

Applications

Expert Evaluation of LLM's Open-Ended Legal Reasoning on the Japanese Bar Exam Writing Task

arXiv:2604.23730v1 Announce Type: new Abstract: Large language models (LLMs) have shown strong performance on legal benchmarks, including multiple-choice components of bar exams. However, their capaci

applicationsarxiv-cs-ai

Peer Identity Bias in Multi-Agent LLM Evaluation: An Empirical Study Using the TRUST Democratic Discourse Analysis Pipeline

arXiv:2604.22971v1 Announce Type: cross Abstract: The TRUST democratic discourse analysis pipeline exposes its large language model (LLM) components to peer model identity through multiple structural

Secure On-Premise Deployment of Open-Weights Large Language Models in Radiology: An Isolation-First Architecture with Prospective Pilot Evaluation

arXiv:2604.22768v1 Announce Type: cross Abstract: Purpose: To design, implement, evaluate, and report on the regulatory requirements of a self-hosted LLM infrastructure for radiology adhering to the p

StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems

arXiv:2604.22757v1 Announce Type: cross Abstract: We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasonin

OpenAI just released HealthBench Professional on Hugging Face A medical evaluation benchmark designed to improve AI assistants for clinician…

model-releasesclem-delangue--x

OpenAI just released HealthBench Professional on Hugging Face A medical evaluation benchmark designed to improve AI assistants for clinicians, featuring physician-curated conversations and rubric-base

25 Apr 2026

Rethinking Cross-Domain Evaluation for Face Forgery Detection with Semantic Fine-grained Alignment and Mixture-of-Experts

arXiv:2604.21478v1 Announce Type: new Abstract: Nowadays, visual data forgery detection plays an increasingly important role in social and economic security with the rapid development of generative mo

safetyarxiv-cs-cv

24 Apr 2026

A Network-Aware Evaluation of Distributed Energy Resource Control in Smart Distribution Systems

arXiv:2604.19715v1 Announce Type: new Abstract: Distribution networks with high penetration of Distributed Energy Resources (DERs) increasingly rely on communication networks to coordinate grid-intera

researcharxiv-cs-cv

22 Apr 2026

When Graph Structure Becomes a Liability: A Critical Re-Evaluation of Graph Neural Networks for Bitcoin Fraud Detection under Temporal Distribution Shift

arXiv:2604.19514v1 Announce Type: cross Abstract: The consensus that GCN, GraphSAGE, GAT, and EvolveGCN outperform feature-only baselines on the Elliptic Bitcoin Dataset is widely cited but has not be

researcharxiv-cs-ai

22 Apr 2026

Beyond Word Boundaries: A Hebrew Coreference Benchmark and an Evaluation Protocol for Morphologically Complex Text

arXiv:2604.17108v1 Announce Type: new Abstract: Coreference Resolution (CR) is a fundamental NLP task critical for long-form tasks as information extraction, summarization, and many business applicati

21 Apr 2026

HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

arXiv:2511.01066v3 Announce Type: replace Abstract: We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 tr

21 Apr 2026

Spotlights and Blindspots: Evaluation Machine-Generated Text Detection

arXiv:2604.16607v1 Announce Type: new Abstract: With the rise of generative language models, machine-generated text detection has become a critical challenge. A wide variety of models is available, bu

21 Apr 2026

CTSCAN: Evaluation Leakage in Chest CT Segmentation and a Reproducible Patient-Disjoint Benchmark

model-releasesarxiv-cs-cv

arXiv:2604.15561v1 Announce Type: cross Abstract: Reported chest CT segmentation performance can be strongly inflated when train and test partitions mix slices from the same study. We present CTSCAN,

20 Apr 2026

Enabling Predictive Maintenance in District Heating Substations: A Labelled Dataset and Fault Detection Evaluation Framework based on Service Data

arXiv:2511.14791v2 Announce Type: replace-cross Abstract: Early detection of faults in district heating substations is imperative to reduce return temperatures and enhance efficiency. However, progres

20 Apr 2026

Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation

arXiv:2601.07338v2 Announce Type: replace Abstract: Large Language Models (LLMs) have significantly advanced Machine Translation (MT), applying them to linguistically complex domains-such as Social Ne

agentsarxiv-cs-cl

17 Apr 2026

Blinded Multi-Rater Comparative Evaluation of a Large Language Model and Clinician-Authored Responses in CGM-Informed Diabetes Counseling

arXiv:2604.15124v1 Announce Type: new Abstract: Continuous glucose monitoring (CGM) is central to diabetes care, but explaining CGM patterns clearly and empathetically remains time-intensive. Evidence

17 Apr 2026

IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation

arXiv:2511.01014v3 Announce Type: replace Abstract: Instruction-following is a fundamental ability of Large Language Models (LLMs), requiring their generated outputs to follow multiple constraints imp

17 Apr 2026

A Dataset and Evaluation for Complex 4D Markerless Human Motion Capture

arXiv:2604.12765v1 Announce Type: new Abstract: Marker-based motion capture (MoCap) systems have long been the gold standard for accurate 4D human modeling, yet their reliance on specialized hardware

safetyarxiv-cs-cv

15 Apr 2026

Development, Evaluation, and Deployment of a Multi-Agent System for Thoracic Tumor Board

arXiv:2604.12161v1 Announce Type: new Abstract: Tumor boards are multidisciplinary conferences dedicated to producing actionable patient care recommendations with live review of primary radiology and

agentsarxiv-cs-ai

15 Apr 2026

FACT-E: Causality-Inspired Evaluation for Trustworthy Chain-of-Thought Reasoning

arXiv:2604.10693v1 Announce Type: new Abstract: Chain-of-Thought (CoT) prompting has improved LLM reasoning, but models often generate explanations that appear coherent while containing unfaithful int

FinTrace: Holistic Trajectory-Level Evaluation of LLM Tool Calling for Long-Horizon Financial Tasks

arXiv:2604.10015v1 Announce Type: new Abstract: Recent studies demonstrate that tool-calling capability enables large language models (LLMs) to interact with external environments for long-horizon fin

Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation

arXiv:2604.11662v1 Announce Type: new Abstract: Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation and hallucination detection, mot

Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMs

arXiv:2604.11120v1 Announce Type: new Abstract: Personality imbuing customizes LLM behavior, but safety evaluations almost always study prompt-based personas alone. We show this is incomplete: prompti

UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents

arXiv:2604.11557v1 Announce Type: new Abstract: Tool-use capability is a fundamental component of LLM agents, enabling them to interact with external systems through structured function calls. However

Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation

arXiv:2604.11611v1 Announce Type: new Abstract: To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information S

Local Ai

From Frames to Events: Rethinking Evaluation in Human-Centric Video Anomaly Detection

arXiv:2604.09327v1 Announce Type: new Abstract: Pose-based Video Anomaly Detection (VAD) has gained significant attention for its privacy-preserving nature and robustness to environmental variations.

local-aiarxiv-cs-cv

13 Apr 2026

Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition