Search: “evaluation”

Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects

arXiv:2510.07890v3 Announce Type: replace Abstract: Research on cross-dialectal transfer from a standard to a non-standard dialect variety has typically focused on text data. However, dialects are pri

applicationsarxiv-cs-cl

17 Apr 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research

DGX agent

arXiv:2511.07412v2 Announce Type: replace Abstract: Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, saf

safetyarxiv-cs-cv

17 Apr 2026

VoxSafeBench: Not Just What Is Said, but Who, How, and Where

arXiv:2604.14548v1 Announce Type: cross Abstract: As speech language models (SLMs) transition from personal devices into shared, multi-user environments, their responses must account for far more than

safetyarxiv-cs-lg

17 Apr 2026

EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development

model-releasesarxiv-cs-ro

arXiv:2604.13800v1 Announce Type: new Abstract: Embodied AI research is increasingly moving beyond single-task, single-environment policy learning toward multi-task, multi-scene, and multi-model setti

Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints

arXiv:2604.13371v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly described as possessing strong reasoning capabilities, supported by high performance on mathematical, logi

researcharxiv-cs-cl

Giving Voice to the Constitution: Low-Resource Text-to-Speech for Quechua and Spanish Using a Bilingual Legal Corpus

arXiv:2604.13288v1 Announce Type: new Abstract: We present a unified pipeline for synthesizing high-quality Quechua and Spanish speech for the Peruvian Constitution using three state-of-the-art text-t

applicationsarxiv-cs-cl

HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

model-releasesarxiv-cs-lg

arXiv:2604.13954v1 Announce Type: new Abstract: Existing agent-safety evaluation has focused mainly on externally induced risks. Yet agents may still enter unsafe trajectories under benign conditions.

Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

model-releasesarxiv-cs-cl

arXiv:2510.14232v2 Announce Type: replace-cross Abstract: Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models

Agents

very excited for this one! a year ago, most of what was being traced to LangSmith was 'LLM apps'. now everything is becoming an agent. with …

very excited for this one! a year ago, most of what was being traced to LangSmith was 'LLM apps'. now everything is becoming an agent. with that shift, it's getting harder to know what your software i

agentsharrison-chase--x

Local Ai

Which video model currently has the best face likeness for LoRA training?

local-air-stablediffusion

This r/StableDiffusion thread discusses community comparisons of video generation models (such as those built on Stable Diffusion or FLUX architectures) evaluated specifically for how well they preser

BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design

arXiv:2604.12898v1 Announce Type: new Abstract: Large Language Model-based Hyper Heuristic (LHH) has recently emerged as an efficient way for automatic heuristic design. However, most existing LHHs ju

GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics

arXiv:2604.02830v2 Announce Type: replace Abstract: Detecting whether a model's internal knowledge is sufficient to correctly answer a given question is a fundamental challenge in deploying responsibl

applicationsarxiv-cs-cl

Human-Inspired Context-Selective Multimodal Memory for Social Robots

arXiv:2604.12081v1 Announce Type: new Abstract: Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the con

Industry

More people should work on harnesses for open and local models!

Hugging Face co-founder and CEO Clément Delangue advocates for more developers and researchers to focus on building evaluation harnesses and testing frameworks specifically designed for open-source an

industryclem-delangue--x

Red Teaming Large Reasoning Models

arXiv:2512.00412v4 Announce Type: replace-cross Abstract: Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, offering enhanced transparency and logical

A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness

arXiv:2604.10853v1 Announce Type: new Abstract: Task-oriented evaluation of knowledge graph (KG) quality increasingly asks whether an ontology-based representation can answer the competency questions

A Dual-Positive Monotone Parameterization for Multi-Segment Bids and a Validity Assessment Framework for Reinforcement Learning Agent-based Simulation of Electricity Markets

arXiv:2604.10252v1 Announce Type: new Abstract: Reinforcement learning agent-based simulation (RL-ABS) has become an important tool for electricity market mechanism analysis and evaluation. In the mod

safetyarxiv-cs-ai

ACCIDENT: A Benchmark Dataset for Vehicle Accident Detection from Traffic Surveillance Videos

arXiv:2604.09819v1 Announce Type: cross Abstract: We introduce ACCIDENT, a benchmark dataset for traffic accident detection in CCTV footage, designed to evaluate models in supervised (IID and OOD) and

AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

arXiv:2601.11044v3 Announce Type: replace Abstract: Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. Howev

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

arXiv:2604.10547v1 Announce Type: new Abstract: We introduce Agent^2 RL-Bench, a benchmark for evaluating agentic RL post-training -- whether LLM agents can autonomously design, implement, and run com

Agents

BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving

arXiv:2604.10856v1 Announce Type: cross Abstract: Open-loop (OL) to closed-loop (CL) gap (OL-CL gap) exists when OL-pretrained policies scoring high in OL evaluations fail to transfer effectively in c

agentsarxiv-cs-ai

ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care

arXiv:2509.00891v3 Announce Type: replace Abstract: Real-world adoption of closed-loop insulin delivery systems (CLIDS) in type 1 diabetes remains low, driven not by technical failure, but by diverse

Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Using a Large Language Model

arXiv:2604.11287v1 Announce Type: new Abstract: Background: Large language models (LLMs) have been explored as tools for generating personalized exercise prescriptions, yet the consistency of outputs

Decision-Theoretic Safety Assessment of Persona-Driven Multi-Agent Systems in O-RAN

arXiv:2604.09682v1 Announce Type: cross Abstract: Autonomous network management in Open Radio Access Networks requires intelligent decision making across conflicting objectives, yet existing LLM based

safetyarxiv-cs-ai

Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky

arXiv:2507.03336v4 Announce Type: replace Abstract: Large language models (LLMs) are increasingly tasked with invoking enterprise APIs, yet they routinely falter when near-duplicate tools vie for the

Improving understanding and trust in AI: How users benefit from interval-based counterfactual explanations

arXiv:2604.09573v1 Announce Type: cross Abstract: Experimental user studies evaluating the effectiveness of different subtypes of post-hoc explanations for black-box models are largely nonexistent. Th

researcharxiv-cs-lg

ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

arXiv:2604.03765v2 Announce Type: replace Abstract: Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing

LLM-as-Judge on a Budget

arXiv:2602.15481v2 Announce Type: replace Abstract: LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pair

safetyarxiv-cs-lg

MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark

arXiv:2604.10755v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely unte

NameBERT: Scaling Name-Based Nationality Classification with LLM-Augmented Open Academic Data

arXiv:2604.10401v1 Announce Type: new Abstract: Inferring nationality from personal names is a critical capability for equity and bias monitoring, personalization, and a valuable tool in biomedical an

safetyarxiv-cs-cl

Property-Preserving Hashing for ell_1-Distance Predicates: Applications to Countering Adversarial Input Attacks

arXiv:2504.16355v2 Announce Type: replace-cross Abstract: Perceptual hashing is used to detect whether an input image is similar to a reference image with a variety of security applications. Recently,

researcharxiv-cs-lg

Quantization Robustness to Input Degradations for Object Detection

arXiv:2508.19600v2 Announce Type: replace Abstract: Post-training quantization (PTQ) is crucial for deploying efficient object detection models, like YOLO, on resource-constrained devices. However, th

applicationsarxiv-cs-cv

Should We be Pedantic About Reasoning Errors in Machine Translation?

arXiv:2604.09890v1 Announce Type: cross Abstract: Across multiple language pairings (English o {Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese}), we find reasoning errors in translation.

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

arXiv:2604.09557v1 Announce Type: cross Abstract: Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optim

SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models

arXiv:2604.09553v1 Announce Type: cross Abstract: LLM development has aroused great interest in Sequential Recommendation (SR) applications. However, comprehensive evaluation of SR models remains lack

TrajOnco: a multi-agent framework for temporal reasoning over longitudinal EHR for multi-cancer early detection

arXiv:2604.10386v1 Announce Type: new Abstract: Accurate estimation of cancer risk from longitudinal electronic health records (EHRs) could support earlier detection and improved care, but modeling su

TS-Haystack: A Multi-Scale Retrieval Benchmark for Time Series Language Models

model-releasesarxiv-cs-lg

arXiv:2602.14200v4 Announce Type: replace Abstract: Time Series Language Models (TSLMs) are emerging as unified models for reasoning over continuous signals in natural language. However, long-context

ALMAB-DC: Active Learning, Multi-Armed Bandits, and Distributed Computing for Sequential Experimental Design and Black-Box Optimization

model-releasesarxiv-cs-lg

arXiv:2603.21180v3 Announce Type: replace Abstract: Sequential experimental design under expensive, gradient-free objectives is a central challenge in computational statistics: evaluation budgets are

Enhancing LLM Problem Solving via Tutor-Student Multi-Agent Interaction

arXiv:2604.08931v1 Announce Type: new Abstract: Human cognitive development is shaped not only by individual effort but by structured social interaction, where role-based exchanges such as those betwe

EXAONE 4.5 Technical Report

model-releasesarxiv-cs-cl

arXiv:2604.08644v1 Announce Type: new Abstract: This technical report introduces EXAONE 4.5, the first open-weight vision language model released by LG AI Research. EXAONE 4.5 is architected by integr

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

arXiv:2604.09249v1 Announce Type: new Abstract: Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, ex

GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing

arXiv:2604.08896v1 Announce Type: new Abstract: Recent advances in multimodal large language models (MLLMs) have accelerated progress in domain-oriented AI, yet their development in geoscience and rem

GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification

model-releasesarxiv-cs-cl

arXiv:2604.08879v1 Announce Type: new Abstract: Moving beyond the traditional binary classification paradigm of Multimodal Sarcasm Detection, Multimodal Sarcasm Target Identification (MSTI) presents a

Hidden in Plain Sight: Visual-to-Symbolic Analytical Solution Inference from Field Visualizations

arXiv:2604.08863v1 Announce Type: new Abstract: Recovering analytical solutions of physical fields from visual observations is a fundamental yet underexplored capability for AI-assisted scientific rea

HM-Bench: A Comprehensive Benchmark for Multimodal Large Language Models in Hyperspectral Remote Sensing

arXiv:2604.08884v1 Announce Type: cross Abstract: While multimodal large language models (MLLMs) have made significant strides in natural image understanding, their ability to perceive and reason over

Predicting Metabolic Dysfunction-Associated Steatotic Liver Disease using Machine Learning Methods: A Retrospective Cohort Study

arXiv:2510.22293v4 Announce Type: replace Abstract: Background: Metabolic dysfunction-associated steatotic liver disease (MASLD) affects 30-40% of US adults and is the most common chronic liver diseas

safetyarxiv-cs-lg

PSIRNet: Deep Learning-based Free-breathing Rapid Acquisition Late Enhancement Imaging

arXiv:2604.08781v1 Announce Type: cross Abstract: Purpose: To develop and evaluate a deep learning (DL) method for free-breathing phase-sensitive inversion recovery (PSIR) late gadolinium enhancement

QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation

arXiv:2604.08570v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks