Search: “evaluation”

Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations

arXiv:2604.15302v1 Announce Type: cross Abstract: LLM-as-judge frameworks are increasingly used for automatic NLG evaluation, yet their per-instance reliability remains poorly understood. We present a

researcharxiv-cs-cl

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Internal Knowledge Without External Expression: Probing the Generalization Boundary of a Classical Chinese Language Model

DGX agent

arXiv:2604.14180v1 Announce Type: new Abstract: We train a 318M-parameter Transformer language model from scratch on a curated corpus of 1.56 billion tokens of pure Classical Chinese, with zero Englis

model-releasesarxiv-cs-cl

17 Apr 2026

Learning to Think Like a Cartoon Captionist: Incongruity-Resolution Supervision for Multimodal Humor Understanding

arXiv:2604.15210v1 Announce Type: cross Abstract: Humor is one of the few cognitive tasks where getting the reasoning right matters as much as getting the answer right. While recent work evaluates hum

safetyarxiv-cs-cl

Meituan Merchant Business Diagnosis via Policy-Guided Dual-Process User Simulation

arXiv:2604.15190v1 Announce Type: cross Abstract: Simulating group-level user behavior enables scalable counterfactual evaluation of merchant strategies without costly online experiments. However, bui

safetyarxiv-cs-cl

Mitigating LLM biases toward spurious social contexts using direct preference optimization

arXiv:2604.02585v2 Announce Type: replace-cross Abstract: LLMs are increasingly used for high-stakes decision-making, yet their sensitivity to spurious contextual information can introduce harmful bia

Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects

arXiv:2510.07890v3 Announce Type: replace Abstract: Research on cross-dialectal transfer from a standard to a non-standard dialect variety has typically focused on text data. However, dialects are pri

applicationsarxiv-cs-cl

TwinOR: Photorealistic Digital Twins of Dynamic Operating Rooms for Embodied AI Research

arXiv:2511.07412v2 Announce Type: replace Abstract: Developing embodied AI for intelligent surgical systems requires safe, controllable environments for continual learning and evaluation. However, saf

safetyarxiv-cs-cv

VoxSafeBench: Not Just What Is Said, but Who, How, and Where

arXiv:2604.14548v1 Announce Type: cross Abstract: As speech language models (SLMs) transition from personal devices into shared, multi-user environments, their responses must account for far more than

safetyarxiv-cs-lg

EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development

model-releasesarxiv-cs-ro

arXiv:2604.13800v1 Announce Type: new Abstract: Embodied AI research is increasingly moving beyond single-task, single-environment policy learning toward multi-task, multi-scene, and multi-model setti

Empirical Evidence of Complexity-Induced Limits in Large Language Models on Finite Discrete State-Space Problems with Explicit Validity Constraints

arXiv:2604.13371v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly described as possessing strong reasoning capabilities, supported by high performance on mathematical, logi

researcharxiv-cs-cl

Giving Voice to the Constitution: Low-Resource Text-to-Speech for Quechua and Spanish Using a Bilingual Legal Corpus

arXiv:2604.13288v1 Announce Type: new Abstract: We present a unified pipeline for synthesizing high-quality Quechua and Spanish speech for the Peruvian Constitution using three state-of-the-art text-t

applicationsarxiv-cs-cl

HINTBench: Horizon-agent Intrinsic Non-attack Trajectory Benchmark

model-releasesarxiv-cs-lg

arXiv:2604.13954v1 Announce Type: new Abstract: Existing agent-safety evaluation has focused mainly on externally induced risks. Yet agents may still enter unsafe trajectories under benign conditions.

Scaling Test-Time Compute to Achieve IOI Gold Medal with Open-Weight Models

arXiv:2510.14232v2 Announce Type: replace-cross Abstract: Competitive programming has become a rigorous benchmark for evaluating the reasoning and problem-solving capabilities of large language models

Agents

very excited for this one! a year ago, most of what was being traced to LangSmith was 'LLM apps'. now everything is becoming an agent. with …

very excited for this one! a year ago, most of what was being traced to LangSmith was 'LLM apps'. now everything is becoming an agent. with that shift, it's getting harder to know what your software i

agentsharrison-chase--x

Local Ai

Which video model currently has the best face likeness for LoRA training?

local-air-stablediffusion

This r/StableDiffusion thread discusses community comparisons of video generation models (such as those built on Stable Diffusion or FLUX architectures) evaluated specifically for how well they preser

BEAM: Bi-level Memory-adaptive Algorithmic Evolution for LLM-Powered Heuristic Design

arXiv:2604.12898v1 Announce Type: new Abstract: Large Language Model-based Hyper Heuristic (LHH) has recently emerged as an efficient way for automatic heuristic design. However, most existing LHHs ju

researcharxiv-cs-ai

GRADE: Probing Knowledge Gaps in LLMs through Gradient Subspace Dynamics

arXiv:2604.02830v2 Announce Type: replace Abstract: Detecting whether a model's internal knowledge is sufficient to correctly answer a given question is a fundamental challenge in deploying responsibl

applicationsarxiv-cs-cl

Human-Inspired Context-Selective Multimodal Memory for Social Robots

arXiv:2604.12081v1 Announce Type: new Abstract: Memory is fundamental to social interaction, enabling humans to recall meaningful past experiences and adapt their behavior accordingly based on the con

researcharxiv-cs-ai

Industry

More people should work on harnesses for open and local models!

Hugging Face co-founder and CEO Clément Delangue advocates for more developers and researchers to focus on building evaluation harnesses and testing frameworks specifically designed for open-source an

industryclem-delangue--x

Red Teaming Large Reasoning Models

arXiv:2512.00412v4 Announce Type: replace-cross Abstract: Large Reasoning Models (LRMs) have emerged as a powerful advancement in multi-step reasoning tasks, offering enhanced transparency and logical

A Benchmark for Gap and Overlap Analysis as a Test of KG Task Readiness

arXiv:2604.10853v1 Announce Type: new Abstract: Task-oriented evaluation of knowledge graph (KG) quality increasingly asks whether an ontology-based representation can answer the competency questions

A Dual-Positive Monotone Parameterization for Multi-Segment Bids and a Validity Assessment Framework for Reinforcement Learning Agent-based Simulation of Electricity Markets

arXiv:2604.10252v1 Announce Type: new Abstract: Reinforcement learning agent-based simulation (RL-ABS) has become an important tool for electricity market mechanism analysis and evaluation. In the mod

safetyarxiv-cs-ai

ACCIDENT: A Benchmark Dataset for Vehicle Accident Detection from Traffic Surveillance Videos

arXiv:2604.09819v1 Announce Type: cross Abstract: We introduce ACCIDENT, a benchmark dataset for traffic accident detection in CCTV footage, designed to evaluate models in supervised (IID and OOD) and

AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts

arXiv:2601.11044v3 Announce Type: replace Abstract: Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. Howev

Agent^2 RL-Bench: Can LLM Agents Engineer Agentic RL Post-Training?

arXiv:2604.10547v1 Announce Type: new Abstract: We introduce Agent^2 RL-Bench, a benchmark for evaluating agentic RL post-training -- whether LLM agents can autonomously design, implement, and run com

Agents

BridgeSim: Unveiling the OL-CL Gap in End-to-End Autonomous Driving

arXiv:2604.10856v1 Announce Type: cross Abstract: Open-loop (OL) to closed-loop (CL) gap (OL-CL gap) exists when OL-pretrained policies scoring high in OL evaluations fail to transfer effectively in c

agentsarxiv-cs-ai

ChatCLIDS: Simulating Persuasive AI Dialogues to Promote Closed-Loop Insulin Adoption in Type 1 Diabetes Care

arXiv:2509.00891v3 Announce Type: replace Abstract: Real-world adoption of closed-loop insulin delivery systems (CLIDS) in type 1 diabetes remains low, driven not by technical failure, but by diverse

Consistency of AI-Generated Exercise Prescriptions: A Repeated Generation Study Using a Large Language Model

arXiv:2604.11287v1 Announce Type: new Abstract: Background: Large language models (LLMs) have been explored as tools for generating personalized exercise prescriptions, yet the consistency of outputs

Decision-Theoretic Safety Assessment of Persona-Driven Multi-Agent Systems in O-RAN

arXiv:2604.09682v1 Announce Type: cross Abstract: Autonomous network management in Open Radio Access Networks requires intelligent decision making across conflicting objectives, yet existing LLM based

safetyarxiv-cs-ai

Disambiguation-Centric Finetuning Makes Enterprise Tool-Calling LLMs More Realistic and Less Risky

arXiv:2507.03336v4 Announce Type: replace Abstract: Large language models (LLMs) are increasingly tasked with invoking enterprise APIs, yet they routinely falter when near-duplicate tools vie for the

Improving understanding and trust in AI: How users benefit from interval-based counterfactual explanations

arXiv:2604.09573v1 Announce Type: cross Abstract: Experimental user studies evaluating the effectiveness of different subtypes of post-hoc explanations for black-box models are largely nonexistent. Th

researcharxiv-cs-lg

ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

arXiv:2604.03765v2 Announce Type: replace Abstract: Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing

LLM-as-Judge on a Budget

arXiv:2602.15481v2 Announce Type: replace Abstract: LLM-as-a-judge has emerged as a cornerstone technique for evaluating large language models by leveraging LLM reasoning to score prompt-response pair

safetyarxiv-cs-lg

MMRareBench: A Rare-Disease Multimodal and Multi-Image Medical Benchmark

arXiv:2604.10755v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have advanced clinical tasks for common conditions, but their performance on rare diseases remains largely unte

NameBERT: Scaling Name-Based Nationality Classification with LLM-Augmented Open Academic Data

arXiv:2604.10401v1 Announce Type: new Abstract: Inferring nationality from personal names is a critical capability for equity and bias monitoring, personalization, and a valuable tool in biomedical an

safetyarxiv-cs-cl

Property-Preserving Hashing for ell_1-Distance Predicates: Applications to Countering Adversarial Input Attacks

arXiv:2504.16355v2 Announce Type: replace-cross Abstract: Perceptual hashing is used to detect whether an input image is similar to a reference image with a variety of security applications. Recently,

researcharxiv-cs-lg

Quantization Robustness to Input Degradations for Object Detection

arXiv:2508.19600v2 Announce Type: replace Abstract: Post-training quantization (PTQ) is crucial for deploying efficient object detection models, like YOLO, on resource-constrained devices. However, th

applicationsarxiv-cs-cv

Should We be Pedantic About Reasoning Errors in Machine Translation?

arXiv:2604.09890v1 Announce Type: cross Abstract: Across multiple language pairings (English o {Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese}), we find reasoning errors in translation.

researcharxiv-cs-ai

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

arXiv:2604.09557v1 Announce Type: cross Abstract: Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optim

SRBench: A Comprehensive Benchmark for Sequential Recommendation with Large Language Models

arXiv:2604.09553v1 Announce Type: cross Abstract: LLM development has aroused great interest in Sequential Recommendation (SR) applications. However, comprehensive evaluation of SR models remains lack

TrajOnco: a multi-agent framework for temporal reasoning over longitudinal EHR for multi-cancer early detection

arXiv:2604.10386v1 Announce Type: new Abstract: Accurate estimation of cancer risk from longitudinal electronic health records (EHRs) could support earlier detection and improved care, but modeling su

TS-Haystack: A Multi-Scale Retrieval Benchmark for Time Series Language Models

model-releasesarxiv-cs-lg

arXiv:2602.14200v4 Announce Type: replace Abstract: Time Series Language Models (TSLMs) are emerging as unified models for reasoning over continuous signals in natural language. However, long-context

ALMAB-DC: Active Learning, Multi-Armed Bandits, and Distributed Computing for Sequential Experimental Design and Black-Box Optimization

model-releasesarxiv-cs-lg

arXiv:2603.21180v3 Announce Type: replace Abstract: Sequential experimental design under expensive, gradient-free objectives is a central challenge in computational statistics: evaluation budgets are

Enhancing LLM Problem Solving via Tutor-Student Multi-Agent Interaction

arXiv:2604.08931v1 Announce Type: new Abstract: Human cognitive development is shaped not only by individual effort but by structured social interaction, where role-based exchanges such as those betwe

EXAONE 4.5 Technical Report

arXiv:2604.08644v1 Announce Type: new Abstract: This technical report introduces EXAONE 4.5, the first open-weight vision language model released by LG AI Research. EXAONE 4.5 is architected by integr

FashionStylist: An Expert Knowledge-enhanced Multimodal Dataset for Fashion Understanding

arXiv:2604.09249v1 Announce Type: new Abstract: Fashion understanding requires both visual perception and expert-level reasoning about style, occasion, compatibility, and outfit rationale. However, ex

GeoMMBench and GeoMMAgent: Toward Expert-Level Multimodal Intelligence in Geoscience and Remote Sensing

arXiv:2604.08896v1 Announce Type: new Abstract: Recent advances in multimodal large language models (MLLMs) have accelerated progress in domain-oriented AI, yet their development in geoscience and rem

GRASP: Grounded CoT Reasoning with Dual-Stage Optimization for Multimodal Sarcasm Target Identification

arXiv:2604.08879v1 Announce Type: new Abstract: Moving beyond the traditional binary classification paradigm of Multimodal Sarcasm Detection, Multimodal Sarcasm Target Identification (MSTI) presents a