Search: “evaluation”

LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation

arXiv:2510.26412v3 Announce Type: replace-cross Abstract: Recent advances in text-to-video generation have achieved impressive performance on short clips, yet evaluating long-form generation under com

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

ReasonBreak: Probing Vulnerabilities in Reasoning-Enabled Vision-Language-Action Models for Autonomous Driving

DGX agent

arXiv:2605.29114v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models with integrated reasoning have been proposed for end-to-end autonomous driving, assuming a tight coupling between

model-releasesarxiv-cs-lg

29 May 2026

Redundant or Necessary? A Benchmark for Detecting Redundant Steps in Agent Trajectories

arXiv:2605.29893v1 Announce Type: new Abstract: LLM-based agents have demonstrated strong capabilities in solving complex tasks through multi-step reasoning and tool use. However, existing evaluation

Selective QA over Conflicting Multi-Source Personal Memory: A Diagnostic Testbed and Method Comparison

arXiv:2605.30087v1 Announce Type: new Abstract: Emerging personal AI agents are moving toward persistent, multi-source memory. This creates an evaluation problem: systems must decide how to use confli

TIMEGATE: Sustainable Time-Boxed Promotion Gates for Continual ML Adaptation Under Resource Constraints

arXiv:2605.29183v1 Announce Type: cross Abstract: As machine learning(ML) systems evolve to continual adaptation, each re-training cycle uses compute, annotation, and energy. We introduce TIMEGATE, a

Toward Ethical Facial Age Estimation: A Generalized Zero-Shot Benchmark Without Training on Children's Data

arXiv:2605.29230v1 Announce Type: cross Abstract: Age estimation from facial images typically relies on training data that includes images of minors, a practice that raises serious ethical, legal, and

Applications

When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs

arXiv:2605.29420v1 Announce Type: new Abstract: Persona prompting is widely used to steer large language models, yet its practical value remains unclear. Prior work often evaluates persona prompting u

applicationsarxiv-cs-ai

Applications

Analyzing Cancer Patients' Experiences with Embedding-based Topic Modeling and LLMs

arXiv:2601.12154v2 Announce Type: replace Abstract: This study investigates the use of neural topic modeling and LLMs to uncover meaningful themes from patient storytelling data, to offer insights tha

applicationsarxiv-cs-cl

Asking Is Not Enough: Protocol Sensitivity in LLM Confidence Calibration

arXiv:2605.27752v1 Announce Type: new Abstract: LLM confidence calibration is often evaluated by comparing two signals: token-probability scores and verbalized confidence. These signals are sometimes

Auditable Decision Models with Learned Abstention and Real-Time Steering

arXiv:2605.27768v1 Announce Type: new Abstract: Production AI systems often operate with incomplete, conflicting, or insufficient evidence. Forced classifiers collapse such cases into action labels, w

Benchmarking Fairness in Spiking Neural Networks: Data Bias, Spurious Features, and Hardware Effects

arXiv:2605.27407v1 Announce Type: cross Abstract: Evaluating fairness in Spiking Neural Networks (SNNs) demands rigorous benchmarks that reflect real-world complexities, yet existing assessments remai

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

arXiv:2605.28183v1 Announce Type: cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER d

Better Accuracies, Worse Reasoning: A Step-Level Audit of Medical Chain-of-Thought Distillation

arXiv:2605.28301v1 Announce Type: new Abstract: Chain-of-thought (CoT) distillation trains a smaller model to imitate a teacher's reasoning trace, but it is typically evaluated by final-answer metrics

Breaking the Script Barrier: Enabling Automatic Alignment for PoS-based ASR Error Analysis in Non-Latin Scripts

arXiv:2605.28438v1 Announce Type: new Abstract: Automatic Speech Recognition (ASR) systems are commonly evaluated using aggregate metrics such as Word Error Rate (WER), which do not capture the lingui

safetyarxiv-cs-cl

Do Clinical Models Change Treatment Decisions?

arXiv:2605.28129v1 Announce Type: new Abstract: Clinical foundation models are evaluated with factual or exam-style medical QA, but treatment decisions must change when patient context changes. We int

EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

arXiv:2605.27820v1 Announce Type: new Abstract: As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop

Evolving and Detecting Multi-Turn Deception using Geometric Signatures

arXiv:2605.27671v1 Announce Type: cross Abstract: Safety defenses for large language models (LLMs) are typically trained and evaluated on single-turn prompts, yet real attacks often unfold as indirect

safetyarxiv-cs-lg

LLM Zeroth-Order Fine-Tuning is an Inference Workload

model-releasesarxiv-cs-lg

arXiv:2605.28760v1 Announce Type: new Abstract: Zeroth-order (ZO) fine-tuning is attractive for large language models because it replaces backpropagation with forward objective evaluations. Existing i

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

arXiv:2605.28579v1 Announce Type: new Abstract: Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design. Ex

PubMedCausal: A Span-Level Annotated Corpus for Causal Relation Extraction in Biomedical Text

arXiv:2605.28363v1 Announce Type: new Abstract: Causal relation extraction (CRE) is central to biomedical text mining, but current resources often conflate causal relations with broader associations,

SilentRetrieval: Hijacking Retrieval-Augmented Generation via Semantically-Preserving Adversarial Data Poisoning

arXiv:2605.28074v1 Announce Type: cross Abstract: Retrieval-Augmented Generation (RAG) mitigates LLM hallucinations but introduces a critical vulnerability: corpus integrity. We present SilentRetrieva

researcharxiv-cs-cl

StoryMI: Steerable Multi-Agent Therapeutic Dialogue Generation

arXiv:2605.27393v1 Announce Type: cross Abstract: Large language models (LLMs) can generate fluent dialogue, but prior works lack situational grounding, dynamic strategy control, and evaluation aligne

Applications

Surrogate modeling for Bayesian optimization beyond a single Gaussian process

arXiv:2205.14090v2 Announce Type: replace-cross Abstract: Bayesian optimization (BO) has well-documented merits for optimizing black-box functions with an expensive evaluation cost. Such functions eme

applicationsarxiv-cs-lg

The Cases LJP Never Sees: Prosecution Decision Prediction for More Complete Criminal Liability Assessment

arXiv:2605.28464v1 Announce Type: cross Abstract: Legal Judgment Prediction (LJP) has become a core benchmark for evaluating AI in the criminal legal domain, but it only sees criminal cases that have

The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models

arXiv:2510.20665v3 Announce Type: replace Abstract: Evaluating the quality of reasoning traces from large language models remains understudied, labor-intensive, and unreliable: current practice relies

When Discourse Pressures Conflict: Information Structure in Vision-Language Model Outputs

arXiv:2605.28346v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly evaluated for whether they identify the right visual content, but little is known about whether they expr

researcharxiv-cs-cl

Local Ai

Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models

arXiv:2605.27997v1 Announce Type: cross Abstract: Large language models frequently generate toxic, hateful, or harmful content, yet existing mitigation methods rely on costly retraining or output-leve

local-aiarxiv-cs-ai

ADRD-Bench: A Preliminary LLM Benchmark for Alzheimer's Disease and Related Dementias

arXiv:2602.11460v2 Announce Type: replace Abstract: Large language models (LLMs) have shown great potential for healthcare applications. However, existing evaluation benchmarks provide minimal coverag

Anchor: Mitigating Artifact Drift in Agent Benchmark Generation

arXiv:2605.26321v1 Announce Type: new Abstract: AI agents are beginning to complete valuable, long-horizon business operations tasks, but training and evaluation environments for enterprise work still

Are Video Models Zero-Shot Learners and Reasoners in Education? EduVideoBench, A Knowledge-Skills-Attitude Benchmark for Educational Video Generation

arXiv:2605.26918v1 Announce Type: new Abstract: Video generation models (VGMs) are rapidly entering classrooms, yet existing benchmarks evaluate only perceptual quality, intrinsic faithfulness, generi

Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models

arXiv:2506.09532v5 Announce Type: replace-cross Abstract: We present Athena-PRM, a multimodal process reward model (PRM) designed to evaluate the reward score for each step in solving complex reasonin

BESPOKE: Benchmark for Search-Augmented Large Language Model Personalization via Diagnostic Feedback

arXiv:2509.21106v2 Announce Type: replace Abstract: Search-augmented large language models (LLMs) have advanced information-seeking tasks by integrating retrieval into generation, reducing users' cogn

Detecting Is Not Resolving: The Monitoring Control Gap in Retrieval Augmented LLMs

arXiv:2605.27157v1 Announce Type: new Abstract: Retrieval-augmented LLMs are deployed for tasks where evidence quality determines action safety, yet evaluation protocols assume that single-turn robust

From Norms to Indicators (N2I-RAG): An Agentic Retrieval-Augmented Generation Framework for Legal Indicator Computation

arXiv:2605.26926v1 Announce Type: new Abstract: Computing legal indicators from normative texts is a key task in legal monitoring and policy evaluation, but presents significant challenges due to the

LiveK12Bench: Have Large Multimodal Models Truly Conquered High School-level Examinations?

arXiv:2605.26781v1 Announce Type: new Abstract: Advanced Large Multimodal Models (LMMs) have demonstrated impressive performance in K-12 reasoning tasks, exhibiting great promise as intelligent tutors

MedGuideX: Internalizing Decision Logic from Executable Guidelines into Large Language Models for Clinical Reasoning

arXiv:2605.26567v1 Announce Type: new Abstract: Clinical practice guidelines (CPGs) encode evidence-based decision logic that clinicians apply by evaluating patient variables, conditional criteria, an

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

arXiv:2605.26485v1 Announce Type: cross Abstract: We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-vi

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

arXiv:2605.26526v1 Announce Type: new Abstract: Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage. Underlying these defenses is an ass

safetyarxiv-cs-lg

PashtoTTS-Bench: automated screening for low-resource non-Latin-script text-to-speech

arXiv:2605.26978v1 Announce Type: new Abstract: Text-to-speech (TTS) evaluation for low-resource non-Latin-script languages can fail when it relies on a single ASR round-trip word error rate (WER). A

Local Ai

Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study

arXiv:2605.26870v1 Announce Type: cross Abstract: Background: Large language models are typically evaluated as models, benchmarks, or short conversational episodes. Less is known about what happens wh

local-aiarxiv-cs-ai

PitchBench: Measuring Pitch Hearing in Audio-Language Models

arXiv:2605.26176v1 Announce Type: cross Abstract: Audio-language models (ALMs) are increasingly used in real-world applications that require understanding music, from music tutoring and transcription

Pretrained Approximators for Low-Thrust Trajectory Cost and Reachability

model-releasesarxiv-cs-lg

arXiv:2605.26790v1 Announce Type: new Abstract: Low-thrust trajectory design relies heavily on repeated evaluations of fuel consumption and transfer feasibility, which require expensive optimal contro

Agents

QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

arXiv:2605.27068v1 Announce Type: cross Abstract: Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM)

agentsarxiv-cs-ai

Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure

arXiv:2601.10566v5 Announce Type: replace Abstract: Entity-level unlearning is usually evaluated by what a model says: whether it stops naming the target, refuses a query, or shifts a Truth Ratio dist

Agents

Semigroup Consistency as a Diagnostic for Learned Physics Simulators

arXiv:2605.26324v1 Announce Type: cross Abstract: Learned physics simulators are often evaluated by one-step or short-horizon prediction error, but these metrics can miss failures in temporal composit

agentsarxiv-cs-ai

SoftCap: Soft-Budget Control for Diffusion Transformer Acceleration

arXiv:2605.27075v1 Announce Type: new Abstract: Diffusion Transformers (DiTs) achieve strong visual quality, but their iterative denoising process requires many costly Transformer evaluations. Trainin

researcharxiv-cs-cv

The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncology

arXiv:2605.26346v1 Announce Type: new Abstract: Objective: To describe the design and early clinical evaluation of The Daily Dose (TDD), an LLM-driven, automated clinical summarization and clinical-tr

researcharxiv-cs-cl

TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models

arXiv:2605.26161v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are increasingly pretrained on large corpora, raising concerns that evaluation datasets may have been exposed du