Search: “evaluation”

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

arXiv:2605.26485v1 Announce Type: cross Abstract: We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-vi

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attacks

DGX agent

arXiv:2605.26526v1 Announce Type: new Abstract: Recent defenses for safeguarding open-weight large language models (LLMs) are intended to prevent adversarial usage. Underlying these defenses is an ass

safetyarxiv-cs-lg

27 May 2026

PashtoTTS-Bench: automated screening for low-resource non-Latin-script text-to-speech

arXiv:2605.26978v1 Announce Type: new Abstract: Text-to-speech (TTS) evaluation for low-resource non-Latin-script languages can fail when it relies on a single ASR round-trip word error rate (WER). A

Local Ai

Persistent AI Agents in Academic Research: A Single-Investigator Implementation Case Study

arXiv:2605.26870v1 Announce Type: cross Abstract: Background: Large language models are typically evaluated as models, benchmarks, or short conversational episodes. Less is known about what happens wh

local-aiarxiv-cs-ai

PitchBench: Measuring Pitch Hearing in Audio-Language Models

arXiv:2605.26176v1 Announce Type: cross Abstract: Audio-language models (ALMs) are increasingly used in real-world applications that require understanding music, from music tutoring and transcription

Pretrained Approximators for Low-Thrust Trajectory Cost and Reachability

arXiv:2605.26790v1 Announce Type: new Abstract: Low-thrust trajectory design relies heavily on repeated evaluations of fuel consumption and transfer feasibility, which require expensive optimal contro

QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

arXiv:2605.27068v1 Announce Type: cross Abstract: Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM)

Representation-Aware Unlearning via Activation Signatures: From Suppression to Entity-Signature Erasure

arXiv:2601.10566v5 Announce Type: replace Abstract: Entity-level unlearning is usually evaluated by what a model says: whether it stops naming the target, refuses a query, or shifts a Truth Ratio dist

Semigroup Consistency as a Diagnostic for Learned Physics Simulators

arXiv:2605.26324v1 Announce Type: cross Abstract: Learned physics simulators are often evaluated by one-step or short-horizon prediction error, but these metrics can miss failures in temporal composit

SoftCap: Soft-Budget Control for Diffusion Transformer Acceleration

arXiv:2605.27075v1 Announce Type: new Abstract: Diffusion Transformers (DiTs) achieve strong visual quality, but their iterative denoising process requires many costly Transformer evaluations. Trainin

researcharxiv-cs-cv

The Daily Dose: Workflow-Integrated Large Language Model Automation for Clinical Summarization and Trial Identification in Radiation Oncology

arXiv:2605.26346v1 Announce Type: new Abstract: Objective: To describe the design and early clinical evaluation of The Daily Dose (TDD), an LLM-driven, automated clinical summarization and clinical-tr

researcharxiv-cs-cl

TSFMAudit: Data Contamination Auditing in Forecasting Time Series Foundation Models

arXiv:2605.26161v1 Announce Type: cross Abstract: Time series foundation models (TSFMs) are increasingly pretrained on large corpora, raising concerns that evaluation datasets may have been exposed du

researcharxiv-cs-ai

Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems

arXiv:2605.26302v1 Announce Type: new Abstract: Long-lived AI agents are increasingly deployed as persistent operational systems, yet they are still evaluated like freshly initialized models. Day-one

Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

model-releasesarxiv-cs-ro

arXiv:2601.16578v2 Announce Type: replace Abstract: We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Auto

A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks

arXiv:2605.23977v1 Announce Type: new Abstract: This paper audits benchmark evaluation in clinical-interview depression detection through four complementary probes across DAIC/E-DAIC, CMDC, ANDROIDS,

Safety

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

arXiv:2605.25536v1 Announce Type: cross Abstract: Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are pro

safetyarxiv-cs-ai

Local Ai

Catching MRI outliers: unsupervised detection and localization of MRI artefacts and clinical anomalies using deep learning

arXiv:2605.24609v1 Announce Type: cross Abstract: Artificial intelligence is increasingly integrated into radiotherapy workflows, yet such pipelines remain vulnerable to out-of-distribution image data

local-aiarxiv-cs-ai

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking

arXiv:2605.26087v1 Announce Type: cross Abstract: Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of establis

Safety

Energy Shields for Fairness

arXiv:2605.24926v1 Announce Type: new Abstract: Runtime fairness is not a one-time constraint but a dynamic property evaluated over a sequence of decisions. To ensure fairness at runtime, it is necess

safetyarxiv-cs-ai

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

arXiv:2605.25246v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimizat

Safety

Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization

arXiv:2605.24960v1 Announce Type: cross Abstract: Chain-of-Thought (CoT) faithfulness, i.e., whether CoTs genuinely reflect large language models' (LLM) underlying behavior, is typically evaluated und

safetyarxiv-cs-ai

Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework

arXiv:2605.24661v1 Announce Type: new Abstract: LLMs have achieved remarkable success in complex reasoning tasks, yet current evaluation approaches predominantly rely on final-answer correctness, offe

MuJoCoUni:Persistent Batched Runtime Primitives for MuJoCo

model-releasesarxiv-cs-ro

arXiv:2605.24922v1 Announce Type: new Abstract: We present MuJoCoUni, a downstream MuJoCo distribution for online robot learning and batched physics evaluation. Alongside the open-loop batched traject

SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

arXiv:2605.25420v1 Announce Type: cross Abstract: Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed g

StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios

arXiv:2605.25758v1 Announce Type: new Abstract: Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the re

Applications

Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions

arXiv:2605.24452v1 Announce Type: cross Abstract: Legal NLP benchmarks evaluate models on randomly split data, implicitly assuming that legal language is stationary. We test this assumption by fine-tu

applicationsarxiv-cs-ai

TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models

arXiv:2605.24292v1 Announce Type: new Abstract: Log-likelihood is a standard metric for evaluating generative models. Unfortunately, in contrast to autoregressive models (ARMs), discrete diffusion mod

researcharxiv-cs-lg

Are Targeted Data Poisoning Attacks as Effective as We Think?

arXiv:2509.06896v2 Announce Type: replace Abstract: Targeted data poisoning attacks manipulate model predictions on specific test samples by injecting malicious data into training. Yet existing evalua

researcharxiv-cs-lg

BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems

arXiv:2605.22866v1 Announce Type: new Abstract: Compound AI systems route tasks through hierarchies of specialised components. Attribution is dominated by Shapley-based methods (SHAP), which decompose

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

arXiv:2605.23238v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior i

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

arXiv:2605.23657v1 Announce Type: new Abstract: Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improvin

TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning

arXiv:2602.01665v2 Announce Type: replace-cross Abstract: The design of environments plays a critical role in shaping the development and evaluation of cooperative multi-agent reinforcement learning (

The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management

arXiv:2605.23071v1 Announce Type: new Abstract: Large language models (LLMs) increasingly rely on long-context processing, but expanding context windows introduces substantial computational and financ

researcharxiv-cs-cl

When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems

arXiv:2605.23414v1 Announce Type: new Abstract: LLM-based multi-agent systems can fail even when planned actions are executed correctly because agents may misjudge their knowledge when evaluating plan

Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

arXiv:2605.22168v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) map complex visual inputs to semantic spaces, but interpreting the cross-modal reasoning of VLMs currently relies on pos

23 May 2026

The Distillation Game: Adaptive Attacks & Efficient Defenses

arXiv:2605.22737v1 Announce Type: new Abstract: Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitat

researcharxiv-cs-lg

23 May 2026

VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation

arXiv:2605.22368v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed for software engineering, constructing high-quality benchmarks is crucial for evaluating not j

23 May 2026

Access Paths for Efficient Ordering with Large Language Models

arXiv:2509.00303v3 Announce Type: replace-cross Abstract: In this work, we present the exttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical imp

researcharxiv-cs-ai

AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture

model-releasesarxiv-cs-cv

arXiv:2605.22366v1 Announce Type: new Abstract: Agricultural decision-making increasingly requires multimodal systems that can transform visual observations into reliable, executable actions. However,

Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation

arXiv:2605.21949v1 Announce Type: new Abstract: Medical RAG systems in high-risk QA settings are often evaluated through a single answer-or-abstain decision, but mixed evidence may support one claim,

researcharxiv-cs-cl

HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine

arXiv:2605.21496v1 Announce Type: cross Abstract: Frontier language models are being deployed into clinical workflows faster than the infrastructure to evaluate them safely. Static medical-QA benchmar

VEELA: A Clinically-Constrained Benchmark for Liver Vessel Segmentation in Computed Tomography Angiography

model-releasesarxiv-cs-cv

arXiv:2605.22357v1 Announce Type: new Abstract: Accurate segmentation of hepatic and portal vessels in contrast-enhanced computed tomography angiography (CTA) remains challenging due to complex vascul

VISTA: Validation-Guided Integration of Spatial and Temporal Foundation Models with Anatomical Decoding for Rare-Pathology VCE Event Detection -- after competition results

arXiv:2605.22096v1 Announce Type: new Abstract: Capsule endoscopy event detection is challenging because clinically relevant findings are sparse, visually heterogeneous, and evaluated at the event lev

researcharxiv-cs-cv

Tutorials

Bayesian Optimization by Kernel Regression and Density-based Exploration

arXiv:2502.06178v5 Announce Type: replace-cross Abstract: Bayesian optimization is highly effective for optimizing expensive-to-evaluate black-box functions, but it faces significant computational cha

tutorialsarxiv-cs-lg

DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline

arXiv:2512.14896v2 Announce Type: replace Abstract: In our study, we evaluated large language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external kno

Pseudo-Formalization for Automatic Proof Verification

arXiv:2605.20531v1 Announce Type: cross Abstract: Reliable verification of proofs remains a bottleneck for training and evaluating AI systems on hard mathematical reasoning. Fully formal proofs, in la

RT @huntlovell: re: interpreters (since we''ve gotten asked this a few times) The proof will be in the evals/task-set for your agent, but…

This tweet by Hunt Lovell, shared by Harrison Chase, addresses questions about interpreters in the context of AI agent evaluation. It suggests that the effectiveness of interpreters should be validate

agentsharrison-chase--x

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

model-releasesarxiv-cs-cv

arXiv:2605.20676v1 Announce Type: new Abstract: Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimoda