Search: “evaluation”

Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems

arXiv:2605.26302v1 Announce Type: new Abstract: Long-lived AI agents are increasingly deployed as persistent operational systems, yet they are still evaluated like freshly initialized models. Day-one

27 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

DGX agent

arXiv:2601.16578v2 Announce Type: replace Abstract: We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Auto

model-releasesarxiv-cs-ro

27 May 2026

A Multi-Probe Audit of Clinical-Interview Depression Detection Benchmarks

arXiv:2605.23977v1 Announce Type: new Abstract: This paper audits benchmark evaluation in clinical-interview depression detection through four complementary probes across DAIC/E-DAIC, CMDC, ANDROIDS,

A Tertiary Review of Large Language Model-Based Code Generating Tasks: Trends, Challenges, and Future Directions

arXiv:2605.25536v1 Announce Type: cross Abstract: Context. Large language models (LLMs) are increasingly applied to code-generating tasks (CGTs) in software engineering. While reported results are pro

Local Ai

Catching MRI outliers: unsupervised detection and localization of MRI artefacts and clinical anomalies using deep learning

arXiv:2605.24609v1 Announce Type: cross Abstract: Artificial intelligence is increasingly integrated into radiotherapy workflows, yet such pipelines remain vulnerable to out-of-distribution image data

local-aiarxiv-cs-ai

DiscoverPhysics: Benchmarking LLMs for Out-of-the-Box Scientific Thinking

arXiv:2605.26087v1 Announce Type: cross Abstract: Frontier LLMs now perform strongly across a wide range of physics evaluations, but it is hard to disentangle genuine reasoning from recall of establis

Energy Shields for Fairness

arXiv:2605.24926v1 Announce Type: new Abstract: Runtime fairness is not a one-time constraint but a dynamic property evaluated over a sequence of decisions. To ensure fairness at runtime, it is necess

FrontierOR: Benchmarking LLMs' Capacity for Efficient Algorithm Design in Large-Scale Optimization

arXiv:2605.25246v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for optimization modeling and solver-code generation, yet practical operations research and optimizat

Investigating the Interplay between Contextual and Parametric Chain-of-Thought Faithfulness under Optimization

arXiv:2605.24960v1 Announce Type: cross Abstract: Chain-of-Thought (CoT) faithfulness, i.e., whether CoTs genuinely reflect large language models' (LLM) underlying behavior, is typically evaluated und

Measuring Reasoning Quality in LLMs: A Multi-Dimensional Behavioral Framework

arXiv:2605.24661v1 Announce Type: new Abstract: LLMs have achieved remarkable success in complex reasoning tasks, yet current evaluation approaches predominantly rely on final-answer correctness, offe

MuJoCoUni:Persistent Batched Runtime Primitives for MuJoCo

model-releasesarxiv-cs-ro

arXiv:2605.24922v1 Announce Type: new Abstract: We present MuJoCoUni, a downstream MuJoCo distribution for online robot learning and batched physics evaluation. Alongside the open-loop batched traject

SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

arXiv:2605.25420v1 Announce Type: cross Abstract: Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed g

StreamProfileBench: A Benchmark for Fine-Grained User Profile Inference in Real-World Streaming Scenarios

arXiv:2605.25758v1 Announce Type: new Abstract: Large Language Models (LLMs) have reshaped user profiling, yet current evaluations mainly focus on static data snapshots. This paradigm overlooks the re

Applications

Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions

arXiv:2605.24452v1 Announce Type: cross Abstract: Legal NLP benchmarks evaluate models on randomly split data, implicitly assuming that legal language is stationary. We test this assumption by fine-tu

applicationsarxiv-cs-ai

TUBE: Tangent Upper Bound on Evidence for Discrete Diffusion Language Models

arXiv:2605.24292v1 Announce Type: new Abstract: Log-likelihood is a standard metric for evaluating generative models. Unfortunately, in contrast to autoregressive models (ARMs), discrete diffusion mod

researcharxiv-cs-lg

Are Targeted Data Poisoning Attacks as Effective as We Think?

arXiv:2509.06896v2 Announce Type: replace Abstract: Targeted data poisoning attacks manipulate model predictions on specific test samples by injecting malicious data into training. Yet existing evalua

researcharxiv-cs-lg

BOHM: Zero-Cost Hierarchical Attribution for Compound AI Systems

arXiv:2605.22866v1 Announce Type: new Abstract: Compound AI systems route tasks through hierarchies of specialised components. Attribution is dominated by Shapley-based methods (SHAP), which decompose

GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models

arXiv:2605.23238v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings. Anticipating their behavior i

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

arXiv:2605.23657v1 Announce Type: new Abstract: Skills, i.e., structured workflow instructions distilled for large language models (LLMs), are becoming an increasingly important mechanism for improvin

Agents

TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learning

arXiv:2602.01665v2 Announce Type: replace-cross Abstract: The design of environments plays a critical role in shaping the development and evaluation of cooperative multi-agent reinforcement learning (

agentsarxiv-cs-ai

The Efficiency Frontier: A Unified Framework for Cost-Performance Optimization in LLM Context Management

arXiv:2605.23071v1 Announce Type: new Abstract: Large language models (LLMs) increasingly rely on long-context processing, but expanding context windows introduces substantial computational and financ

researcharxiv-cs-cl

Agents

When Planning Fails Despite Correct Execution: On Epistemic Calibration for LLM-Based Multi-Agent Systems

arXiv:2605.23414v1 Announce Type: new Abstract: LLM-based multi-agent systems can fail even when planned actions are executed correctly because agents may misjudge their knowledge when evaluating plan

agentsarxiv-cs-ai

Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability

arXiv:2605.22168v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) map complex visual inputs to semantic spaces, but interpreting the cross-modal reasoning of VLMs currently relies on pos

23 May 2026

The Distillation Game: Adaptive Attacks & Efficient Defenses

arXiv:2605.22737v1 Announce Type: new Abstract: Distillation attacks create a deployment trade-off for model providers: the same outputs that make a model more useful can also make it easier to imitat

researcharxiv-cs-lg

23 May 2026

VeriScale: Adversarial Test-Suite Scaling for Verifiable Code Generation

arXiv:2605.22368v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed for software engineering, constructing high-quality benchmarks is crucial for evaluating not j

23 May 2026

Access Paths for Efficient Ordering with Large Language Models

arXiv:2509.00303v3 Announce Type: replace-cross Abstract: In this work, we present the exttt{LLM ORDER BY} semantic operator as a logical abstraction and conduct a systematic study of its physical imp

researcharxiv-cs-ai

AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculture

arXiv:2605.22366v1 Announce Type: new Abstract: Agricultural decision-making increasingly requires multimodal systems that can transform visual observations into reliable, executable actions. However,

Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation

arXiv:2605.21949v1 Announce Type: new Abstract: Medical RAG systems in high-risk QA settings are often evaluated through a single answer-or-abstain decision, but mixed evidence may support one claim,

researcharxiv-cs-cl

HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine

arXiv:2605.21496v1 Announce Type: cross Abstract: Frontier language models are being deployed into clinical workflows faster than the infrastructure to evaluate them safely. Static medical-QA benchmar

VEELA: A Clinically-Constrained Benchmark for Liver Vessel Segmentation in Computed Tomography Angiography

arXiv:2605.22357v1 Announce Type: new Abstract: Accurate segmentation of hepatic and portal vessels in contrast-enhanced computed tomography angiography (CTA) remains challenging due to complex vascul

VISTA: Validation-Guided Integration of Spatial and Temporal Foundation Models with Anatomical Decoding for Rare-Pathology VCE Event Detection -- after competition results

arXiv:2605.22096v1 Announce Type: new Abstract: Capsule endoscopy event detection is challenging because clinically relevant findings are sparse, visually heterogeneous, and evaluated at the event lev

researcharxiv-cs-cv

Tutorials

Bayesian Optimization by Kernel Regression and Density-based Exploration

arXiv:2502.06178v5 Announce Type: replace-cross Abstract: Bayesian optimization is highly effective for optimizing expensive-to-evaluate black-box functions, but it faces significant computational cha

tutorialsarxiv-cs-lg

DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline

arXiv:2512.14896v2 Announce Type: replace Abstract: In our study, we evaluated large language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external kno

Pseudo-Formalization for Automatic Proof Verification

arXiv:2605.20531v1 Announce Type: cross Abstract: Reliable verification of proofs remains a bottleneck for training and evaluating AI systems on hard mathematical reasoning. Fully formal proofs, in la

VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence

arXiv:2605.20676v1 Announce Type: new Abstract: Establishing a clear link between model predictions and the visual evidence that supports them is critical for transparency and reliability in multimoda

When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems

arXiv:2604.01449v3 Announce Type: replace-cross Abstract: Artificial intelligence (AI) systems are increasingly integrated into healthcare and pharmacy workflows, supporting tasks such as medication r

safetyarxiv-cs-lg

You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks

arXiv:2506.09521v2 Announce Type: replace-cross Abstract: Speaker anonymization systems hide the identity of speakers while preserving other information such as linguistic content and emotions. To eva

researcharxiv-cs-cl

CogScale: Scalable Benchmark for Sequence Processing

arXiv:2605.19758v1 Announce Type: new Abstract: The ability to maintain and manipulate information over time is a fundamental aspect of living beings and Artificial Intelligence. While modern models h

Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth

arXiv:2605.19797v1 Announce Type: new Abstract: Monocular depth estimation has improved significantly in recent years, driven by increasingly powerful models and large-scale training data. Predicted d

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

arXiv:2605.18911v1 Announce Type: cross Abstract: Wildfire prediction is important for early warning and resource allocation, yet existing Earth foundation models (Earth FMs) are pretrained for genera

researcharxiv-cs-ai

Embedding by Elicitation: Dynamic Representations for Bayesian Optimization of System Prompts

arXiv:2605.19093v1 Announce Type: new Abstract: System prompts are a central control mechanism in modern AI systems, shaping behavior across conversations, tasks, and user populations. Yet they are di

Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models

arXiv:2410.15362v2 Announce Type: replace-cross Abstract: Aligned Large Language Models (LLMs) have attracted significant attention for their safety, particularly in the context of jailbreak attacks t

GenAI-FDIA: Physics-Informed Generative Models for False Data Injection Attacks

arXiv:2605.18873v1 Announce Type: cross Abstract: Training and evaluating false data injection attack (FDIA) detectors for power systems is constrained by data scarcity. Operational grid measurements

researcharxiv-cs-ai

Applications

Geo-Data-Driven HD Map Generation Workflow with Integrated Reference-Free Constraint-Based Verification

arXiv:2605.18921v1 Announce Type: new Abstract: High-definition (HD) maps are core artifacts for automated driving systems, but their generation commonly relies on sensor-intensive mobile mapping camp

applicationsarxiv-cs-ro

Local Ai

GRASP: Deterministic argument ranking in interaction graphs

arXiv:2605.19141v1 Announce Type: cross Abstract: Large language models are increasingly deployed as automated judges to evaluate the strength of arguments. As this role expands, their legitimacy depe

local-aiarxiv-cs-ai

HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding

arXiv:2605.19223v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) exhibit strong performance on standard video tasks, their ability to faithfully summarize and reason over

LLM Benchmark Datasets Should Be Contamination-Resistant

arXiv:2605.19999v1 Announce Type: cross Abstract: Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark

Applications

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

arXiv:2605.19815v1 Announce Type: cross Abstract: Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates

applicationsarxiv-cs-ai