Search: “evaluation”

WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems

arXiv:2605.24579v1 Announce Type: new Abstract: Long-context memory systems often fail under fixed budgets, but end-to-end evaluation does not reveal whether evidence was discarded during compression

26 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Adversarial Vulnerability Under Temporal Concept Drift: A Longitudinal Study of Android Malware Detection

DGX agent

arXiv:2605.23623v1 Announce Type: cross Abstract: We present a longitudinal, drift-aware evaluation of adversarial robustness across more than a decade of Android applications using static and dynamic

researcharxiv-cs-ai

25 May 2026

Class-Dependent Hybrid Data Augmentation for Multiclass Migraine Classification under Severe Class Imbalance

arXiv:2605.23453v1 Announce Type: new Abstract: We conducted a reproducibility-oriented re-evaluation of prior migraine classification studies, correcting for data leakage and metric bias. We then int

safetyarxiv-cs-lg

25 May 2026

Agents

EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation

arXiv:2605.23271v1 Announce Type: cross Abstract: The rapid evolution of generative video foundation models has propelled the field toward professional-grade cinematic synthesis. To achieve such deman

agentsarxiv-cs-ai

25 May 2026

VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

arXiv:2605.22907v1 Announce Type: new Abstract: Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal s

25 May 2026

HIDBench: Benchmarking Large Language Models for Host-Based Intrusion Detection

model-releasesarxiv-cs-lg

arXiv:2605.21773v1 Announce Type: cross Abstract: Recent benchmark efforts have advanced the evaluation of large language models (LLMs) in cybersecurity, including tasks such as penetration testing an

23 May 2026

Support-aware offline policy selection for advertising marketplaces

arXiv:2605.21736v1 Announce Type: cross Abstract: Logged advertising auctions make offline reserve-price evaluation attractive but risky. Replay tables can identify policies with large apparent yield

safetyarxiv-cs-lg

23 May 2026

AMEL: Accumulated Message Effects on LLM Judgments

arXiv:2605.22714v1 Announce Type: cross Abstract: Large language models are routinely used as automated evaluators: to review code, moderate content, or score outputs, often with many items passing th

22 May 2026

RealUserSim: Bridging the Reality Gap in Agent Benchmarking via Grounded User Simulation

arXiv:2605.20204v1 Announce Type: cross Abstract: LLM-based user simulation is the primary mechanism for end-to-end agent evaluation, yet simulated users are poor proxies for real humans: unconstraine

22 May 2026

VGenST-Bench: A Benchmark for Spatio-Temporal Reasoning via Active Video Synthesis

arXiv:2605.22570v1 Announce Type: new Abstract: Spatio-temporal reasoning is a core capability for Multimodal Large Language Models (MLLMs) operating in the real world. As such, evaluating it precisel

22 May 2026

ArchSIBench: Benchmarking the Architectural Spatial Intelligence of Vision-Language Models

arXiv:2605.20837v1 Announce Type: new Abstract: Architectural spatial intelligence, the ability to recognize and infer architectural space, is fundamental to tasks such as robot navigation, embodied i

MemGym: a Long-Horizon Memory Environment for LLM Agents

arXiv:2605.20833v1 Announce Type: new Abstract: Memory is a central capability for LLM agents operating across long-horizon tasks. Existing memory benchmarks predominantly evaluate retention of person

What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

model-releasesarxiv-cs-lg

arXiv:2605.21404v1 Announce Type: new Abstract: We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was ru

When Reasoning Supervision Hurts: TTCW-Based Long-Form Literary Review Generation

arXiv:2605.20364v1 Announce Type: new Abstract: Automatic evaluation of long-form literary writing remains challenging, as generic LLM-as-Judge approaches may not fully capture creativity-related dime

researcharxiv-cs-cl

Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels

arXiv:2605.19092v1 Announce Type: cross Abstract: Reasoning systems increasingly separate intermediate computation into private and public channels, creating evaluation cases that look similar in tran

researcharxiv-cs-ai

CriterAlign: Criterion-Centric Rationale Alignment for Code Preference Judging

arXiv:2605.19665v1 Announce Type: cross Abstract: Pairwise human preference prediction is central to evaluating code-generation systems, where quality often depends on task-specific trade-offs beyond

safetyarxiv-cs-ai

Does Code Cleanliness Affect Coding Agents? A Controlled Minimal-Pair Study

arXiv:2605.20049v1 Announce Type: cross Abstract: As autonomous coding agents see rapid adoption, their evaluation has primarily focused on task completion rates holding the target codebase fixed. Thi

GRAB: A Risk Taxonomy--Grounded Benchmark for Unsupervised Topic Discovery in Financial Disclosures

arXiv:2509.21698v2 Announce Type: replace Abstract: Risk categorization in 10-K risk disclosures matters for oversight and investment, yet no public benchmark evaluates unsupervised topic models for t

LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

arXiv:2605.19597v1 Announce Type: new Abstract: Evaluating large language models (LLMs) on natural-language logical reasoning is essential because rule-governed tasks require conclusions to follow str

Prompting language influences diagnostic reasoning and accuracy of large language models

arXiv:2605.19173v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly explored for clinical decision support, yet most evaluations are conducted in English, leaving their relia

A Comprehensive Survey of Action Quality Assessment: Method and Benchmark

arXiv:2412.11149v2 Announce Type: replace Abstract: Action Quality Assessment (AQA) aims to automatically evaluate how well human actions are performed and has been widely applied in sports analysis,

AMARIS: A Memory-Augmented Rubric Improvement System for Rubric-Based Reinforcement Learning

arXiv:2605.18592v1 Announce Type: cross Abstract: Rubric-based reward shaping is an effective method for fine-tuning LLMs via RL, where structured rubrics decompose standard outcome rewards into multi

researcharxiv-cs-ai

Auditing Multimodal LLM Raters: Central Tendency Bias in Clinical Ordinal Scoring

arXiv:2605.16386v1 Announce Type: new Abstract: Multimodal large language models (LLMs) are increasingly explored as automated evaluators in clinical settings, yet their scoring behavior on ordinal cl

Bench2Drive-Robust: Benchmarking Closed-Loop Autonomous Driving under Deployment Perturbations

model-releasesarxiv-cs-ro

arXiv:2605.18059v1 Announce Type: new Abstract: Robustness is a critical requirement for deploying autonomous driving systems in the real world. Existing robustness benchmarks for autonomous driving h

Beyond Accuracy: Decomposing the Reasoning Efficiency of LLMs

arXiv:2602.09805v2 Announce Type: replace-cross Abstract: As reasoning LLMs increasingly trade tokens for accuracy through deliberation, search, and self-correction, a single accuracy score can no lon

ClawArena: Benchmarking AI Agents in Evolving Information Environments

arXiv:2604.04202v2 Announce Type: replace-cross Abstract: AI agents deployed as persistent assistants must maintain correct beliefs as their information environment evolves. In practice, evidence is s

Conv-FinRe: A Conversational and Longitudinal Benchmark for Utility-Grounded Financial Recommendation

arXiv:2602.16990v2 Announce Type: replace Abstract: Most recommendation benchmarks evaluate how well a model imitates user behavior. In financial advisory, however, observed actions can be noisy or sh

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

arXiv:2601.11895v3 Announce Type: replace-cross Abstract: DevBench is a telemetry-driven benchmark designed to evaluate Large Language Models (LLMs) on realistic code completion tasks. It includes 1,8

MANTA: Multi-turn Assessment for Nonhuman Thinking & Alignment

arXiv:2605.16301v1 Announce Type: cross Abstract: Single-turn benchmarks such as AnimalHarmBench (AHB) have established important baselines for measuring animal welfare alignment in large language mod

OmniPro: A Comprehensive Benchmark for Omni-Proactive Streaming Video Understanding

arXiv:2605.18577v1 Announce Type: new Abstract: Omni-proactive streaming video understanding, i.e., autonomously deciding when to speak and what to say from continuous audio-visual streams, is an emer

REBAR: Reference Ethical Benchmark for Autonomy Readiness

model-releasesarxiv-cs-ro

arXiv:2605.18423v1 Announce Type: new Abstract: As autonomous systems grow more advanced, objective metrics to evaluate their ethical and legal compliance are critical for informing end users of their

TeleCom-Bench: How Far Are Large Language Models from Industrial Telecommunication Applications?

arXiv:2605.18025v1 Announce Type: new Abstract: While Large Language Models have achieved remarkable integration in various vertical scenarios, their deployment in the telecommunications domain remain

When Marginals Match but Structure Fails: Covariance Fidelity in Generative Models

arXiv:2603.17041v2 Announce Type: replace-cross Abstract: Generative models are increasingly deployed as substitutes for real data in downstream scientific workflows, yet standard evaluation criteria

researcharxiv-cs-ai

Why Do Safety Guardrails Degrade Across Languages?

arXiv:2605.17173v1 Announce Type: cross Abstract: Large language models exhibit safety degradation in non-English languages. Standard evaluation relies on Jailbreak Success Rate (JSR), which confounds

safetyarxiv-cs-ai

YOLO-NAS-Bench: A Surrogate Benchmark with Self-Evolving Predictors for YOLO Architecture Search

arXiv:2603.09405v2 Announce Type: replace Abstract: Neural Architecture Search (NAS) for object detection is severely bottlenecked by high evaluation cost, as fully training each candidate YOLO archit

A Unified Perturbation Framework for Analyzing Leaderboard Stability and Manipulation

arXiv:2605.15761v1 Announce Type: new Abstract: Evaluation leaderboards such as LMArena play a central role in benchmarking large language models by aggregating pairwise human preferences into model r

researcharxiv-cs-lg

Capability Conditioned Scaffolding for Professional Human LLM Collaboration

arXiv:2605.15404v1 Announce Type: new Abstract: Large language model personalization typically adapts outputs to user preferences and style but does not account for differences in user evaluation capa

researcharxiv-cs-cl

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization

arXiv:2605.15222v1 Announce Type: cross Abstract: Large language models (LLMs) can often generate functionally correct code, but their ability to produce efficient implementations for performance-crit

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

arXiv:2605.15622v1 Announce Type: new Abstract: Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in dee

researcharxiv-cs-lg

Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation

model-releasesarxiv-cs-lg

arXiv:2605.15669v1 Announce Type: new Abstract: Manufacturable chip layouts must satisfy thousands of geometry-based design rules, and design rule checking (DRC) enforces them by running executable DR

Agents

BOOST: A Data-Driven Framework for the Automated Joint Selection of Kernel and Acquisition Functions in Bayesian Optimization

arXiv:2508.02332v3 Announce Type: replace Abstract: The performance of Bayesian optimization (BO), a highly sample-efficient method for expensive black-box problems, is critically governed by the sele

agentsarxiv-cs-lg

Near-Miss: Latent Policy Failure Detection in Agentic Workflows

arXiv:2603.29665v2 Announce Type: replace Abstract: Agentic systems for business process automation often require compliance with policies governing conditional updates to the system state. Evaluation

PROVE: A Perceptual RemOVal cohErence Benchmark for Visual Media

arXiv:2605.14534v1 Announce Type: cross Abstract: Evaluating object removal in images and videos remains challenging because the task is inherently one-to-many, yet existing metrics frequently disagre

Agents

Quantum Advantage in Multi Agent Reinforcement Learning

arXiv:2605.14235v1 Announce Type: new Abstract: We present an empirical evaluation of quantum entanglement in agent coordination within quantum multi agent reinforcement learning (QMARL). While QMARL

agentsarxiv-cs-lg

ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

arXiv:2605.14152v1 Announce Type: cross Abstract: Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual

Local Ai

Small, Private Language Models as Teammates for Educational Assessment Design

arXiv:2605.15015v1 Announce Type: new Abstract: Generative AI increasingly supports educational design tasks, e.g., through Large Language Models (LLMs), demonstrating the capability to design assessm

local-aiarxiv-cs-ai

TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

arXiv:2605.13909v1 Announce Type: cross Abstract: Negotiation is a central mechanism of economic exchange, shaping markets, procurement, labor agreements, and resource allocation. It is also a canonic

safetyarxiv-cs-ai

Training ML Models with Predictable Failures

arXiv:2605.15134v1 Announce Type: new Abstract: Estimating how often an ML model will fail at deployment scale is central to pre-deployment safety assessment, but a feasible evaluation set is rarely l

safetyarxiv-cs-lg