Search: “evaluation”

When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems

arXiv:2604.01449v3 Announce Type: replace-cross Abstract: Artificial intelligence (AI) systems are increasingly integrated into healthcare and pharmacy workflows, supporting tasks such as medication r

safetyarxiv-cs-lg

21 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attacks

DGX agent

arXiv:2506.09521v2 Announce Type: replace-cross Abstract: Speaker anonymization systems hide the identity of speakers while preserving other information such as linguistic content and emotions. To eva

researcharxiv-cs-cl

21 May 2026

CogScale: Scalable Benchmark for Sequence Processing

arXiv:2605.19758v1 Announce Type: new Abstract: The ability to maintain and manipulate information over time is a fundamental aspect of living beings and Artificial Intelligence. While modern models h

Depth2Pose: A Pose-Based Benchmark for Monocular Depth Estimation without Ground-Truth Depth

arXiv:2605.19797v1 Announce Type: new Abstract: Monocular depth estimation has improved significantly in recent years, driven by increasingly powerful models and large-scale training data. Predicted d

Does Your Wildfire Prediction Model Actually Work, or Just Score Well?

arXiv:2605.18911v1 Announce Type: cross Abstract: Wildfire prediction is important for early warning and resource allocation, yet existing Earth foundation models (Earth FMs) are pretrained for genera

Embedding by Elicitation: Dynamic Representations for Bayesian Optimization of System Prompts

arXiv:2605.19093v1 Announce Type: new Abstract: System prompts are a central control mechanism in modern AI systems, shaping behavior across conversations, tasks, and user populations. Yet they are di

Faster-GCG: Efficient Discrete Optimization Jailbreak Attacks against Aligned Large Language Models

arXiv:2410.15362v2 Announce Type: replace-cross Abstract: Aligned Large Language Models (LLMs) have attracted significant attention for their safety, particularly in the context of jailbreak attacks t

GenAI-FDIA: Physics-Informed Generative Models for False Data Injection Attacks

arXiv:2605.18873v1 Announce Type: cross Abstract: Training and evaluating false data injection attack (FDIA) detectors for power systems is constrained by data scarcity. Operational grid measurements

Applications

Geo-Data-Driven HD Map Generation Workflow with Integrated Reference-Free Constraint-Based Verification

arXiv:2605.18921v1 Announce Type: new Abstract: High-definition (HD) maps are core artifacts for automated driving systems, but their generation commonly relies on sensor-intensive mobile mapping camp

applicationsarxiv-cs-ro

Local Ai

GRASP: Deterministic argument ranking in interaction graphs

arXiv:2605.19141v1 Announce Type: cross Abstract: Large language models are increasingly deployed as automated judges to evaluate the strength of arguments. As this role expands, their legitimacy depe

local-aiarxiv-cs-ai

HAVEN: Hierarchically Aligned Multimodal Benchmark for Unified Video Understanding

arXiv:2605.19223v1 Announce Type: new Abstract: While Multimodal Large Language Models (MLLMs) exhibit strong performance on standard video tasks, their ability to faithfully summarize and reason over

LLM Benchmark Datasets Should Be Contamination-Resistant

arXiv:2605.19999v1 Announce Type: cross Abstract: Benchmark datasets are critical for reproducible, reliable, and discriminative evaluation of LLMs. However, recent studies reveal that many benchmark

Applications

LP-Eval: Rubric and Dataset for Measuring the Quality of Legal Proposition Generation

arXiv:2605.19815v1 Announce Type: cross Abstract: Legal proposition generation is central to legal reasoning and doctrinal scholarship, yet remain under-examined in Legal NLP. This paper investigates

applicationsarxiv-cs-ai

Local Ai

Multi-Pedestrian Safety Warning at Urban Intersections Use Case of Digital Twin

arXiv:2605.18823v1 Announce Type: new Abstract: Digital twins (DTs) for urban transportation systems have gained increasing attention; however, their systematic evaluation in safety-critical scenarios

local-aiarxiv-cs-lg

Hardware

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

arXiv:2605.19782v1 Announce Type: new Abstract: LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or

hardwarearxiv-cs-ai

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

arXiv:2605.19382v1 Announce Type: new Abstract: Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluat

SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects

arXiv:2605.19587v1 Announce Type: new Abstract: Indoor scene synthesis underpins embodied AI, robotic manipulation, and simulation-based policy evaluation, where a useful scene must specify not only w

SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

arXiv:2605.19219v1 Announce Type: new Abstract: A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistica

Smooth Partial Lotteries for Stable Randomized Selection

arXiv:2605.20069v1 Announce Type: new Abstract: Competitive selection processes, from scientific funding to admissions and hiring, use evaluations to score candidates, and eventually choose a subset o

safetyarxiv-cs-lg

1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?

arXiv:2605.17046v1 Announce Type: cross Abstract: Autonomous AI coding agents are becoming a core tool for ML practitioners in industry and research alike. Despite this growing adoption, no standardiz

Ablating Safety: Mechanisms for Removing Alignment in Language Models for Security Applications

arXiv:2605.17413v1 Announce Type: cross Abstract: Safety-aligned language models often refuse cybersecurity requests whose wording resembles misuse, even when the task is authorized and defensive. Thi

Accelerating Rectified Flow Models via Trajectory-Aware Caching

arXiv:2605.16789v1 Announce Type: new Abstract: Diffusion and rectified flow (RF) models generate high-fidelity images and videos, but their iterative velocity-field evaluations are computationally ex

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

arXiv:2605.16819v1 Announce Type: cross Abstract: GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial

BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting

arXiv:2605.17937v1 Announce Type: cross Abstract: Quantitative backtesting is essential for evaluating trading strategies but remains hampered by high technical barriers and limited scalability. While

CAB: Accelerating Flow and Diffusion Sampling via Rectification and Corrected Adams-Bashforth

arXiv:2605.16736v1 Announce Type: new Abstract: Flow and diffusion models achieve high-fidelity, high-resolution image synthesis, but often require many function evaluations (NFEs) at sampling time. E

researcharxiv-cs-cv

CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean

arXiv:2605.17255v1 Announce Type: new Abstract: Formal theorem-proving benchmarks enable mechanically verifiable evaluation of mathematical reasoning in large language models. However, existing benchm

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

arXiv:2605.17110v1 Announce Type: new Abstract: Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation. Existing cluster

ContractBench: Can LLM Agents Preserve Observation Contracts?

arXiv:2605.17281v1 Announce Type: cross Abstract: Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation co

CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

arXiv:2602.03012v2 Announce Type: replace-cross Abstract: Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing wor

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

arXiv:2605.17307v1 Announce Type: cross Abstract: This study develops and evaluates a deep reinforcement learning framework for dynamic portfolio allocation across global equity markets. The Soft Acto

DexHoldem: Playing Texas Hold'em with Dexterous Embodied System

arXiv:2605.18727v1 Announce Type: cross Abstract: Evaluating embodied systems on real dexterous hardware requires more than isolated primitive skills: an agent must perceive a changing tabletop scene,

DISA: Offline Importance Sampling for Distribution-Matching LLM-RL

arXiv:2605.17295v1 Announce Type: cross Abstract: Modern reasoning agents are increasingly evaluated on their ability to generate multiple valid solution paths, plans, or tool-use traces for a given i

safetyarxiv-cs-cl

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

arXiv:2605.18421v1 Announce Type: cross Abstract: Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agen

Latency-Aware Deep Learning Benchmark for Real-Time Cyber-Physical Attack and Fault Classification in Inverter-Dominated Power Grids

arXiv:2605.17256v1 Announce Type: cross Abstract: This work introduces a latency-aware benchmarking framework for evaluating deep learning models in power system anomaly detection using high-fidelity,

LEAF: A Living Benchmark for Event-Augmented Forecasting

arXiv:2605.16358v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, se

Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairs

arXiv:2508.15878v2 Announce Type: replace-cross Abstract: Formal theorem proving (FTP) has emerged as a critical foundation for evaluating the reasoning capabilities of large language models, enabling

LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

arXiv:2605.16675v1 Announce Type: new Abstract: We introduce LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier large language models on structured linear algebra computation across a strict

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

arXiv:2603.23231v2 Announce Type: replace Abstract: Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of t

PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures

arXiv:2605.16551v1 Announce Type: new Abstract: Evaluating LLM-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic

safetyarxiv-cs-cl

ShareChat: A Dataset of Chatbot Conversations in the Wild

arXiv:2512.17843v4 Announce Type: replace-cross Abstract: By evaluating Large Language Models (LLMs) through uniform, text-only interfaces, current academic benchmarks obscure how the unique designs a

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

arXiv:2605.18693v1 Announce Type: new Abstract: As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether t

Sustainability via LLM Right-sizing

arXiv:2504.13217v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have become increasingly embedded in organizational workflows. This has raised concerns over their energy consump

Symmetry Matters: Auditing and Symmetrizing 3D Generative Models

arXiv:2512.18953v2 Announce Type: replace Abstract: Symmetry is a strong prior present in many object categories, yet standard benchmarks for 3D generative models rarely report whether this prior is p

researcharxiv-cs-cv

T-FIX: Text-Based Explanations with Features Interpretable to eXperts

arXiv:2511.04070v3 Announce Type: replace Abstract: As LLMs are deployed in knowledge-intensive settings (e.g., surgery, astronomy, therapy), users are often domain experts who expect not just answers

safetyarxiv-cs-cl

UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation

arXiv:2605.17140v1 Announce Type: cross Abstract: Brain tumor diagnosis is largely dependent on Magnetic Resonance Imaging (MRI) evaluation, which requires radiologists to synthesize thousands of imag

UniPPTBench: A Unified Benchmark for Presentation Generation Across Diverse Input Settings

arXiv:2605.17356v1 Announce Type: new Abstract: Existing works typically focus on presentation generation under isolated input settings, whereas real-world use cases span diverse scenarios, including

Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks

arXiv:2603.04459v3 Announce Type: replace-cross Abstract: The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastruct

18 May 2026

Breakeven complexity: A new perspective on neural partial differential equation solvers

arXiv:2605.15399v1 Announce Type: cross Abstract: Neural surrogate solvers of partial differential equations (PDEs) promise dramatic speedups over numerical methods, especially in scenarios requiring