Search: “evaluation”

arXiv:2605.18823v1 Announce Type: new Abstract: Digital twins (DTs) for urban transportation systems have gained increasing attention; however, their systematic evaluation in safety-critical scenarios

local-aiarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Hardware

Prior Knowledge or Search? A Study of LLM Agents in Hardware-Aware Code Optimization

DGX agent

arXiv:2605.19782v1 Announce Type: new Abstract: LLM discovery and optimization systems are increasingly applied across domains, implementing a common propose-evaluate-revise loop. Such optimization or

hardwarearxiv-cs-ai

20 May 2026

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

arXiv:2605.19382v1 Announce Type: new Abstract: Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluat

SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objects

arXiv:2605.19587v1 Announce Type: new Abstract: Indoor scene synthesis underpins embodied AI, robotic manipulation, and simulation-based policy evaluation, where a useful scene must specify not only w

SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents

arXiv:2605.19219v1 Announce Type: new Abstract: A/B testing remains the gold standard for evaluating modifications to e-commerce storefronts, yet it diverts traffic, requires weeks to reach statistica

Smooth Partial Lotteries for Stable Randomized Selection

arXiv:2605.20069v1 Announce Type: new Abstract: Competitive selection processes, from scientific funding to admissions and hiring, use evaluations to score candidates, and eventually choose a subset o

safetyarxiv-cs-lg

1GC-7RC: One Graphic Card -- Seven Research Challenges! How Good Are AI Agents at Doing Your Job?

arXiv:2605.17046v1 Announce Type: cross Abstract: Autonomous AI coding agents are becoming a core tool for ML practitioners in industry and research alike. Despite this growing adoption, no standardiz

Ablating Safety: Mechanisms for Removing Alignment in Language Models for Security Applications

arXiv:2605.17413v1 Announce Type: cross Abstract: Safety-aligned language models often refuse cybersecurity requests whose wording resembles misuse, even when the task is authorized and defensive. Thi

Accelerating Rectified Flow Models via Trajectory-Aware Caching

model-releasesarxiv-cs-cv

arXiv:2605.16789v1 Announce Type: new Abstract: Diffusion and rectified flow (RF) models generate high-fidelity images and videos, but their iterative velocity-field evaluations are computationally ex

AgentKernelArena: Generalization-Aware Benchmarking of GPU Kernel Optimization Agents

arXiv:2605.16819v1 Announce Type: cross Abstract: GPU kernel optimization is increasingly critical for efficient deep learning systems, but writing high-performance kernels still requires substantial

BacktestBench: Benchmarking Large Language Models for Automated Quantitative Strategy Backtesting

arXiv:2605.17937v1 Announce Type: cross Abstract: Quantitative backtesting is essential for evaluating trading strategies but remains hampered by high technical barriers and limited scalability. While

CAB: Accelerating Flow and Diffusion Sampling via Rectification and Corrected Adams-Bashforth

arXiv:2605.16736v1 Announce Type: new Abstract: Flow and diffusion models achieve high-fidelity, high-resolution image synthesis, but often require many function evaluations (NFEs) at sampling time. E

researcharxiv-cs-cv

CAM-Bench: A Benchmark for Computational and Applied Mathematics in Lean

arXiv:2605.17255v1 Announce Type: new Abstract: Formal theorem-proving benchmarks enable mechanically verifiable evaluation of mathematical reasoning in large language models. However, existing benchm

Capturing LLM Capabilities via Evidence-Calibrated Query Clustering

arXiv:2605.17110v1 Announce Type: new Abstract: Query clustering organizes queries into groups that reflect shared latent capability demands, enabling capability-aware LLM evaluation. Existing cluster

researcharxiv-cs-ai

ContractBench: Can LLM Agents Preserve Observation Contracts?

arXiv:2605.17281v1 Announce Type: cross Abstract: Tool-augmented LLM agents call APIs whose intermediate outputs, such as presigned URLs, session tokens, and OAuth state parameters, are observation co

CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability

arXiv:2602.03012v2 Announce Type: replace-cross Abstract: Evaluating and improving the security capabilities of code agents requires high-quality, executable vulnerability tasks. However, existing wor

Deep Reinforcement Learning Framework for Diversified Portfolio Management Across Global Equity Markets

arXiv:2605.17307v1 Announce Type: cross Abstract: This study develops and evaluates a deep reinforcement learning framework for dynamic portfolio allocation across global equity markets. The Soft Acto

DexHoldem: Playing Texas Hold'em with Dexterous Embodied System

arXiv:2605.18727v1 Announce Type: cross Abstract: Evaluating embodied systems on real dexterous hardware requires more than isolated primitive skills: an agent must perceive a changing tabletop scene,

DISA: Offline Importance Sampling for Distribution-Matching LLM-RL

arXiv:2605.17295v1 Announce Type: cross Abstract: Modern reasoning agents are increasingly evaluated on their ability to generate multiple valid solution paths, plans, or tool-use traces for a given i

safetyarxiv-cs-cl

EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective

arXiv:2605.18421v1 Announce Type: cross Abstract: Recent benchmarks for Large Language Model (LLM) agents mainly evaluate reasoning, planning, and execution. However, memory is also essential for agen

Latency-Aware Deep Learning Benchmark for Real-Time Cyber-Physical Attack and Fault Classification in Inverter-Dominated Power Grids

arXiv:2605.17256v1 Announce Type: cross Abstract: This work introduces a latency-aware benchmarking framework for evaluating deep learning models in power system anomaly detection using high-fidelity,

LEAF: A Living Benchmark for Event-Augmented Forecasting

arXiv:2605.16358v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly applied to forecasting. To evaluate this capability while mitigating pre-training data contamination, se

Lean Meets Theoretical Computer Science: Scalable Synthesis of Theorem Proving Challenges in Formal-Informal Pairs

arXiv:2508.15878v2 Announce Type: replace-cross Abstract: Formal theorem proving (FTP) has emerged as a critical foundation for evaluating the reasoning capabilities of large language models, enabling

researcharxiv-cs-ai

LinAlg-Bench: A Forensic Benchmark Revealing Structural Failure Modes in LLM Mathematical Reasoning

arXiv:2605.16675v1 Announce Type: new Abstract: We introduce LinAlg-Bench, a diagnostic benchmark evaluating 10 frontier large language models on structured linear algebra computation across a strict

PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments

arXiv:2603.23231v2 Announce Type: replace Abstract: Empowering large language models with long-term memory is crucial for building agents that adapt to users' evolving needs. Existing evaluations of t

PQR: A Framework to Generate Diverse and Realistic User Queries that Elicit QA Agent Failures

arXiv:2605.16551v1 Announce Type: new Abstract: Evaluating LLM-based agents remains challenging because identifying meaningful failure cases often requires substantial human effort to design realistic

safetyarxiv-cs-cl

ShareChat: A Dataset of Chatbot Conversations in the Wild

arXiv:2512.17843v4 Announce Type: replace-cross Abstract: By evaluating Large Language Models (LLMs) through uniform, text-only interfaces, current academic benchmarks obscure how the unique designs a

SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents

arXiv:2605.18693v1 Announce Type: new Abstract: As LLM agents are increasingly built around reusable skills, a central challenge is no longer only whether agents can use provided skills, but whether t

Sustainability via LLM Right-sizing

arXiv:2504.13217v3 Announce Type: replace-cross Abstract: Large language models (LLMs) have become increasingly embedded in organizational workflows. This has raised concerns over their energy consump

Symmetry Matters: Auditing and Symmetrizing 3D Generative Models

arXiv:2512.18953v2 Announce Type: replace Abstract: Symmetry is a strong prior present in many object categories, yet standard benchmarks for 3D generative models rarely report whether this prior is p

researcharxiv-cs-cv

T-FIX: Text-Based Explanations with Features Interpretable to eXperts

arXiv:2511.04070v3 Announce Type: replace Abstract: As LLMs are deployed in knowledge-intensive settings (e.g., surgery, astronomy, therapy), users are often domain experts who expect not just answers

safetyarxiv-cs-cl

UCSF-PDGM-VQA: Visual Question Answering dataset for brain tumor MRI interpretation

arXiv:2605.17140v1 Announce Type: cross Abstract: Brain tumor diagnosis is largely dependent on Magnetic Resonance Imaging (MRI) evaluation, which requires radiologists to synthesize thousands of imag

UniPPTBench: A Unified Benchmark for Presentation Generation Across Diverse Input Settings

model-releasesarxiv-cs-cv

arXiv:2605.17356v1 Announce Type: new Abstract: Existing works typically focus on presentation generation under isolated input settings, whereas real-world use cases span diverse scenarios, including

Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarks

arXiv:2603.04459v3 Announce Type: replace-cross Abstract: The rapid expansion of research in LLM safety presents challenges in tracking advancements, making benchmarks important evaluation infrastruct

Breakeven complexity: A new perspective on neural partial differential equation solvers

arXiv:2605.15399v1 Announce Type: cross Abstract: Neural surrogate solvers of partial differential equations (PDEs) promise dramatic speedups over numerical methods, especially in scenarios requiring

Agents

CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

arXiv:2512.01089v2 Announce Type: replace Abstract: Automated Scientific Discovery (ASD) systems can help automatically generate and run code-based experiments, but their capabilities are limited by t

agentsarxiv-cs-ai

From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discovery

arXiv:2605.15412v1 Announce Type: cross Abstract: Modern quantitative trading increasingly relies on systematic models to extract predictive signals from large-scale financial data, where alpha factor

HAI-Eval: Measuring Human-AI Synergy in Collaborative Coding

arXiv:2512.04111v2 Announce Type: replace-cross Abstract: LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans n

Latent Video Prediction Learns Better World Models

arXiv:2605.15618v1 Announce Type: cross Abstract: Self-supervised video models are increasingly framed as world models, yet their evaluation remains largely confined to a single top-1 accuracy score o

researcharxiv-cs-ai

A Benchmark for Early-stage Parkinson's Disease Detection from Speech

arXiv:2605.14066v1 Announce Type: cross Abstract: Early-stage Parkinson's disease (EarlyPD) detection from speech is clinically meaningful yet underexplored, and published results are hard to compare

A Regret Perspective on Online Multiple Testing

arXiv:2605.13916v1 Announce Type: cross Abstract: Online Multiple Testing (OMT), a fundamental pillar of sequential statistical inference, traditionally evaluates the False Discovery Rate (FDR) and st

Agentic Design of Compositional Descriptors via Autoresearch for Materials Science Applications

arXiv:2605.14671v1 Announce Type: cross Abstract: Autoresearch offers a flexible paradigm for automating scientific tasks, in which an AI agent proposes, implements, evaluates, and refines candidate s

Are Agents Ready to Teach? A Multi-Stage Benchmark for Real-World Teaching Workflows

arXiv:2605.14322v1 Announce Type: new Abstract: Language agents are increasingly deployed in complex professional workflows, with tutoring emerging as a particularly high-stakes capability that remain

Feature Visualization Recovers Known Cortical Selectivity from TRIBE v2

arXiv:2605.13904v1 Announce Type: cross Abstract: Brain encoder models predict cortical fMRI responses from the internal activations of pretrained vision and language networks, and are typically evalu

researcharxiv-cs-lg

Agents

Gradient Iterated Temporal-Difference Learning

arXiv:2603.07833v2 Announce Type: replace-cross Abstract: Temporal-difference (TD) learning is highly effective at controlling and evaluating an agent's long-term outcomes. Most approaches in this par

agentsarxiv-cs-ai

Agents

Heuristic Pathologies and Further Variance Reduction via Uncertainty Propagation in the AIVAT Family of Techniques

arXiv:2605.14261v1 Announce Type: new Abstract: How should an agent's performance in a multiagent environment be evaluated when there is a limited sample size or a high cost of running a trial? The AI

agentsarxiv-cs-ai

Multi-Dimensional Model Integrity and Responsibility Assessment Index and Scoring Framework

arXiv:2605.14550v1 Announce Type: new Abstract: Artificial intelligence in high-stakes tabular domains cannot be evaluated by predictive performance alone, yet current practice still assesses explaina

safetyarxiv-cs-lg

NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces

arXiv:2605.14698v1 Announce Type: cross Abstract: Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks. They have emerged across fields, including