Search: “evaluation”

Unsteady Metrics and Benchmarking Cultures of AI Model Builders

arXiv:2605.14164v1 Announce Type: new Abstract: The primary way to establish and compare competencies in foundation and generative AI models has shifted from peer-reviewed literature to press releases

15 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

A Benchmark for Multi-Party Negotiation Games from Real Negotiation Data

DGX agent

arXiv:2603.14066v2 Announce Type: replace-cross Abstract: Many real-world multi-party negotiations unfold as sequences of binding, action-level commitments rather than a single final outcome, yet this

model-releasesarxiv-cs-ai

14 May 2026

Code-Centric Detection of Vulnerability-Fixing Commits: A Unified Benchmark and Empirical Study

model-releasesarxiv-cs-lg

arXiv:2605.13138v1 Announce Type: cross Abstract: Automated detection of vulnerability-fixing commits (VFCs) is critical for timely security patch deployment, as advisory databases lag patch releases

Context Matters: Auditing Gender Bias in T2I Generation through Risk-Tiered Use-Case Profiles

arXiv:2605.13113v1 Announce Type: cross Abstract: Text-to-image (T2I) generative models are increasingly used to produce content for education, media, and public-facing communication, and are starting

Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models

arXiv:2605.12519v1 Announce Type: cross Abstract: Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards

researcharxiv-cs-ai

CRAFT: Clinical Reward-Aligned Finetuning for Medical Image Synthesis

arXiv:2605.12650v1 Announce Type: new Abstract: Foundation diffusion models can generate photorealistic natural images, but adapting them to medical imaging remains challenging. In medical adaptation,

safetyarxiv-cs-cv

Digital Twins as Synthetic Controls in Single-Arm Trials

arXiv:2605.12832v1 Announce Type: cross Abstract: Single-arm trials are an important study design for evaluating drug efficacy and safety without enrolling patients into a control arm. Although they d

Local Ai

Discovery of Hidden Miscalibration Regimes

arXiv:2605.13484v1 Announce Type: cross Abstract: Calibration is commonly evaluated by comparing model confidence with its empirical correctness, implicitly treating reliability as a function of the c

local-aiarxiv-cs-ai

How Well Do Large-Scale Chemical Language Models Transfer to Downstream Tasks?

model-releasesarxiv-cs-lg

arXiv:2602.11618v4 Announce Type: replace Abstract: Chemical Language Models (CLMs) pre-trained on large scale molecular data are widely used for molecular property prediction. However, the common bel

Applications

MILM: Large Language Models for Multimodal Irregular Time Series with Informative Sampling

arXiv:2605.13711v1 Announce Type: new Abstract: Multimodal irregular time series (MITS) consist of asynchronous and irregularly sampled observations from heterogeneous numerical and textual channels.

applicationsarxiv-cs-lg

No One Knows the State of the Art in Geospatial Foundation Models

arXiv:2605.12678v1 Announce Type: new Abstract: Geospatial foundation models (GFMs) have been proposed as generalizable backbones for disaster response, land-cover mapping, food-security monitoring, a

ODRPO: Ordinal Decompositions of Discrete Rewards for Robust Policy Optimization

arXiv:2605.12667v1 Announce Type: cross Abstract: The alignment of Large Language Models (LLMs) utilizes Reinforcement Learning from AI Feedback (RLAIF) for non-verifiable domains such as long-form qu

PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Records

arXiv:2601.09636v2 Announce Type: replace Abstract: While GUI agents have shown strong performance under explicit and completion instructions, real-world deployment requires aligning with users' more

PhysEditBench: A Protocol-Conditioned Benchmark for Dense Physical-Map Prediction with Image Editors

arXiv:2605.13493v1 Announce Type: new Abstract: Can general-purpose image editors predict physical maps from a single RGB image? General-purpose image editors differ from standard task-specific dense-

PreFIQs: Face Image Quality Is What Survives Pruning

arXiv:2605.13396v1 Announce Type: new Abstract: Face Image Quality Assessment (FIQA) evaluates the utility of a face image for automated face recognition (FR) systems. In this work, we propose PreFIQs

Quantitative Certification of Agentic Tool Selection

arXiv:2510.03992v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly deployed in agentic systems, where a fundamental task is mapping user intents to relevant extern

Qwen-Image-VAE-2.0 Technical Report

arXiv:2605.13565v1 Announce Type: new Abstract: We present Qwen-Image-VAE-2.0, a suite of high-compression Variational Autoencoders (VAEs) that achieve significant advances in both reconstruction fide

Tutorials

Retrieval-Augmented Tutoring for Algorithm Tracing and Problem-Solving in AI Education

arXiv:2605.12988v1 Announce Type: new Abstract: Students learning algorithms often need support as they interpret traces, debug reasoning errors, and apply procedures across unfamiliar problem instanc

tutorialsarxiv-cs-ai

Runtime Monitoring of Perception-Based Autonomous Systems via Embedding Temporal Logic

arXiv:2605.12651v1 Announce Type: new Abstract: Runtime monitoring of autonomous systems traditionally relies on mapping continuous sensor observations to discrete logical propositions defined over lo

SpatialReward: Bridging the Perception Gap in Online RL for Image Editing via Explicit Spatial Reasoning

arXiv:2602.07458v4 Announce Type: replace Abstract: Online Reinforcement Learning (RL) offers a promising avenue for complex image editing but is currently constrained by the scarcity of reliable and

StreamGaze: Gaze-Guided Temporal Reasoning and Proactive Understanding in Streaming Videos

arXiv:2512.01707v3 Announce Type: replace-cross Abstract: Streaming video understanding requires models not only to process temporally incoming frames, but also to anticipate user intention for realis

Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirements

arXiv:2605.12963v1 Announce Type: new Abstract: As AI systems become increasingly capable, safety strategies must be evaluated not only by how much they reduce present risk, but by whether they could

Test-time Offline Reinforcement Learning on Goal-related Experience

arXiv:2507.18809v2 Announce Type: replace Abstract: Foundation models compress a large amount of information in a single, large neural network, which can then be queried for individual tasks. There ar

Agents

1/5 We’re seeing 4 common agent optimization methods for hitting the right accuracy-cost or accuracy-latency tradeoff. We tried them all out…

AI21 Labs discusses four common agent optimization methods used to balance accuracy against cost and latency constraints. The post indicates the team evaluated all four approaches, likely covering tec

agentsai21-labs--x

AgentDisCo: Towards Disentanglement and Collaboration in Open-ended Deep Research Agents

model-releasesarxiv-cs-cl

arXiv:2605.11732v1 Announce Type: cross Abstract: In this paper, we present AgentDisCo, a novel Disentangled and Collaborative agentic architecture that formulates deep research as an adversarial opti

Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Models

arXiv:2601.21944v2 Announce Type: replace Abstract: The widespread adoption of deep learning models in computer vision has intensified concerns about interpretability. Despite strong performance, thes

Agents

DreamAvoid: Critical-Phase Test-Time Dreaming to Avoid Failures in VLA Policies

arXiv:2605.11750v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models are often brittle in fine-grained manipulation, where minor action errors during the critical phases can rapidly e

agentsarxiv-cs-cl

FLAME: A New Dataset on FLemish Accounts of Momentary Experiences

model-releasesarxiv-cs-cl

arXiv:2504.14707v3 Announce Type: replace Abstract: We introduce FLAME (FLemish Accounts of Momentary Experiences), a new corpus of nearly 25,000 daily personal narratives in Belgian-Dutch (Flemish),

Generative AI for Visualizing Highway Construction Hazards Through Synthetic Images and Temporal Sequences

arXiv:2605.11276v1 Announce Type: new Abstract: Highway construction workers face a high risk of serious injury or death. Image-based training materials depicting hazardous scenarios are essential for

safetyarxiv-cs-cv

Google Named a Leader in the Gartner® Magic Quadrant™ for AI Application Development Platforms: Mid-cycle update

model-releasesgoogle-cloud-ai

May 2026 update: We’ve refreshed this post to reflect our mid-cycle positioning and the evolution of our platform since the report was first published last November. Last fall, Google was recognized a

HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench

model-releasesarxiv-cs-cl

arXiv:2601.20255v2 Announce Type: replace-cross Abstract: SWE-bench has emerged as the premier benchmark for evaluating Large Language Models on complex software engineering tasks. While these capabil

Applications

Interpretability Can Be Actionable

arXiv:2605.11161v1 Announce Type: new Abstract: Interpretability aims to explain the behavior of deep neural networks. Despite rapid growth, there is mounting concern that much of this work has not tr

applicationsarxiv-cs-lg

LiBrA-Net: Lie-Algebraic Bilateral Affine Fields for Real-Time 4K Video Dehazing

arXiv:2605.11508v1 Announce Type: new Abstract: Currently, there is a gap in the field of ultra-high-definition (UHD) video dehazing due to the lack of a benchmark for evaluation. Furthermore, existin

Nautilus: From One Prompt to Plug-and-Play Robot Learning

model-releasesarxiv-cs-ro

arXiv:2605.11665v1 Announce Type: new Abstract: Robot learning research is fragmented across policy families, benchmark suites, and real robots; each implementation is entangled with the others in a c

OUI as a Structural Observable: Towards an Activation-Centric View of Neural Network Training

arXiv:2605.11570v1 Announce Type: new Abstract: Activation functions are what make deep networks expressive: without them, the model collapses to a linear map. Yet we still evaluate training mostly fr

researcharxiv-cs-lg

Premover: Fast Vision-Language-Action Control by Acting Before Instructions Are Complete

model-releasesarxiv-cs-ro

arXiv:2605.12160v1 Announce Type: new Abstract: Vision-Language-Action (VLA) policies are typically evaluated as if the user had finished typing or speaking before the robot begins acting. In real dep

Tutorials

Sharpen Your Flow: Sharpness-Aware Sampling for Flow Matching

arXiv:2605.11547v1 Announce Type: new Abstract: Flow matching models generate samples by numerically integrating a learned velocity field, with each integration step requiring a neural network evaluat

tutorialsarxiv-cs-lg

UHR-Micro: Diagnosing and Mitigating the Resolution Illusion in Earth Observation VLMs

arXiv:2605.12237v1 Announce Type: new Abstract: Vision-Language Models (VLMs) increasingly operate on ultra-high-resolution (UHR) Earth observation imagery, yet they remain vulnerable to a severe scal

XWOD: A Real-World Benchmark for Object Detection under Extreme Weather Conditions

arXiv:2605.11521v1 Announce Type: new Abstract: Autonomous driving and intelligent transportation systems remain vulnerable under extreme weather. The U.S. Federal Highway Administration reports that

A Paired Point-of-Care Ultrasound Dataset for Image Quality Enhancement and Benchmarking via a cGAN Baseline

arXiv:2605.08282v1 Announce Type: cross Abstract: Purpose: We aim to enhance the image quality of point-of-care ultrasound (POCUS) devices using deep learning and a novel paired dataset of POCUS and h

researcharxiv-cs-ai

A Scalable Entity-Based Framework for Auditing Bias in LLMs

arXiv:2601.12374v2 Announce Type: replace-cross Abstract: Existing approaches to bias evaluation in large language models (LLMs) trade ecological validity for statistical control, relying either on ar

Adaptive DNN Partitioning and Offloading in Heterogeneous Edge-Cloud Continuum

arXiv:2605.09623v1 Announce Type: cross Abstract: In recent years, the use of artificial intelligence on resource-constrained IoT devices has grown significantly. However, existing approaches to DNN p

researcharxiv-cs-ai

Advanced Tumor Segmentation in PET/CT Imaging: A Training Strategy Study with nnU-Net for AutoPET III

arXiv:2605.08161v1 Announce Type: new Abstract: Tumor segmentation in whole-body PET/CT imaging is crucial for precise disease evaluation and treatment planning. However, it remains challenging due to

researcharxiv-cs-cv

AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators

arXiv:2605.08647v1 Announce Type: cross Abstract: Multi-agent systems achieve state-of-the-art outcomes through peer collaboration. However, when an agent in the pipeline silently drops a constraint,

Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling

arXiv:2604.08178v2 Announce Type: replace Abstract: In classical Reinforcement Learning from Human Feedback (RLHF), Reward Models (RMs) serve as the fundamental signal provider for model alignment. As

Ambig-DS: A Benchmark for Task-Framing Ambiguity in Data-Science Agents

arXiv:2605.09698v1 Announce Type: new Abstract: As data-science agents shift from co-pilots to auto-pilots, silent misframing becomes a critical failure mode. Agents quietly commit to plausible but un

An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classification

arXiv:2603.02719v2 Announce Type: replace Abstract: As artificial intelligence systems move toward clinical deployment, ensuring reliable prediction behavior is fundamental for safety-critical decisio

Assessing Trustworthiness of AI Training Dataset using Subjective Logic -- A Use Case on Bias

arXiv:2508.13813v2 Announce Type: replace-cross Abstract: As AI systems increasingly rely on training data, assessing dataset trustworthiness has become critical, particularly for properties like fair