Search: “evaluation”

Rethinking Incompleteness: Formalizing Protocol Divergence and Train-Once Learning for Robust IMVC

arXiv:2606.04857v1 Announce Type: new Abstract: Standard IMVC evaluation retrains separate models for different missing-data configurations. We show that this paradigm obscures a fundamental vulnerabi

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

DGX agent

arXiv:2606.04739v1 Announce Type: cross Abstract: Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generatio

model-releasesarxiv-cs-ai

4 Jun 2026

RIDE: An Open Dataset and Benchmark for Train Delay Prediction

model-releasesarxiv-cs-lg

arXiv:2606.05070v1 Announce Type: new Abstract: Train delay prediction is an important problem for both passengers and railway operators, yet progress in the field remains difficult to assess due to t

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

arXiv:2606.04202v1 Announce Type: new Abstract: As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coor

Uncertainty Estimation using Variance-Gated Distributions

arXiv:2509.08846v2 Announce Type: replace-cross Abstract: Evaluation of per-sample uncertainty quantification from neural networks is essential for decision-making involving high-risk applications. A

Validity Threats for Foundation Model Research

arXiv:2606.05029v1 Announce Type: cross Abstract: Controlled experiments are the backbone of machine learning research, but at the scale of modern foundation models, they have become prohibitively exp

researcharxiv-cs-cl

Applications

Variance Reduction for Heavy-Tailed Monetization Metrics in Ranking Experiments via Post-Stratification

arXiv:2606.04110v1 Announce Type: new Abstract: Online evaluation of ranking and retrieval systems often relies on downstream monetization metrics such as app revenue or creator earnings. These metric

applicationsarxiv-cs-lg

WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia

arXiv:2507.03373v2 Announce Type: replace Abstract: Given Wikipedia's role as a trusted source of high-quality, reliable content, concerns are growing about the proliferation of low-quality machine-ge

What Are We Actually Benchmarking in Robot Manipulation?

model-releasesarxiv-cs-ro

arXiv:2606.04233v1 Announce Type: new Abstract: A robotics benchmark score measures success under one fixed evaluation setup, yet is routinely treated as evidence of general manipulation capability. W

CAPER: Clause-Aligned Process Supervision for Text-to-SQL

arXiv:2606.03327v1 Announce Type: cross Abstract: Text-to-SQL systems are typically evaluated by query-level execution correctness, but this terminal signal provides little guidance about which interm

ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

arXiv:2606.03157v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making

Consistent Yet Wrong: Evidence Insensitivity in Spatial Vision-Language Models

model-releasesarxiv-cs-cv

arXiv:2606.02742v1 Announce Type: new Abstract: Spatial reasoning is fundamental to robotics, autonomy, and embodied AI, yet modern vision-language models (VLMs) remain unreliable on metric distance q

Don't Gamble, GAMBLe: An Analytical Framework for AI-Driven Research Systems

arXiv:2606.02863v1 Announce Type: new Abstract: AI-Driven Research Systems (ADRS) -- systems coupling LLMs with automated evaluation to discover algorithms, proofs, and designs -- are being optimized

Safety

DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference

arXiv:2606.02982v1 Announce Type: cross Abstract: The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inf

safetyarxiv-cs-lg

DXA-Derived Skeletal Phenotypes and Hip Fracture Risk: A Backdoor-Adjusted Causal Analysis

arXiv:2606.02625v1 Announce Type: cross Abstract: Purpose: To compare dual-energy X-ray absorptiometry (DXA)-derived hip skeletal phenotypes in relation to hip fracture risk using prespecified confoun

Efficient ASR Training with Conversations that Never Happened

arXiv:2606.03957v1 Announce Type: cross Abstract: Conversational ASR for lower-resource languages and niche domains is limited by the scarcity of domain-matched multi-speaker training data. We propose

EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge

arXiv:2606.03363v1 Announce Type: new Abstract: Text-to-SQL enables natural language access to databases, and recent LLMs have substantially advanced its capabilities. Existing benchmarks such as Spid

EURO-5K: When Does Domain Pretraining Matter? Benchmarking Transformers for EU Reporting Obligation Extraction

arXiv:2606.02971v1 Announce Type: new Abstract: Extracting reporting obligations from EU legislation is critical for assessing and reducing regulatory reporting burden. However, distinguishing reporti

Fast Unlearning at Scale via Margin Self-Correction

arXiv:2606.02920v1 Announce Type: new Abstract: Language-model unlearning updates a trained model to behave as if it had not seen selected training examples, while preserving utility and avoiding cost

From Prompt to Service: An SLM-Based Agent Orchestration Gateway for AI-Driven Virtual Worlds

arXiv:2606.03557v1 Announce Type: new Abstract: As generative AI capabilities expand, AI-driven virtual worlds face a growing architectural challenge. Users interact through in-world interfaces in mul

Safety

HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models

arXiv:2606.03131v1 Announce Type: new Abstract: Reward models are central to large language model (LLM) alignment, but they remain vulnerable to reward hacking. To evaluate reward-model robustness, we

safetyarxiv-cs-lg

Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings

arXiv:2606.03365v1 Announce Type: new Abstract: Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs. Standard evaluation protocols emphasize rank-based m

Microsoft’s open trust stack runs on OpenInference

Microsoft's open trust stack for AI agents puts ASSERT and Agent Control Specification on top of OpenInference, connecting evaluation, runtime controls, and observability through a shared trace contra

agentsarize-ai

Pretraining Language Models on Historical Text

arXiv:2606.02991v1 Announce Type: cross Abstract: We introduce TypewriterLM, a 7.24B History language model (LM) trained exclusively on English text predating 1913. Developing History LMs requires add

Reasoning Structure of Large Language Models

arXiv:2606.03883v1 Announce Type: new Abstract: Large reasoning models (LRMs) are often evaluated using metrics such as final-answer accuracy or token count. However, identical scores on these metrics

RESCAST-100K: A Comprehensive Dataset for Cross-Domain Residential Load and Indoor Temperature Forecasting

model-releasesarxiv-cs-lg

arXiv:2606.02852v1 Announce Type: new Abstract: Accurate short-term forecasting of residential energy load and indoor temperature is essential for home energy management systems, grid-level demand res

Rethinking Molecular Text Representations for LLMs: An Empirical Study

arXiv:2606.03057v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for molecular tasks, but it remains unclear which molecular representation to use. We present a sys

scTranslation: A Comprehensive Benchmark for Single-Cell Multi-Omics Modality Translation

arXiv:2606.03906v1 Announce Type: new Abstract: Simultaneous measurement of multiple omics modalities in single cells enables researchers to gain a more comprehensive understanding of cellular states

Industry

S&P Global and Snowflake bring AI-driven financial analysis to qualitative investment research

Historically, investment research involved human analysts spending days reading and evaluating massive financial reports. Now, AI-driven financial analysis is covering all of that work in just a few m

industrysiliconangle

Testing the Test: Score-Direction Instability in Class-Split Anomaly Detection

arXiv:2606.02601v1 Announce Type: new Abstract: Within-dataset class-split evaluation is widely used as a proxy for fully unconditional out-of-distribution anomaly detection. We show that this protoco

The Agent's First Day: Benchmarking Learning, Exploration, and Scheduling in the Workplace Scenarios

arXiv:2601.08173v2 Announce Type: replace Abstract: The rapid evolution of Multi-modal Large Language Models (MLLMs) has advanced workflow automation; however, existing research mainly targets perform

agentsarxiv-cs-ai

The Unsampled Truth: Psychometrics in SLMs Measure Prompt Artifacts, Not Psychological Constructs

arXiv:2606.03357v1 Announce Type: cross Abstract: When prompting SLMs for psychometric assessments, researchers assume the outputs reflect semantic reasoning. We evaluate this premise across 13 open-w

Safety

Towards a Science of AI Agent Reliability

arXiv:2602.16666v3 Announce Type: replace Abstract: AI agents are increasingly deployed to execute important tasks. While rising accuracy scores on standard benchmarks suggest rapid progress, many age

safetyarxiv-cs-ai

TriEval: A Resource-Efficient Pipeline for LLM Bias, Toxicity, and Truthfulness Assessment

arXiv:2606.03036v1 Announce Type: new Abstract: LLMs have evolved from basic chatbots to the backbone of the AI ecosystem, now widely used in healthcare, schools, and government services. The domain-w

Industry

Trump plan to test AI models has a problem—US security teams were gutted by DOGE

Trump signed an executive order on June 2, 2026, establishing a voluntary framework for the federal government to evaluate risks posed by advanced AI systems before release and encouraging companies t

industryars-technica

Very cool work from the Quarq team Their agent is the current LongMemEval-S leader, and it's built on LangGraph Many such cases :)

Quarq's team has developed an AI agent built on LangGraph that currently leads the LongMemEval-S benchmark, demonstrating advances in long-context memory evaluation. The work represents a notable achi

agentsharrison-chase--x

VistaHop: Benchmarking Multi-hop Visual Reasoning for Visual DeepSearch

arXiv:2606.03273v1 Announce Type: cross Abstract: Visual DeepSearch requires multimodal large reasoning model (MLRM) agents to answer complex visual queries by repeatedly inspecting image regions, gro

With LangSmith Engine, systemic issues get surfaced automatically instead of getting buried in traces. @ollieelmgren from @ListenLabs on how…

With LangSmith Engine, systemic issues get surfaced automatically instead of getting buried in traces. @ollieelmgren from @ListenLabs on how LangSmith Engine changed the way his team evaluates their a

agentsharrison-chase--x

Tutorials

A Developer’s Guide to Managing Models, Cost and Quality in Microsoft Foundry

tutorialsmicrosoft-foundry

Learn a practical model lifecycle for Microsoft Foundry: select the right model, evaluate quality, optimize cost, operate safely, and improve as production needs change. The post A Developer’s Guide t

A Fiber Criterion for Representation Identifiability in Supervised Learning

arXiv:2606.01092v1 Announce Type: cross Abstract: Supervised learning evaluates predictors through their input-output behavior. When a predictor is implemented as a composition f=circ h, supervised ev

A Methodological Framework for Explicit Control of the Speed-Accuracy Trade-off in Brain-Computer Interfaces

arXiv:2606.00106v1 Announce Type: cross Abstract: Brain-computer interfaces (BCIs) are limited by low signal-to-noise ratio in modalities such as electroencephalography, which requires multiple trials

Acoustic and perceptual differences between standard and accented speech and their voice clones

arXiv:2604.01562v2 Announce Type: replace-cross Abstract: Voice cloning is often evaluated in terms of overall quality, but less is known about accent preservation and its perceptual consequences. We

Adaptive Dense Evidence Refinement for Video Relational Reasoning for VRR-QA Challenge

arXiv:2606.01104v1 Announce Type: new Abstract: VRR-QA evaluates whether video-language systems can infer spatial, temporal, viewpoint, depth, and visibility relations that are not always resolved by

researcharxiv-cs-cv

Industry

AI Governance Maturity Model: Matrix, Assessment, and Roadmap

This Databricks resource presents a framework for assessing and advancing an organization's AI governance capabilities through a maturity model structure. It likely provides a matrix for evaluating cu

industrydatabricks

Before and After Temperature: A Distributional View of Creative LLM Generation

arXiv:2606.01451v1 Announce Type: new Abstract: Reference-free evaluation of large language model (LLM) creativity relies on perplexity, entropy, and top-1 margin. We show that a much stronger signal

Benchmarking Local LLMs for Natural-Language-to-SQL Querying in Biopharmaceutical Manufacturing: An Empirical Benchmark on Consumer-Grade Hardware

arXiv:2606.01338v1 Announce Type: new Abstract: Biopharmaceutical manufacturing organizations operate under regulatory frameworks such as FDA guidance, EU Good Manufacturing Practice (GMP), and the EU

Beyond Rigid: Benchmarking Non-Rigid Video Editing

model-releasesarxiv-cs-cv

arXiv:2601.18340v2 Announce Type: replace Abstract: As video generation models are increasingly expected to manipulate physical dynamics, there is a growing need to move evaluation beyond appearance f

Beyond Text and Tables: Vision-Language Model Integration in ComProScanner for Extracting Materials Data from Scientific Figures with High Accuracy

arXiv:2606.00065v1 Announce Type: cross Abstract: Automated extraction of materials composition-property data from scientific literature has advanced considerably with the development of large languag