Search: “evaluation”

arXiv:2504.17768v3 Announce Type: replace Abstract: Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain

applicationsarxiv-cs-cl

24 Jun 2026

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

You Don't Need to Run Every Eval

DGX agent

arXiv:2606.24020v1 Announce Type: new Abstract: A modern model release reports scores on 40+ benchmarks and the same evaluations were run many more times before it: to track training progress, compare

model-releasesarxiv-cs-lg

24 Jun 2026

An intuitive rearranging of the Yates covariance decomposition for probabilistic verification of forecasts with the Brier score

arXiv:2603.05544v2 Announce Type: replace-cross Abstract: Proper scoring rules are essential for evaluating probabilistic forecasts. We propose a simple algebraic rearrangement of the Yates covariance

Applications

Beyond a Single Light: A Large-Scale Aerial Dataset for Urban Scene Reconstruction Under Varying Illumination

arXiv:2512.14200v2 Announce Type: replace Abstract: Recent advances in Neural Radiance Fields and 3D Gaussian Splatting have demonstrated strong potential for large-scale UAV-based 3D reconstruction t

applicationsarxiv-cs-cv

Can Single-View Mesh Reconstruction Generalize to Robot Camera Rotation?

arXiv:2606.22987v1 Announce Type: new Abstract: Single-view mesh reconstruction predicts object meshes and spatial layouts from a single observation, making it attractive for fast robot spatial reason

researcharxiv-cs-cv

Closing the Calibration Gap in Semantic Caching

arXiv:2606.19719v2 Announce Type: replace-cross Abstract: Semantic caching cuts LLM inference costs by serving a cached response to semantically similar queries. Standard practice evaluates these syst

Coherence Under Commitment: Probing Generalization and Vacuous Memorization in LLM Logical Reasoning

arXiv:2606.21083v1 Announce Type: cross Abstract: Large language models (LLMs) deployed for logical reasoning in knowledge-intensive domains exhibit a subtle but critical failure: coherence can be vac

CVSBench: A Comprehensive Benchmark for Cross-view Spatial Reasoning and Dreaming

arXiv:2606.22476v1 Announce Type: new Abstract: Humans can effortlessly reason about scenes across different viewpoints, yet it remains unclear whether Vision-Language Models (VLMs) possess similar cr

Agents

Democratizing and accelerating AI-driven pathology research through agentic intelligence

arXiv:2606.20677v1 Announce Type: cross Abstract: Computational pathology has advanced rapidly with the emergence of foundation models, yet widespread adoption remains limited by substantial technical

agentsarxiv-cs-cv

EEG Benchmarking Needs a Task Specification Layer: NeuroDoc for Rulebook-Guided, Executable Benchmark Construction

arXiv:2606.22925v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models increasingly rely on multi-dataset training and evaluation, yet public EEG datasets still lack a shared t

Agents

Evals are the new PRDs, @hwchase17 It might be my takeaway of the @LangChain Chicago meetup tonight.

Harrison Chase suggests that evaluations (evals) are becoming as critical to AI/LLM development as Product Requirements Documents (PRDs) traditionally were, implying a shift in how teams validate and

agentsharrison-chase--x

Hedgementation = Hedgerow Segmentation: A Remote Sensing Benchmark

arXiv:2606.23615v1 Announce Type: new Abstract: We propose Hedgementation: a new benchmark to evaluate machine learning models for hedgerow mapping from remote sensing data at country scale and 10m^2

How Well Do Self-Supervised Speech Models Encode Age and Gender in Children's Speech? A Layer-Wise Analysis Across Multiple Architectures

arXiv:2606.22177v1 Announce Type: cross Abstract: Self-supervised learning (SSL) models have become a central component of modern speech processing systems, as they enable the learning of rich acousti

Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset

arXiv:2606.22382v1 Announce Type: cross Abstract: Purpose: To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE, a large-scale public

Measuring Intent Comprehension in LLMs

arXiv:2506.16584v3 Announce Type: replace-cross Abstract: People judge interactions with large language models (LLMs) as successful when outputs match what they want, not what they type. Yet LLMs are

MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models

arXiv:2606.21194v1 Announce Type: new Abstract: Medical Vision-Language Models (Med-VLMs) achieve strong expert-level performance, yet their ability to generate patient-accessible descriptions remains

Metric-Dependent Annotation Saturation for Learning from Label Distributions

researchapple-ml-research

When annotators disagree on a label, the disagreement itself carries signal—and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distrib

Local Ai

MIRAGE: Stealthy Visual Prompt Injection for Vulnerability Detection in Web Agents

arXiv:2606.20717v1 Announce Type: new Abstract: Multimodal Large Language Model (MLLM)-based web agents provide practical, high-precision solutions for visual browser automation; however, they inheren

local-aiarxiv-cs-cv

MMGist: A Comprehensive Multimodal Benchmark for 2027

arXiv:2606.22437v1 Announce Type: new Abstract: We conduct a systematic study of 18 widely used vision-language benchmarks and identify three major issues: 1) many items do not rely on visual cues and

MotionHalluc: Diagnosing Kinematic Hallucinations in Fine-Grained Motion Reasoning

arXiv:2606.23061v1 Announce Type: new Abstract: Motion instruction generation in cross-video comparison aims to produce corrective feedback that describes the differences between a query and a referen

Hardware

Multi-cancer detection using a computationally efficient CNN with transfer learning

arXiv:2606.22400v1 Announce Type: new Abstract: This study introduces a computationally efficient convolutional neural network (CNN) architecture enhanced with transfer learning for multi-cancer detec

hardwarearxiv-cs-cv

Tutorials

Native space based pipelines outperform template space based pipeline in subcortical segmentation

arXiv:2606.21463v1 Announce Type: new Abstract: Accurate segmentation of subcortical regions is critical for neurosurgical planning and functional research. Most automated methods rely on template spa

tutorialsarxiv-cs-cv

Hardware

ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)

ParallelKernelBench is a benchmark from Together AI that evaluates frontier large language models' ability to write optimized multi-GPU CUDA kernels, revealing significant limitations in current LLMs'

hardwaretogether-ai-blog

R2HandoverSim: A Simulation Framework and Benchmark for Robot-to-Human Object Handovers

model-releasesarxiv-cs-ro

arXiv:2606.21011v1 Announce Type: new Abstract: We present R2HandoverSim, a simulation benchmark for robot-to-human (R2H) object handovers. Although R2H handover methods have advanced rapidly, the lac

Rethinking Molecular Graph Backdoors under Chemistry-aware Admission

arXiv:2606.23361v1 Announce Type: new Abstract: Backdoor attacks on molecular graph neural networks (GNNs) are typically evaluated as abstract graph edits, but real molecular learning pipelines do not

Tutorials

Selective Ensemble Based on Preference-Directed Multi-Objective Bandits

arXiv:2606.21929v1 Announce Type: new Abstract: Selective ensemble for modern machine learning systems requires choosing promising model candidates under limited evaluation budgets, while downstream t

tutorialsarxiv-cs-lg

Set-based v.s. Distribution-based Representations of Epistemic Uncertainty: A Comparative Study

arXiv:2602.22747v2 Announce Type: replace Abstract: Epistemic uncertainty in neural networks is commonly modeled using two second-order paradigms: distribution-based representations, which rely on pos

Hardware

The Rise of Alternative Cloud Providers: Why Enterprises Are Rethinking Cloud Strategy

Enterprises are increasingly evaluating alternative cloud providers beyond the major hyperscalers (AWS, Azure, Google Cloud) due to cost, performance, and vendor lock-in concerns. This shift reflects

hardwarevultr

Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays

arXiv:2606.20852v1 Announce Type: new Abstract: Inference-time engineering can alter model behavior without fine-tuning. However, its utility for improving diagnostic performance in medical vision-lan

Weighted Score-Oriented Losses for Temporally Localized Event Prediction

arXiv:2606.23145v1 Announce Type: new Abstract: Operational event-detection systems are rarely assessed by pointwise accuracy alone. In anomaly detection, changepoint detection, and warning systems, t

When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration

arXiv:2606.20724v1 Announce Type: cross Abstract: Long-horizon web agents often fail in ways hidden by final-answer evaluation: they may visit useful pages, produce a well-formed answer, and terminate

Tools

Introducing The Blind Test. Two landing pages. One built by GLM 5.2 and one by Opus 4.8. Can you tell which is which? It's very difficult to…

Together AI conducted a blind test comparing two landing pages—one created by GLM 5.2 and one by Opus 4.8—to evaluate whether users could distinguish between AI-generated designs. The test highlights

toolstogether-ai--x

22 Jun 2026

Industry

So true

So true Many people think any given ML project is 99% training. In reality, it’s 50% evaluation, 40% data cleaning, 8% integration, and 2% training. The first two set the noise floor for learning. No

industryelon-musk--x

20 Jun 2026

TSAuditor: A time-series auditing framework [P]

researchr-machinelearning

TSAuditor is a contamination auditing framework for time series foundation models that uses probe adaptation dynamics to detect when evaluation datasets may have been exposed during pretraining, valid

20 Jun 2026

Tools

also hearing from other customers that GLM 5.2 approaches GPT 5.5 on their evals + use cases

Fireworks AI reports that GLM 5.2, their language model, demonstrates performance comparable to GPT 5.5 across various evaluation benchmarks and real-world use cases. This claim suggests GLM 5.2 is co

toolsfireworks-ai--x

19 Jun 2026

A Five-Plane Reference Architecture for Runtime Governance of Production AI Agents

arXiv:2606.12320v1 Announce Type: new Abstract: Enterprise security was built to govern data boundaries: the protected surface was data at rest and in transit, and the controls -- access control, data

AI Coding Agents Can Reproduce Social Science Findings

model-releasesarxiv-cs-cl

arXiv:2606.11447v1 Announce Type: new Abstract: Recent anecdotal evidence suggests that AI coding agents can reproduce published findings when provided with original data and code; yet systematic eval

Safety

Beyond Third-Person Audits: Situated Interaction Auditing for User-Centered LLM Bias Research

arXiv:2606.12247v1 Announce Type: cross Abstract: Research on bias in large language models (LLMs) has predominantly focused on third-person audits, which study how models represent or evaluate demogr

safetyarxiv-cs-cl

Local Ai

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

arXiv:2606.11672v1 Announce Type: cross Abstract: This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model

local-aiarxiv-cs-ai

FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback

model-releasesarxiv-cs-cl

arXiv:2601.04203v2 Announce Type: replace Abstract: We present FronTalk, a benchmark for front-end code generation that pioneers the study of a unique interaction dynamic: conversational code generati

OCSVM-Guided Representation Learning for Unsupervised Anomaly Detection

arXiv:2507.21164v2 Announce Type: replace-cross Abstract: Unsupervised anomaly detection (UAD) aims to detect anomalies without labeled data, a necessity in many machine learning applications where an

Safety

Scenario-based Probing and Steering Cultural Values in Large Language Models--Extended Version

arXiv:2606.11399v1 Announce Type: new Abstract: Large Language Models (LLMs) are deployed across cultural contexts but often reflect homogenized values inherited from training data. Evaluations of cul

safetyarxiv-cs-cl

Agents

Search Discipline for Long-Horizon Research Agents

arXiv:2606.11522v1 Announce Type: new Abstract: Autoresearch agents now propose, evaluate, and select scientific candidates against a metric, and that metric is usually an aggregate reduced over a het

agentsarxiv-cs-ai

Understanding Cross-Sensor Feature Variations for Generalizable 3D Perception

arXiv:2606.11573v1 Announce Type: new Abstract: Radar-camera BEV perception often suffers from degraded performance when evaluated across datasets, as changes in driving scenes, sensor configurations,

researcharxiv-cs-cv

Applications

uva-irlab-conv at SemEval-2026 Task 8: Multi-Turn RAG with Learned Sparse Retrieval and Listwise Reranking

arXiv:2606.11945v1 Announce Type: new Abstract: This report describes our participation in SemEval-2026 Task 8 on multi-turn retrieval and question answering. The task evaluates conversational systems

applicationsarxiv-cs-cl

A Controlled Audit of Pretraining Contamination in Public Medical Vision-Language Benchmarks

arXiv:2606.10066v1 Announce Type: cross Abstract: Medical vision-language models (VLMs) are evaluated on public benchmarks whose images and question-answer pairs have been freely downloadable for year

researcharxiv-cs-ai

10 Jun 2026

Alignment Collapse Under KV Cache Quantization: Diagnosis and Mitigation

arXiv:2606.09864v1 Announce Type: cross Abstract: Key-value (KV) cache quantization is widely used to reduce Large Language Model (LLM) inference memory, yet existing evaluations solely focus on measu

10 Jun 2026

Benchmarking Knowledge Editing using Logical Rules

arXiv:2606.10554v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly deployed in real-world applications that require access to up-to-date knowledge. However, retraining LLM