Search: “evaluation”

Confounding Masquerading as Improvement: A Systematic Evaluation of Offline Reinforcement Learning for Stroke Antithrombotic Treatment in a 129,000-Patient Registry

arXiv:2608.30442v1 Announce Type: new Abstract: Recent offline reinforcement learning (RL) studies report policies that outperform physician decisions on clinical outcomes. We conduct a systematic, pa

safetyarxiv-cs-lg

1 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

RoboPhys-3D: A Comprehensive Embodied World Model Evaluation via 3D Reconstruction

DGX agent

arXiv:2608.28718v1 Announce Type: cross Abstract: Video world models increasingly serve as data engines, action planners, and simulators for embodied AI, but conventional embodied world model (EWM) be

model-releasesarxiv-cs-ai

1 Sept 2026

Simulation-Based Evaluation of Energy-Constrained Quantum-Classical Competition

arXiv:2308.08025v2 Announce Type: replace-cross Abstract: This paper develops a simulation-based framework for evaluating the energy implications of quantum and classical computing firms competing in

1 Sept 2026

TaxCE : A Framework for Automated Taxonomy Construction and Evaluation at Scale

arXiv:2608.30614v1 Announce Type: new Abstract: Organizing unstructured feedback text into hierarchical taxonomy is a fundamental challenge in NLP, particularly in domains where feedback arrives at ma

researcharxiv-cs-cl

1 Sept 2026

SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction

model-releasesarxiv-cs-cl

arXiv:2608.27461v1 Announce Type: new Abstract: Relational reasoning requires the process of perceptual understanding, comparing, and integrating the underlying relationships between concepts. This ab

31 Aug 2026

Banger paper from Apple. If you build MCP servers, this can help you turn your specification into an evaluation suite. (bookmark it) It's ac…

Banger paper from Apple. If you build MCP servers, this can help you turn your specification into an evaluation suite. (bookmark it) It's actually a really neat idea that's easy to implement. And it s

model-releasesdair-ai--x

29 Aug 2026

Leakage-Free Evaluation and Distribution-Robust Spatio-Temporal Graph Learning for Inductive Kriging

arXiv:2509.23631v2 Announce Type: replace Abstract: Inductive kriging estimates values at unobserved locations from sparse sensor data, enabling continuous field reconstruction when dense deployment i

researcharxiv-cs-lg

28 Aug 2026

STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation

arXiv:2505.20781v2 Announce Type: replace-cross Abstract: Off-policy evaluation (OPE) estimates the performance of a target policy using offline data collected from a behavior policy, and is crucial i

safetyarxiv-cs-lg

28 Aug 2026

TraceBench: Controlled Evaluation of LLM Agents for Time-Series Root-Cause Attribution

arXiv:2608.27182v1 Announce Type: new Abstract: LLM agents are increasingly applied to anomaly detection and root-cause analysis in time-series observations collected from real-world systems; however,

28 Aug 2026

Design and Evaluation of a Compliant Quasi Direct Drive End-effector for Safe Robotic Ultrasound Imaging

arXiv:2410.03086v2 Announce Type: replace Abstract: Robot-assisted ultrasound scanning promises to advance autonomous and accessible medical imaging. However, ensuring patient safety and compliant hum

safetyarxiv-cs-ro

26 Aug 2026

LLM-Guided Contextual Action Evaluation for Operational Decisions in Industrial Processes

arXiv:2608.24156v1 Announce Type: cross Abstract: Industrial actor--critic methods usually represent continuous actions as anonymous numerical coordinates. They must therefore learn from limited inter

safetyarxiv-cs-ai

26 Aug 2026

SIREN-Bench: Behavior-Driven Generation and Evaluation of Emergency-Vehicle Interactions

model-releasesarxiv-cs-ro

arXiv:2608.24094v1 Announce Type: new Abstract: Emergency vehicles (EMVs) can reorganize surrounding traffic as civilian vehicles brake, change lanes, or form rescue corridors in response to their pas

26 Aug 2026

Counterfactual Evaluation of Temporal Observation Protocols

arXiv:2608.22221v1 Announce Type: new Abstract: We study counterfactual protocol evaluation: whether data collected under a realised observation protocol determine the predictive value of alternatives

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

model-releasesarxiv-cs-cv

arXiv:2409.16183v2 Announce Type: replace Abstract: Radiology is a vital and complex component of modern clinical workflow and covers many tasks. Recently, vision-language (VL) foundation models in me

Explainable Adaptive Zero Trust Framework for AWS with Adversarial Robustness Evaluation

arXiv:2608.21477v1 Announce Type: cross Abstract: Cloud environments built on Amazon Web Services face a structural security vulnerability: once a credential passes authentication, the resulting sessi

applicationsarxiv-cs-lg

How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

model-releasesarxiv-cs-cl

arXiv:2608.23448v1 Announce Type: new Abstract: This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradig

Large-Scale Evaluation of Advanced Imputation Methods for Missing Values in Smart Meter Data

arXiv:2608.21638v1 Announce Type: new Abstract: Accurate and reliable collection of electricity consumption data through Advanced Metering Infrastructure (AMI) is of great importance for the operation

applicationsarxiv-cs-lg

LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications

arXiv:2608.23058v1 Announce Type: new Abstract: Large language models (LLMs) now support forecasting systems that combine language-based reasoning with temporal data, evidence retrieval, external tool

// There Is No Neutral Harness // Great work discussing some of the issues in harness evaluation. Twelve open-weight models answer the same …

// There Is No Neutral Harness // Great work discussing some of the issues in harness evaluation. Twelve open-weight models answer the same 3,679 items from ARC, HellaSwag, MMLU, and TruthfulQA under

model-releasesdair-ai--x

aiXamine: Unified Black-Box Evaluation of Cross-Dimensional Trade-offs in LLM Safety, Security, and Privacy

arXiv:2608.20554v1 Announce Type: cross Abstract: The critical failure modes in deployed large language models (LLMs) are cross-dimensional: a model can score 99.3 in safety alignment while refusing o

safetyarxiv-cs-lg

Can Scientific Claims Be Removed from Large Language Models? A Systematic Evaluation of Claim-Level Unlearning

arXiv:2608.20960v1 Announce Type: new Abstract: Language models (LMs) are trained on static scientific corpora, whereas scientific knowledge continuously evolves through correction and revision. Scien

Efficient Self-Evaluation for Diffusion Language Models via Sequence Regeneration

arXiv:2603.02760v2 Announce Type: replace-cross Abstract: Diffusion large language models (dLLMs) have recently attracted significant attention for their ability to enhance diversity, controllability,

Rigorous Evaluation of Large Language Models for Malaria Drug Discovery: Trade-offs in Performance, Scale, and Resource Utility

arXiv:2608.20418v1 Announce Type: cross Abstract: We introduce Malaria-Instruct, a curated instruction-following dataset derived from the ChEMBL Legacy Malaria corpus for Malaria virtual screening, an

Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation

arXiv:2608.19299v1 Announce Type: new Abstract: Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management ha

21 Aug 2026

AI evaluation lab Irregular's report on its role in hacking incidents involving OpenAI, Anthropic, and Meta models faces criticism over unanswered questions (Alexander Martin/The Record)

Alexander Martin / The Record: AI evaluation lab Irregular's report on its role in hacking incidents involving OpenAI, Anthropic, and Meta models faces criticism over unanswered questions — The compan

applicationstechmeme

Comprehensive framework for evaluation of deep neural networks in detection and quantification of lymphoma from PET/CT images: clinical insights, pitfalls, and observer agreement analyses

arXiv:2311.09614v5 Announce Type: replace-cross Abstract: This study addresses critical gaps in automated lymphoma segmentation from PET/CT images, focusing on issues often overlooked in existing lite

Eval4Sim: An Evaluation Framework for Persona Simulation

arXiv:2603.02876v2 Announce Type: replace Abstract: Large Language Model personas, explicit profiles specifying a user's attributes, preferences, and behavioural tendencies, are increasingly used to s

safetyarxiv-cs-cl

From Global Benchmarks to Local Evaluations: Benchmarking LLMs for the German Public Sector

arXiv:2608.17827v1 Announce Type: new Abstract: Public institutions face a persistent challenge in selecting LLMs suited to their specific context. Existing benchmarks, however, are of limited use as

researcharxiv-cs-cl

Agent Gym: A Framework for Continuous Evaluation and Evolution of LLM Agents Through Human-in-the-Loop Feedback

arXiv:2608.15591v1 Announce Type: new Abstract: Large Language Model (LLM) agents deployed in production environments face a fundamental tension: the agent's behavior is frozen at deployment time, whi

safetyarxiv-cs-ai

An Evaluation Framework for National AI Regulation

arXiv:2608.15417v1 Announce Type: cross Abstract: Governments use laws, institutions, funding programs and nonbinding guidance to shape how AI is developed and used. Comparing these national approache

safetyarxiv-cs-ai

Conditional Evaluation of Language Models with Cheap Auxiliary Signals

arXiv:2608.16210v1 Announce Type: new Abstract: Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap aux

How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

arXiv:2608.14905v1 Announce Type: new Abstract: AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry

agentsarxiv-cs-cl

The Benchmark Trap: Structures of Power and Injustice in AI Evaluations

arXiv:2608.15326v1 Announce Type: new Abstract: Artificial intelligence (AI) benchmarks are not neutral tools of evaluation but socio-technical artefacts that shape competition, power, and research pr

When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation

arXiv:2608.14940v1 Announce Type: new Abstract: Current agent evaluations score models on the state visible at the end of a stopped run which they count as one trial. However, interpreting the score a

agentsarxiv-cs-ai

Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models

arXiv:2510.25577v2 Announce Type: replace-cross Abstract: Recent advances in Speech Foundation Models (SFMs) enable direct processing of raw audio, allowing models to respond to subtle paralinguistic

PHASE: Passive Human Activity Simulation Evaluation

arXiv:2507.13505v2 Announce Type: replace-cross Abstract: Cybersecurity simulation environments, such as cyber ranges, honeypots, and sandboxes, require realistic human behavior to be effective, yet n

applicationsarxiv-cs-ai

Style or Signature? Artist-Disjoint Evaluation of Style Classification in Frozen Vision Embeddings

arXiv:2608.14435v1 Announce Type: new Abstract: Frozen image embeddings from models such as CLIP are increasingly used to classify paintings by art-historical style, with high reported accuracy. We as

researcharxiv-cs-cv

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

arXiv:2608.13835v1 Announce Type: new Abstract: Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning pers

researcharxiv-cs-cl

SF-based Vals, which develops evaluations and benchmarks to test AI models on real-world tasks, raised a 40M Series A led by a16z at a 400M valuation (Abhinaya Prabhu/Tech Funding News)

Abhinaya Prabhu / Tech Funding News: SF-based Vals, which develops evaluations and benchmarks to test AI models on real-world tasks, raised a 40M Series A led by a16z at a 400M valuation — - Vals AI r

applicationstechmeme

15 Aug 2026

Tutorials

Evaluation of Clinically Steerable Retinal Image Generation from Foundation Model Latent Spaces

arXiv:2608.13455v1 Announce Type: new Abstract: Medical foundation models learn latent representations of clinically meaningful phenotypes, yet their ability to support controllable image generation r

tutorialsarxiv-cs-cv

Incremental Evaluation and Training in Relational Deep Learning

arXiv:2608.13023v1 Announce Type: new Abstract: Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs to enable end-to-end representation learning. However, pr

applicationsarxiv-cs-lg

Labels Are Not Endpoints: Treatment Leakage and Construct Validity in MCP Agent Security Evaluation

arXiv:2608.12880v1 Announce Type: cross Abstract: Security evaluations of tool-using agents often equate stored labels with behavioral facts. We audit a preserved campaign by tracing 10,200 execution

agentsarxiv-cs-ai

Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation

arXiv:2608.13337v1 Announce Type: new Abstract: Sparse autoencoders are meant to name the things a language model computes, and the usual way to check that a latent matters is to switch it off and see

Benchmark-Based Comparative Assessment of Publicly Benchmarked Indian Foundation Models: A Capability and Evaluation-Maturity Framework

arXiv:2608.11891v1 Announce Type: cross Abstract: Governments increasingly fund indigenous foundation models to strengthen national AI capability, digital sovereignty, and multilingual computing. Asse

13 Aug 2026

A Comparative Evaluation of Deep Learning Object Detection Models on a Real-World Multi-Plant Dataset from Africa

arXiv:2608.11053v1 Announce Type: cross Abstract: The application of computer vision in agriculture has shown significant potential for improving crop monitoring and precision farming. However, many e

applicationsarxiv-cs-ai

// Actions Speak Louder Than Words // Multilingual agent evaluation compares final answers and throws the trajectory away. The trajectory fi…

// Actions Speak Louder Than Words // Multilingual agent evaluation compares final answers and throws the trajectory away. The trajectory fixes cost, latency, failure mode, and auditability. New resea

safetydair-ai--x

OpenPM: Auditable Point-in-Time Evaluation for LLM Portfolio-Management Agents

arXiv:2608.09988v1 Announce Type: cross Abstract: Large language models are increasingly used to read markets, assess risk, and allocate capital. However, reported results for LLM trading agents can b

agentsarxiv-cs-cl

The Evaluation Protocol Determines the Result: An Independent Reproduction of LeWorldModel on TwoRoom

arXiv:2608.10145v1 Announce Type: new Abstract: LeWorldModel trains a latent world model with a prediction loss and a single anti-collapse regulariser, and reports approximately 87% of goals reached o