Search: “evaluation”

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

arXiv:2606.26346v1 Announce Type: new Abstract: Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-doma

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

Learning User Simulators with Turing Rewards

DGX agent

arXiv:2606.19336v2 Announce Type: replace Abstract: Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, rese

agentsarxiv-cs-cl

26 Jun 2026

RouterVLA: Turning Smoke Tests into Supervision for Heterogeneous VLA Selection

arXiv:2606.27355v1 Announce Type: new Abstract: We study whether pre-deployment evaluation rollouts can be reused to supervise policy selection. Robot teams routinely smoke test candidate vision-langu

safetyarxiv-cs-ro

SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context

model-releasesarxiv-cs-cl

arXiv:2606.26654v1 Announce Type: new Abstract: Personalized language-model assistants are often evaluated through a memory lens: can a model recall preferences users have explicitly stated in dialogu

The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

arXiv:2606.26529v1 Announce Type: cross Abstract: AI safety is evaluated by how reliably a model detects the hazards it is told to find, yet accidents often arise from the hazard no one specified. We

Theory-Scale Auto-Formalization of Logics for Computer Science

arXiv:2606.26525v1 Announce Type: new Abstract: Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-f

AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents

arXiv:2606.24893v1 Announce Type: new Abstract: For agents to learn continuously from interaction with the world at test time, they must be able to explore effectively, acquire new world knowledge and

agentsarxiv-cs-cl

AI translation of literary texts is 'fine', but readers still prefer human translations

arXiv:2606.26040v1 Announce Type: new Abstract: AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experienc

agentsarxiv-cs-cl

Tools

[AINews] It's Meta-Harness Summer

Meta released Harness, an open-source framework for evaluating and benchmarking AI model performance across diverse tasks and datasets. The tool enables standardized testing of language models and aim

toolslatent-space

ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

arXiv:2606.25207v1 Announce Type: cross Abstract: Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding

safetyarxiv-cs-cl

Fuzzy Quantification over OWL Ontologies and Knowledge Graphs

arXiv:2606.25778v1 Announce Type: new Abstract: This paper presents a versatile framework for evaluating fuzzy quantification queries over both standard and fuzzy ontologies as well as knowledge graph

researcharxiv-cs-ai

Geo-Strat-RL: Learning Geological Event Reasoning from Verifiable Tasks

arXiv:2606.25000v1 Announce Type: new Abstract: To evaluate whether vision-language models can reason about geological histories, it is necessary to construct observations for which the underlying pro

researcharxiv-cs-lg

LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

arXiv:2606.25057v1 Announce Type: new Abstract: The rapid growth of scientific submissions has pushed traditional peer review toward its scalability limits, motivating the exploration of large languag

safetyarxiv-cs-cl

Long-Term Simulation Exposes Cognitive-Developmental Risks in AI Companions

arXiv:2606.25396v1 Announce Type: new Abstract: AI companions powered by large language models increasingly interact with cognition-developing users, including children and adolescents, creating risks

Measuring Research Difficulty of Academic Papers: A Case Study in Natural Language Processing

arXiv:2606.25307v1 Announce Type: cross Abstract: With the rapid growth of the number of academic papers, systematically evaluating the difficulty of research and its relationship to academic impact o

applicationsarxiv-cs-cl

Tools

More on how we're constraining eval environments so that scores better reflect model intelligence: http://cursor.com/blog/reward-hacking-cod…

Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio

toolscursor--x

Proactive Systems in HCI and AI: Concepts, Challenges, and Opportunities

arXiv:2606.25149v1 Announce Type: cross Abstract: The last few years have seen a significant rise in interest in highly autonomous and proactive systems, fueled by advances in AI. Systems that anticip

agentsarxiv-cs-ai

Local Ai

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

arXiv:2606.24834v1 Announce Type: new Abstract: LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional co

local-aiarxiv-cs-ai

Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web

arXiv:2606.24236v1 Announce Type: cross Abstract: Visual assessment of residual plots is a common approach for diagnosing linear models, but it relies on manual evaluation, which does not scale well a

researcharxiv-cs-cv

BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling

arXiv:2606.20146v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions. In engi

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

arXiv:2606.24797v1 Announce Type: cross Abstract: Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on video question answering (VideoQA). Nevertheless, ex

EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

arXiv:2606.24422v1 Announce Type: new Abstract: We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of mo

Governed Shared Memory for Multi-Agent LLM Systems

arXiv:2606.24535v1 Announce Type: new Abstract: Multi-agent LLM environments require robust mechanisms for shared knowledge management. This paper formalizes the fleet-memory problem and identifies fo

Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World

The FFASR Leaderboard is a benchmarking platform introduced by Hugging Face for evaluating Automatic Speech Recognition (ASR) systems on real-world performance metrics. It provides a standardized eval

applicationshugging-face

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

model-releasesarxiv-cs-cl

arXiv:2606.24530v1 Announce Type: new Abstract: We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

model-releasesarxiv-cs-cl

arXiv:2606.24648v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches

Promise and challenges of heart chamber segmentation from non-contrast CT scans using contrastive unpaired image translation: a feasibility study

arXiv:2606.23879v1 Announce Type: cross Abstract: Purpose: To evaluate the feasibility and challenges of heart chamber segmentation from non-contrast CT scans using contrastive unpaired image translat

researcharxiv-cs-ai

REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs

arXiv:2606.23892v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems, whe

What Does ODRL Mean? A Cross-Level Ontological Grounding of Permissions, Prohibitions, and Duties in UFO-L

arXiv:2606.24344v1 Announce Type: cross Abstract: ODRL policy evaluators produce verdicts, but say nothing about the normative positions a policy brings into existence, the authority structures those

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

arXiv:2606.24619v1 Announce Type: new Abstract: Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natur

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

arXiv:2602.14257v2 Announce Type: replace-cross Abstract: While Large Language Model (LLM) agents have made remarkable progress on complex reasoning, evaluating them in real-world environments remains

An Empirical Study of OpenPangu Quantization on Ascend NPUs

arXiv:2606.21257v1 Announce Type: new Abstract: OpenPangu models are attractive targets for private and domestic large-language-model deployment, yet their robustness under aggressive post-training qu

researcharxiv-cs-lg

Biological Sex Determination in Cadavers Using Deep Learning Algorithms from Computed Tomography Images of Pelvis and Skull

arXiv:2606.22515v1 Announce Type: new Abstract: Sexual identification of decomposed cadavers challenges traditional methods dependent on visual anthropological analysis. This study evaluates state-of-

researcharxiv-cs-cv

CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales

arXiv:2606.21949v1 Announce Type: new Abstract: Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, fe

Chem2Gen-Bench: Benchmarking Chemical-to-Genetic Translation in Perturbation Response Space

arXiv:2606.21109v1 Announce Type: new Abstract: Virtual-cell and perturbation models are increasingly used to predict cellular responses for biomedical discovery, but chemical and genetic perturbation

Direct Raw Audio Signal Processing via Reservoir Computing: An Investigation into 'Feature-Free' Architectures

arXiv:2606.21335v1 Announce Type: cross Abstract: This paper evaluates Reservoir Computing (RC) as an autonomous, 'feature-free' framework for audio processing, designed to eliminate traditional, hand

agentsarxiv-cs-lg

DiT-Reward: Generative Representations for Text-to-Image Reward Modeling

arXiv:2606.23626v1 Announce Type: new Abstract: Can representations learned for image generation also support the evaluation of generated images? We study text-to-image reward prediction as a downstre

safetyarxiv-cs-lg

Tutorials

Full-Body Golf Swing Kinematic Reconstruction From a Smartwatch IMU

arXiv:2606.22876v1 Announce Type: new Abstract: Quantitative measurement of the golf swing is critical for evaluating technique and enabling individualized feedback. However, existing methods are impr

tutorialsarxiv-cs-cv

Policy4OOD: A Knowledge-Guided World Model for Policy Intervention Simulation against the Opioid Overdose Crisis

arXiv:2602.12373v2 Announce Type: replace Abstract: The opioid epidemic remains one of the most severe public health crises in the United States, yet evaluating policy interventions before implementat

Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts

arXiv:2606.23175v1 Announce Type: new Abstract: AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argue

agentsarxiv-cs-lg

Reference-Free Assessment of Physical Consistency in World Model-based Video Generation

arXiv:2606.22363v1 Announce Type: cross Abstract: We introduce reference-free measures for evaluating the physical consistency of generated videos, combining relative and absolute approaches to assess

applicationsarxiv-cs-lg

Skill Coverage: A Test Adequacy Metric for Agent Skills

arXiv:2606.20659v1 Announce Type: cross Abstract: Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts. Existing evaluations pr

Stage-dependent integer-binary encoding in factorization-machine black-box optimization

arXiv:2606.23188v1 Announce Type: new Abstract: Black-box optimization (BBO) deals with problems where objective functions lack explicit analytical forms and are expensive to evaluate. Factorization m

researcharxiv-cs-lg

Embed the world: Multimodal AI for searchable aerial imagery at scale

model-releasesaws-ml-blog

In this post, we walk through the problem space, our architecture on Amazon Bedrock and Amazon OpenSearch Serverless, the evaluation methodology we built on OpenStreetMap ground truth, four experiment

22 Jun 2026

Few-Shot Resampling for Scalable Statistically-Sound Data Mining

arXiv:2606.11235v1 Announce Type: new Abstract: A key step in knowledge discovery is the evaluation of data mining results. In several applications, including pattern mining, graph analysis, and other

applicationsarxiv-cs-lg

Geometric Metrics and LLMs: What They Measure and When They Work

arXiv:2509.25359v2 Announce Type: replace-cross Abstract: We present a systematic stress-test of geometric metrics for LLM evaluation. Rank-based geometric properties of internal representations have

researcharxiv-cs-ai

How Seemingly Inconsequential Design Choices Dictate Performance of LLMs in Pathology

arXiv:2606.12407v1 Announce Type: new Abstract: General-purpose large language models (LLMs) are routinely used as baselines when evaluating specialized pathology models on whole-slide images (WSIs).

IntElicit: Eliciting and Assessing Contextualized Creativity via Dialogue Policy Optimization

arXiv:2606.12086v1 Announce Type: new Abstract: Contextualized assessment offers high ecological validity for evaluating creativity but introduces a critical challenge: observed performance may be con