Search: “evaluation”

Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform

arXiv:2606.26590v1 Announce Type: new Abstract: Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

DGX agent

arXiv:2606.26535v1 Announce Type: cross Abstract: Current VLM evaluations often conflate language priors with genuine spatial reasoning. To address this, we introduce CRISP, a novel structural-diagnos

safetyarxiv-cs-ai

26 Jun 2026

From query to action: Introducing SQL alerting in Cloud Monitoring Observability Analytics

Traditional alerting systems often force a compromise: you can either alert immediately on simple, noisy log events, or monitor rigid, pre-configured metrics that fail when faced with data with many u

safetygoogle-cloud-ai

HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models

arXiv:2606.27187v1 Announce Type: cross Abstract: Large vision-language models (LVLMs) have recently shown immense potential in automated content moderation, sparking growing interest in developing ha

How Do Tool-Augmented LLM Agents Perform on Real-World Energy Analytics Tasks?

arXiv:2606.26346v1 Announce Type: new Abstract: Agentic benchmarks have emerged across general-purpose and domain-specific settings, including finance, coding, law, and drug discovery, yet energy-doma

Learning User Simulators with Turing Rewards

arXiv:2606.19336v2 Announce Type: replace Abstract: Learning to simulate human users in interactive settings could advance the training of agent assistants, evaluation of personalization systems, rese

agentsarxiv-cs-cl

RouterVLA: Turning Smoke Tests into Supervision for Heterogeneous VLA Selection

arXiv:2606.27355v1 Announce Type: new Abstract: We study whether pre-deployment evaluation rollouts can be reused to supervise policy selection. Robot teams routinely smoke test candidate vision-langu

safetyarxiv-cs-ro

SocialPersona: Benchmarking Personalized Profiling and Response with Multimodal Social-Media Context

arXiv:2606.26654v1 Announce Type: new Abstract: Personalized language-model assistants are often evaluated through a memory lens: can a model recall preferences users have explicitly stated in dialogu

The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

arXiv:2606.26529v1 Announce Type: cross Abstract: AI safety is evaluated by how reliably a model detects the hazards it is told to find, yet accidents often arise from the hazard no one specified. We

Theory-Scale Auto-Formalization of Logics for Computer Science

arXiv:2606.26525v1 Announce Type: new Abstract: Auto-formalization is critical for scalable formal verification, but existing progress largely focuses on isolated statements, while theory-scale auto-f

AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents

arXiv:2606.24893v1 Announce Type: new Abstract: For agents to learn continuously from interaction with the world at test time, they must be able to explore effectively, acquire new world knowledge and

agentsarxiv-cs-cl

AI translation of literary texts is 'fine', but readers still prefer human translations

arXiv:2606.26040v1 Announce Type: new Abstract: AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experienc

agentsarxiv-cs-cl

Tools

[AINews] It's Meta-Harness Summer

Meta released Harness, an open-source framework for evaluating and benchmarking AI model performance across diverse tasks and datasets. The tool enables standardized testing of language models and aim

toolslatent-space

ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments

arXiv:2606.25207v1 Announce Type: cross Abstract: Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding

safetyarxiv-cs-cl

Fuzzy Quantification over OWL Ontologies and Knowledge Graphs

arXiv:2606.25778v1 Announce Type: new Abstract: This paper presents a versatile framework for evaluating fuzzy quantification queries over both standard and fuzzy ontologies as well as knowledge graph

researcharxiv-cs-ai

Geo-Strat-RL: Learning Geological Event Reasoning from Verifiable Tasks

arXiv:2606.25000v1 Announce Type: new Abstract: To evaluate whether vision-language models can reason about geological histories, it is necessary to construct observations for which the underlying pro

researcharxiv-cs-lg

LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges

arXiv:2606.25057v1 Announce Type: new Abstract: The rapid growth of scientific submissions has pushed traditional peer review toward its scalability limits, motivating the exploration of large languag

safetyarxiv-cs-cl

Long-Term Simulation Exposes Cognitive-Developmental Risks in AI Companions

arXiv:2606.25396v1 Announce Type: new Abstract: AI companions powered by large language models increasingly interact with cognition-developing users, including children and adolescents, creating risks

safetyarxiv-cs-ai

Applications

Measuring Research Difficulty of Academic Papers: A Case Study in Natural Language Processing

arXiv:2606.25307v1 Announce Type: cross Abstract: With the rapid growth of the number of academic papers, systematically evaluating the difficulty of research and its relationship to academic impact o

applicationsarxiv-cs-cl

Tools

More on how we're constraining eval environments so that scores better reflect model intelligence: http://cursor.com/blog/reward-hacking-cod…

Cursor discusses methods for improving AI evaluation environments to prevent reward hacking and ensure benchmark scores more accurately measure model capabilities rather than exploitation of evaluatio

toolscursor--x

Proactive Systems in HCI and AI: Concepts, Challenges, and Opportunities

arXiv:2606.25149v1 Announce Type: cross Abstract: The last few years have seen a significant rise in interest in highly autonomous and proactive systems, fueled by advances in AI. Systems that anticip

agentsarxiv-cs-ai

Local Ai

Accuracy and Satisfaction in Multi-Turn LLM Dialogues for NFR Assessment

arXiv:2606.24834v1 Announce Type: new Abstract: LLM-based dialogue assistants have become mainstream tools for software developers, yet current evaluation benchmarks focus exclusively on functional co

local-aiarxiv-cs-ai

Automated Residual Plot Assessment With the R Package autovi and the Shiny Application autovi.web

arXiv:2606.24236v1 Announce Type: cross Abstract: Visual assessment of residual plots is a common approach for diagnosing linear models, but it relies on manual evaluation, which does not scale well a

researcharxiv-cs-cv

BIM-Edit: Benchmarking Large Language Models for IFC-Based Building Information Modeling

arXiv:2606.20146v3 Announce Type: replace Abstract: Large language models (LLMs) are increasingly applied to computer-aided design (CAD) to generate design artifacts from textual instructions. In engi

EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence

arXiv:2606.24797v1 Announce Type: cross Abstract: Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on video question answering (VideoQA). Nevertheless, ex

EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

model-releasesarxiv-cs-cv

arXiv:2606.24422v1 Announce Type: new Abstract: We introduce EgoSAT, the first comprehensive benchmark for egocentric video reasoning in streaming settings, designed to evaluate the capabilities of mo

Governed Shared Memory for Multi-Agent LLM Systems

arXiv:2606.24535v1 Announce Type: new Abstract: Multi-agent LLM environments require robust mechanisms for shared knowledge management. This paper formalizes the fleet-memory problem and identifies fo

safetyarxiv-cs-ai

Applications

Introducing the FFASR Leaderboard: Benchmarking ASR in the Real World

The FFASR Leaderboard is a benchmarking platform introduced by Hugging Face for evaluating Automatic Speech Recognition (ASR) systems on real-world performance metrics. It provides a standardized eval

applicationshugging-face

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

arXiv:2606.24530v1 Announce Type: new Abstract: We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether

ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge

arXiv:2606.24648v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) have been widely used as judge models for the automatic evaluation of generated speech. However, prior approaches

Promise and challenges of heart chamber segmentation from non-contrast CT scans using contrastive unpaired image translation: a feasibility study

arXiv:2606.23879v1 Announce Type: cross Abstract: Purpose: To evaluate the feasibility and challenges of heart chamber segmentation from non-contrast CT scans using contrastive unpaired image translat

researcharxiv-cs-ai

REALM: A Unified Red-Teaming Benchmark for Physical-World VLMs

model-releasesarxiv-cs-cv

arXiv:2606.23892v1 Announce Type: new Abstract: Vision-language models (VLMs) are increasingly used as perception-reasoning backbones for embodied intelligence in safety-critical physical systems, whe

What Does ODRL Mean? A Cross-Level Ontological Grounding of Permissions, Prohibitions, and Duties in UFO-L

arXiv:2606.24344v1 Announce Type: cross Abstract: ODRL policy evaluators produce verdicts, but say nothing about the normative positions a policy brings into existence, the authority structures those

When CQs Go Wrong: Challenges in CQ Verification with OE-Assist

arXiv:2606.24619v1 Announce Type: new Abstract: Competency Questions (CQs) are the central component of CQ-verification, an established process in which an ontology is evaluated against a set of natur

safetyarxiv-cs-ai

AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

arXiv:2602.14257v2 Announce Type: replace-cross Abstract: While Large Language Model (LLM) agents have made remarkable progress on complex reasoning, evaluating them in real-world environments remains

An Empirical Study of OpenPangu Quantization on Ascend NPUs

arXiv:2606.21257v1 Announce Type: new Abstract: OpenPangu models are attractive targets for private and domestic large-language-model deployment, yet their robustness under aggressive post-training qu

researcharxiv-cs-lg

Biological Sex Determination in Cadavers Using Deep Learning Algorithms from Computed Tomography Images of Pelvis and Skull

arXiv:2606.22515v1 Announce Type: new Abstract: Sexual identification of decomposed cadavers challenges traditional methods dependent on visual anthropological analysis. This study evaluates state-of-

researcharxiv-cs-cv

CapRiCorn-1K: A Comprehensive Benchmark for Video Captioning and Subject Referential Consistency Across Temporal Scales

model-releasesarxiv-cs-cv

arXiv:2606.21949v1 Announce Type: new Abstract: Accurate and comprehensive video captions with consistent subject references are critical for downstream understanding and generation tasks. However, fe

Chem2Gen-Bench: Benchmarking Chemical-to-Genetic Translation in Perturbation Response Space

arXiv:2606.21109v1 Announce Type: new Abstract: Virtual-cell and perturbation models are increasingly used to predict cellular responses for biomedical discovery, but chemical and genetic perturbation

Direct Raw Audio Signal Processing via Reservoir Computing: An Investigation into 'Feature-Free' Architectures

arXiv:2606.21335v1 Announce Type: cross Abstract: This paper evaluates Reservoir Computing (RC) as an autonomous, 'feature-free' framework for audio processing, designed to eliminate traditional, hand

agentsarxiv-cs-lg

DiT-Reward: Generative Representations for Text-to-Image Reward Modeling

arXiv:2606.23626v1 Announce Type: new Abstract: Can representations learned for image generation also support the evaluation of generated images? We study text-to-image reward prediction as a downstre

safetyarxiv-cs-lg

Tutorials

Full-Body Golf Swing Kinematic Reconstruction From a Smartwatch IMU

arXiv:2606.22876v1 Announce Type: new Abstract: Quantitative measurement of the golf swing is critical for evaluating technique and enabling individualized feedback. However, existing methods are impr

tutorialsarxiv-cs-cv

Policy4OOD: A Knowledge-Guided World Model for Policy Intervention Simulation against the Opioid Overdose Crisis

arXiv:2602.12373v2 Announce Type: replace Abstract: The opioid epidemic remains one of the most severe public health crises in the United States, yet evaluating policy interventions before implementat

Position: Correct Answer, Wrong Mechanism -- When AI Scientists Defend General Claims Their Own Data Contradicts

arXiv:2606.23175v1 Announce Type: new Abstract: AI scientist systems are described as tools, coauthors, or founders, but we evaluate them as if only the final answer matters. This position paper argue

agentsarxiv-cs-lg

Applications

Reference-Free Assessment of Physical Consistency in World Model-based Video Generation

arXiv:2606.22363v1 Announce Type: cross Abstract: We introduce reference-free measures for evaluating the physical consistency of generated videos, combining relative and absolute approaches to assess

applicationsarxiv-cs-lg

Skill Coverage: A Test Adequacy Metric for Agent Skills

arXiv:2606.20659v1 Announce Type: cross Abstract: Agent skills encode reusable procedural knowledge that guides large language model agents across tasks and execution contexts. Existing evaluations pr

Stage-dependent integer-binary encoding in factorization-machine black-box optimization

arXiv:2606.23188v1 Announce Type: new Abstract: Black-box optimization (BBO) deals with problems where objective functions lack explicit analytical forms and are expensive to evaluate. Factorization m

researcharxiv-cs-lg

Embed the world: Multimodal AI for searchable aerial imagery at scale