Search: “evaluation”

AgentLTL: A Trace-Verification Framework for Measuring, Enforcing, and Training Procedural Compliance in Tool-Using LLM Agents

arXiv:2607.02599v1 Announce Type: cross Abstract: Tool-using LLM agents are usually evaluated by final-answer correctness or LLM judges. Neither captures how an answer was produced. In safety-critical

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

DGX agent

arXiv:2607.02586v1 Announce Type: new Abstract: Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common fo

model-releasesarxiv-cs-lg

7 Jul 2026

Safety

Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems

arXiv:2607.04433v1 Announce Type: cross Abstract: The rapid integration of large language model-based agents into recommender systems has driven a shift from static, ranking-based pipelines toward aut

safetyarxiv-cs-cl

Beyond Forecasting: The Belief-to-Trade Layer in Prediction-Market Agents

arXiv:2607.03015v1 Announce Type: new Abstract: Forecasting future events has attracted growing attention as a testbed for general-purpose AI. A natural way to ground this evaluation is let the models

Conflict-Based Lazy Search for Fast Multi-Manipulator Planning

arXiv:2607.04124v1 Announce Type: cross Abstract: Employing multiple manipulators can boost efficiency and accomplish tasks that a single manipulator cannot do. However, real-time planning for multipl

agentsarxiv-cs-ai

Safety

CrossHallu: Do Hallucination Signals Generalize Across Languages and Domains in Large Language Model's Internals?

arXiv:2607.04029v1 Announce Type: new Abstract: Recent hallucination detection techniques in large language models (LLMs) focus on directly extracting features from a model's internal representations

safetyarxiv-cs-cl

Determinants and Limits of LLM Security-Tool Orchestration: A Study with HexStrike-AI

arXiv:2607.02873v1 Announce Type: cross Abstract: Large language model agents driving security tool suites over the Model Context Protocol are increasingly common. Yet the factors that bound their cap

Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs

arXiv:2607.03647v1 Announce Type: new Abstract: Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidenc

How Many Initial Points Does Bayesian Optimization Need?

arXiv:2607.04356v1 Announce Type: new Abstract: Bayesian Optimization (BO) generally begins with an initialization phase: a batch of n_0 uninformed evaluations. The choice of n_0 remains largely heuri

In-span learning: adapting reduced-order models using their own predictions

arXiv:2607.02937v1 Announce Type: new Abstract: Reduced-order models compress high-dimensional dynamics into low-dimensional representations that can be evaluated rapidly, but they lose accuracy when

Tutorials

Interactive Multi-Objective Probabilistic Preference Learning with Soft and Hard Bounds

arXiv:2506.21887v2 Announce Type: replace Abstract: High-stakes decision-making involves navigating multiple competing objectives with expensive evaluations. For instance, in brachytherapy, clinicians

tutorialsarxiv-cs-ai

IRC-Bench: Recognizing Entities from Contextual Cues in First-Person Reminiscences

arXiv:2605.06142v2 Announce Type: replace-cross Abstract: When people recount personal memories, they often refer to people, places, and events indirectly, relying on con-textual cues rather than expl

Industry

My thoughts on Fable

This article presents Ben's personal analysis and perspective on Fable, likely covering the platform's features, capabilities, and potential impact in the AI or data space. The piece probably evaluate

industryben-s-bites

Natural Language Camera Movement Understanding

arXiv:2607.03043v1 Announce Type: new Abstract: Understanding camera movement in natural language is critical for training and evaluating video generation models, among other applications. However, we

NEST: Nascent Encoded Steganographic Thoughts

arXiv:2602.14095v2 Announce Type: replace Abstract: Monitoring chain-of-thought (CoT) reasoning is a foundational safety technique for large language model agents; however, this oversight is compromis

Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

arXiv:2607.04537v1 Announce Type: cross Abstract: Code language models are now trusted collaborators in production workflows for debugging, refactoring, and iterative repair, and every benchmark that

Operator-on-F complements value-equivalence: a planning-time diagnostic for latent world models

arXiv:2607.04464v1 Announce Type: cross Abstract: World-model evaluation for model-based reinforcement learning typically asks whether the learned model predicts reward and value well, which can leave

PDFBench: A Benchmark for De novo Protein Design from Function

arXiv:2505.20346v3 Announce Type: replace-cross Abstract: Function-guided protein design is a crucial task with significant applications in drug discovery and enzyme engineering. However, the field la

Probing Geospatial SSL Representations with Environmental Signals

arXiv:2607.05207v1 Announce Type: new Abstract: Self-supervised learning (SSL) is designed to learn generic, transferable representations rather than representations optimized for a single task. Most

Programming over Thinking: Efficient and Robust Multi-Constraint Planning

arXiv:2601.09097v3 Announce Type: replace Abstract: Multi-constraint planning involves identifying, evaluating, and refining candidate plans while satisfying multiple, potentially conflicting constrai

Quantum-Inspired Harmonic Decision Models: A Computational Framework for Music Generation

arXiv:2607.05007v1 Announce Type: new Abstract: This paper introduces a quantum-inspired computational framework for harmonic decision-making in music. The proposed approach formulates harmonization a

Quickest Detection of Hallucination Onset: Delay Bounds and Learned CUSUM Statistics

arXiv:2606.12476v3 Announce Type: replace-cross Abstract: Token-level hallucination detectors are evaluated as classifiers, by AUC over all tokens, yet a streaming monitor is judged by its reaction ti

ResearchStudio-Idea: An Evidence-Grounded Research-Ideation Skill Suite from ML Conference Outcomes

arXiv:2607.04439v1 Announce Type: new Abstract: Large language models have made research ideation increasingly accessible, yet effective idea development requires more than generating candidate direct

SafeGuard: A Multi-Agent Perception-Reasoning Framework for Social-Risk AI-Generated Video Detection

arXiv:2607.03069v1 Announce Type: new Abstract: As video generation paradigms evolve from localized manipulation to full-scene synthesis, AI-generated video detection becomes increasingly challenging,

Separating Representation from Reconstruction Enables Scalable Text Encoders

arXiv:2607.04011v1 Announce Type: cross Abstract: While decoders have rapidly scaled, encoders have remained largely unchanged since BERT. We revisit this disparity by frozen backbone evaluation via p

Applications

SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks

arXiv:2603.10002v2 Announce Type: replace-cross Abstract: We consider the task of end-to-end spreadsheet generation, where language models produce spreadsheet artifacts to satisfy users' explicit and

applicationsarxiv-cs-ai

TabQueryBench: A Query-Centric Benchmark for Synthetic Tabular Data

arXiv:2607.03926v1 Announce Type: cross Abstract: Synthetic tabular data support use cases like data sharing, model development under access restrictions, and rapid prototyping of analytical workflows

Tightening the Score Matching Gap for Diffusion Models

arXiv:2607.04442v1 Announce Type: cross Abstract: Diffusion models (DMs) are a state-of-the-art generative method to approximately sample from an unknown distribution. Their training and evaluation pr

Triple-Phase Multimodal Knowledge Aggregation Framework for Microbial Keratitis Subtype Diagnosis on Slit-Lamp Photography

arXiv:2607.03740v1 Announce Type: cross Abstract: Microbial keratitis requires rapid pathogen identification to guide treatment, but culture- and PCR-based diagnostics are slow and resource-intensive.

TSP with Predictions: Heatmap to Tour with Provable Guarantees

model-releasesarxiv-cs-lg

arXiv:2607.03791v1 Announce Type: cross Abstract: The Traveling Salesperson Problem (TSP) has long served as a benchmark for evaluating the strength of optimization techniques in the classical theory

URSA: Chemistry-Aware Benchmark for Utilitarian Retrosynthesis Assessment

arXiv:2607.04688v1 Announce Type: cross Abstract: Synthesis planning aiming to find pathways of reactions for a target molecule is one of the most important and challenging tasks in drug discovery. Re

When Rubrics Fail: Error Enumeration as Reward in Reference-Free RL Post-Training for Virtual Try-On

arXiv:2603.05659v3 Announce Type: replace-cross Abstract: Reinforcement learning with verifiable rewards (RLVR) and Rubrics as Rewards (RaR) have driven strong gains in domains with clear correctness

Reporting benchmark results as a scalar number, e.g. '75% on XYZ' is completely meaningless at this point. You should always report efficien…

model-releasesfrancois-chollet--x

Reporting benchmark results as a single scalar percentage is insufficient for meaningful evaluation of model performance. Comprehensive benchmark reporting should include efficiency metrics alongside

6 Jul 2026

AgenticDataBench: A Comprehensive Benchmark for Data Agents

arXiv:2607.01647v1 Announce Type: cross Abstract: Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern so

BRIDGE: Predicting Human Task Completion Time From Model Performance

arXiv:2602.07267v2 Announce Type: replace Abstract: Evaluating the real-world capabilities of AI systems requires grounding benchmark performance in human-interpretable measures of task difficulty. Ex

CommonRoad-Game: A Human-in-the-Loop Simulation Framework for Autonomous Driving

arXiv:2607.01382v1 Announce Type: new Abstract: Motion planning algorithms should be evaluated in human-in-the-loop environments to ensure they produce safe and efficient behaviors during interactions

agentsarxiv-cs-ro

Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition

arXiv:2408.01139v4 Announce Type: replace Abstract: Perturbation robustness evaluates the vulnerabilities of models, arising from a variety of perturbations, such as data corruptions and adversarial a

Janus: a Playground for User-Involved Agentic Permission Management

arXiv:2607.01510v1 Announce Type: new Abstract: AI agents that autonomously execute tool calls on a user's behalf raise pressing questions about permission management: what role could users play, and

agentsarxiv-cs-ai

Leveraging Metamemory Agent for Enhanced Data-Free Code Generation in Large Language Models

arXiv:2501.07892v2 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown strong performance in automated code generation, with few-shot prompting widely used for its simplicit

agentsarxiv-cs-ai

Liquid Latent State Dynamics for Interpretable Turbofan Degradation Modeling

model-releasesarxiv-cs-lg

arXiv:2607.01986v1 Announce Type: new Abstract: Multivariate time-series models for prognostics are often evaluated by point prediction accuracy, yet their internal states rarely expose a coherent deg

Safety

Morphology-Aware Sample Assignment: Overcoming IoU Insensitivity for Surface Defect Detection

arXiv:2606.13723v2 Announce Type: replace-cross Abstract: Intersection-over-Union (IoU), as a pivotal metric for evaluating the spatial alignment between candidate proposals and ground-truth annotatio

safetyarxiv-cs-ai

OntoLearner: A Modular Python Library for Ontology Learning with Large Language Models

arXiv:2607.01977v1 Announce Type: new Abstract: Ontology learning (OL) aims to automatically construct structured knowledge models from text, yet progress remains fragmented across methods, domains, a

PPTArena: A Benchmark for PowerPoint Editing

arXiv:2512.03042v3 Announce Type: replace-cross Abstract: We introduce PPTArena, a benchmark for PowerPoint editing that evaluates how agents modify real slides from natural-language instructions. Unl

Local Ai

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

arXiv:2607.01480v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifi

local-aiarxiv-cs-ai

Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

arXiv:2607.01793v1 Announce Type: new Abstract: LLM agents increasingly perform autonomous actions through external tools, leading to complex and evolving safety risks. However, existing safety testin

Scalable and Distributed Silhouette Approximation

arXiv:2607.01993v1 Announce Type: cross Abstract: The silhouette is one of the most widely used measures to assess the quality of a k-clustering of a dataset of n elements. Its evaluation requires no

Single-Channel EEG-Based Cognitive Load Assessment in Online Learning: A Hybrid Deep Learning Approach

arXiv:2607.01795v1 Announce Type: cross Abstract: Monitoring cognitive load during online learning could help instructors identify content that learners find difficult, but remote settings remove the

Something about this year’s @aiDotEngineer World’s Fair just hit different. Last year was the year of “let the agents rip.” This year was th…

model-releasesyohei-nakajima--x

Something about this year’s @aiDotEngineer World’s Fair just hit different. Last year was the year of “let the agents rip.” This year was the year of realizing that autonomy without structure creates