Search: “evaluation”

Not All Errors Matter: Decision-Relevant Prediction Error Predicts Planning Quality

arXiv:2609.32322v1 Announce Type: new Abstract: World models are typically trained and evaluated by prediction error, assuming that more accurate predictions lead to better decisions. We show that thi

safetyarxiv-cs-lg

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Physics-Informed Neural Networks for Depth-Averaged Avalanche Dynamics

DGX agent

arXiv:2609.34916v1 Announce Type: cross Abstract: Accurate prediction of avalanche motion is essential for hazard assessment in mountainous terrain. This study develops and evaluates a physics-informe

researcharxiv-cs-lg

29 Sept 2026

PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?

arXiv:2609.34314v1 Announce Type: new Abstract: Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether

Printability-Constrained Adversarial Decals for Near-Nadir Aerial Perception: Measured Ink Gamuts, Nested Realism Constraints, and a Physical-World Bound

arXiv:2609.33513v1 Announce Type: new Abstract: Adversarial patches for aerial perception are typically evaluated as digital composites, with printing left as an implementation detail. This study impo

researcharxiv-cs-cv

REALIS: A Curated Dataset for Studying the Challenges of AI Image Detection

arXiv:2609.32734v1 Announce Type: new Abstract: AI-generated image detectors are often evaluated on benchmarks where real and synthetic images differ in content, quality, or generation artifacts, allo

The Statistical Benefits of Multiple Responses for Learning from Demonstrations

arXiv:2609.33291v1 Announce Type: cross Abstract: Many generative systems return multiple candidate responses and are evaluated according to the best one. Recent work shows that, when demonstrations a

safetyarxiv-cs-lg

A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory

arXiv:2609.30813v1 Announce Type: new Abstract: Evaluating claim admission in shared agent memory is challenging because repeated claims may be mistaken for independent evidence. An agent may copy or

Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipeline

arXiv:2609.30290v1 Announce Type: new Abstract: Production text-to-SQL pipelines often end with an LLM-as-judge whose agreement with human annotators has never actually been measured. When we checked

Seasonal and Quantum-inspired Models for Neutron Monitor Time Series Forecasting

arXiv:2609.30281v1 Announce Type: new Abstract: We present a focused and reproducible study of multi-horizon forecasting on the Lomnicky Stit neutron monitor (LMKS) time series. Our evaluation suite c

researcharxiv-cs-lg

Tools

What Jev’s probabilities reveal that repeated LLM judgments miss

I tested Jev and five LLM judges across ten Arize Phoenix evaluators to measure how often their answers changed alongside accuracy, cost, and latency. The post What Jev’s probabilities reveal that rep

toolsarize-ai

We released VeriLoop E2 (27B, Apache-2.0). The design question behind it: should an LLM be allowed to commit its own state?

model-releasesr-localllama

Disclosure: I’m one of the authors of VeriLoop E2, a 27B model post-trained from Qwen3.8-27B. The weights are Apache-2.0; the harness we evaluate it in is not open source (details at the bottom). This

27 Sept 2026

Coding Agents for Generalized Task and Motion Planning Problems

arXiv:2609.30233v1 Announce Type: cross Abstract: Task and motion planning (TAMP) problems remain difficult even with full observability and object-centric states because discrete decisions are tightl

COILD: An Indic-Centric Parallel Corpus and Benchmark for Machine Translation Across Indian Languages

arXiv:2609.28826v1 Announce Type: new Abstract: Machine translation (MT) for Indian languages remains constrained by the limited availability of high-quality, Indic-centric parallel corpora and evalua

LiveMathematicianBench: A Live Benchmark for Research-Level Mathematical Reasoning with Proof Sketches

arXiv:2604.01754v2 Announce Type: replace-cross Abstract: Mathematical reasoning is a hallmark of human intelligence, and whether large language models (LLMs) can meaningfully perform it remains a cen

MLPerf Automotive

model-releasesarxiv-cs-lg

arXiv:2510.27065v2 Announce Type: replace Abstract: We present MLPerf Automotive, the first standardized public performance benchmark for evaluating Machine Learning systems that are deployed for AI a

ModularSQL: A Runtime Guardrail for the Multiplicity Blind Spot in Text-to-SQL

arXiv:2609.29573v1 Announce Type: new Abstract: Text-to-SQL systems are increasingly deployed on production databases, where queries that pass benchmark evaluation can still produce results that disto

Agents

Reward Hacking Challenges Oversight of Autonomous Research Agents

arXiv:2609.28614v1 Announce Type: new Abstract: Autonomous research agents can design experiments, evaluate results, and write reports, giving them control over both a scientific result and the eviden

agentsarxiv-cs-cl

TopU-LBVS: A Realistic Multi Target Benchmark for Ligand Based Virtual Screening

arXiv:2609.29740v1 Announce Type: cross Abstract: Ligand-based virtual screening (LBVS) is a practical first-pass tool in early-stage drug discovery, but existing benchmarks can overestimate performan

An Open Pipeline and Dashboard for Systemic-Risk Evidence under the EU AI Act's Code of Practice

arXiv:2609.28335v1 Announce Type: new Abstract: Claims about AI safety reach audiences well beyond the AI community, yet many rely on opaque evidence or static assessments, when supporting evidence is

Applications

Cross-Lingual Legal QA for Vietnamese Labour Law: Retrieval, Translation, and Verifier-Guided Correction

arXiv:2609.27376v1 Announce Type: new Abstract: Cross-lingual legal question answering must retrieve statutes across languages while preventing unsupported legal claims. We introduce a bilingual evalu

applicationsarxiv-cs-cl

Applications

Not What You Meant: Can LLMs Follow a Specified Negation Semantics?

arXiv:2609.27517v1 Announce Type: new Abstract: Negation does not carry a uniform interpretation across domains. In legal, regulatory, and medical reasoning, the intended interpretation depends on the

applicationsarxiv-cs-ai

Recognized but Not Produced: A Generation Benchmark for Culturally Specific Kinship Terms

arXiv:2609.26942v1 Announce Type: cross Abstract: Current literature evaluates large language models (LLMs) on multilingual kinship understanding using multiple choice benchmarks, treating it as a rec

AgentBetta: Verification-Driven Adaptive Configuration of an AI Nano-Agent through Selective Expansion and Verified Contraction

arXiv:2609.23512v1 Announce Type: new Abstract: Large language model agents are typically deployed with predefined configurations, although the required model capability, context, tools, permissions,

Beyond Reconstruction Error: Analytical and Data-Driven Action Tokenization for Autoregressive Vision-Language-Action Models

arXiv:2609.25820v1 Announce Type: cross Abstract: Discrete action tokenization is central to autoregressive vision-language-action (VLA) models, yet action representations are often evaluated primaril

safetyarxiv-cs-lg

BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?

arXiv:2503.15242v3 Announce Type: replace Abstract: We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generati

FineWeb-CLaR: Culture, Language, and Region Annotations for Benchmark-Aligned Corpus Auditing

arXiv:2609.25298v1 Announce Type: new Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely

Tutorials

Identifying Intelligent Processes via Online Sequential Testing

arXiv:2609.26193v1 Announce Type: cross Abstract: Active sequential hypothesis testing studies how to identify an unknown hypothesis with a given set of sensing actions. We study this in the setting o

tutorialsarxiv-cs-lg

Initialization and Stopping Tolerance in CPU Dermoscopic Segmentation

arXiv:2609.25685v1 Announce Type: new Abstract: Contour initialization and numerical stopping can jointly affect the evaluation of active-contour segmentation. We examine their interaction using the o

researcharxiv-cs-cv

SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking

arXiv:2508.15526v2 Announce Type: replace Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabilit

An Iterative LangGraph Agent for Text-to-SQL: Natural Language Access to the Chicago Crime Database

arXiv:2609.22917v1 Announce Type: new Abstract: Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL a

Are Coreset Selection Methods Worth Their Cost?

arXiv:2609.22894v1 Announce Type: cross Abstract: Coreset selection picks a representative subset of the labeled training set to make training cheaper. However, it is usually evaluated by downstream a

As part of our efforts to pace the frontier, we’re committed to supporting independent assessments with deep levels of access across trainin…

As part of our efforts to pace the frontier, we’re committed to supporting independent assessments with deep levels of access across training, evaluation, and deployment. That access should enable thi

model-releasesopenai--x

Agents

Automatic multimodal UX improvement recommendations from LLM agent user simulations

arXiv:2609.22971v1 Announce Type: new Abstract: Evaluating user experience (UX) on live websites through user testing is expensive, subjective, and difficult to scale. LLM agents offer a promising rou

agentsarxiv-cs-cl

DolphinBench: Mapping the Pareto Frontier of Agent Memory

arXiv:2609.24971v1 Announce Type: new Abstract: Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are bui

From Articles to Publishers: Aggregating Language Model Predictions for News Source Reliability Inference

arXiv:2609.24219v1 Announce Type: new Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standa

researcharxiv-cs-cl

GameReplica: A Benchmark for Black-Box Visual Game Replication by Vision-Language Agents

arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research

Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning

arXiv:2609.24033v1 Announce Type: new Abstract: Reliable action evaluation in contact-rich manipulation requires looking beyond the current observation to future visual and contact consequences. Exist

safetyarxiv-cs-ro

LIBERO-VPro: Benchmarking Closed-Loop Visual Robustness of Robotic Foundation Models

arXiv:2609.24350v1 Announce Type: cross Abstract: Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, an

LLMs Anchor on Chief Complaint and Fail to Integrate Evidence in Sequential Clinical Triage

arXiv:2609.22904v1 Announce Type: new Abstract: Triage in the emergency department (ED) is a sequential decision process that unfolds turn by turn. Existing evaluations of large language models (LLMs)

researcharxiv-cs-cl

RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling

arXiv:2609.22947v1 Announce Type: new Abstract: Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existi

XYEval: Agents say yes to bad advice

arXiv:2609.23939v1 Announce Type: new Abstract: Effective communication between users and AI agents is essential for human-AI collaboration. The XY problem is a well-known communication pitfall where

ZoAQ: Adaptive Zeroth-Order Querying via Query-Reuse Coupling

arXiv:2609.22115v1 Announce Type: new Abstract: Zeroth-order optimization (ZOO) estimates updates from function evaluations, making perturbation queries a primary cost. Fixed budgets spend the same nu

researcharxiv-cs-lg

Benchmarking the Explanatory Quality of Open-Weight Vision-Language Models in Face Recognition

arXiv:2609.21879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations a

Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening

arXiv:2609.21763v1 Announce Type: new Abstract: A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about

BrainWideBench: Benchmarking large-scale pretraining and across-animal transfer in multi-region neural recordings

model-releasesarxiv-cs-lg

arXiv:2609.22064v1 Announce Type: new Abstract: Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question o

CIPL: A Channel-Aware Framework for Recoverable Privacy Leakage in LLM Agents

arXiv:2609.21686v1 Announce Type: cross Abstract: Privacy leakage in LLM agents is commonly evaluated within individual components such as memory, retrieval, or tool-use pipelines, which makes it diff

safetyarxiv-cs-ai

From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers

arXiv:2609.21349v1 Announce Type: cross Abstract: Large language models have shown strong potential as role-playing agents for real individuals, yet faithful impersonating remains challenging. Existin

researcharxiv-cs-ai

with @Thom_Wolf in sunny italy ☀️chatting about the @huggingface @OpenAI attack, the “tell” that it was an agentic swarm, how misalignment a…