Search: “evaluation”

How Reusable Are Benchmarks with Richer Feedback?

arXiv:2609.32109v1 Announce Type: new Abstract: We study whether benchmarks reliably guide model selection as developers adapt to evaluation feedback across multiple criteria. We find that the worst-c

Human Motion Prediction for Human-Robot Collaboration

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2609.32346v1 Announce Type: new Abstract: This paper compares three human motion prediction methods - GMM-based clustering, TransFusion, and Graph-Mixer - evaluating their effectiveness in human

safetyarxiv-cs-ro

29 Sept 2026

Maintaining Benchmarks Against Increasingly Capable Agents: Detection and Remediation of Unearned Passes

arXiv:2609.34262v1 Announce Type: cross Abstract: Agentic benchmarks guide model selection and training. Yet an agent can pass a task without demonstrating the intended capability. Such outcomes const

MixBench-TS: A Multivariate Time Series Forecasting Benchmark Where Channel Mixing Pays Off

arXiv:2609.32656v1 Announce Type: cross Abstract: Multivariate Time Series Forecasting (MTSF) models that mix information across channels assume that the past of one channel carries information about

On Memory: A comparison of memory mechanisms in world models

arXiv:2512.06983v2 Announce Type: replace-cross Abstract: World models enable agents to plan within imagined environments by predicting future states conditioned on past observations and actions. Howe

Local Ai

One Sequence, Many Decodings: CAGenMol-2 Recasts Drug Design as Masked Molecular Inference

arXiv:2609.34301v1 Announce Type: new Abstract: Drug design couples property evaluation, conditional generation, structure-based design, and local optimization, yet machine learning systems typically

local-aiarxiv-cs-lg

OpenAI is adopting a structured 'safety case' documentation framework modeled after industries like aviation and nuclear power to govern frontier RL training (OpenAI)

OpenAI: OpenAI is adopting a structured “safety case” documentation framework modeled after industries like aviation and nuclear power to govern frontier RL training — Loading... We believe we are ent

safetytechmeme

OpenTumorBoard: A Real-World Benchmark of Multidisciplinary Tumor Board Discussion Trajectories

arXiv:2609.32810v1 Announce Type: cross Abstract: Multidisciplinary tumor boards integrate multimodal clinical observations and longitudinal patient histories through specialist discussions, yet bench

Prediction Limits and Koopman Closure of Geometry-Induced Soft State Abstractions

arXiv:2609.32652v1 Announce Type: cross Abstract: We study when geometry-induced soft state abstractions admit accurate finite-dimensional linear dynamics. Each state is represented by simplex-valued

Presence Is Not Faithfulness: Figurative Vehicle Intrusion in Text-to-Image Generation

arXiv:2609.32188v1 Announce Type: new Abstract: Text-to-image (TTI) models increasingly generate high-quality images from natural-language prompts, yet figurative language exposes a failure: a vehicle

Protected Cores Are Not Enough: Certifying AI-Proposed Revisions of Temporal Specifications

arXiv:2609.33461v1 Announce Type: cross Abstract: Runtime monitoring traditionally evaluates a specification that is fixed before execution or externally modified when requirements change. In learning

Quantifying Behavioral Tails in Black-Box Language Models

arXiv:2609.33638v1 Announce Type: new Abstract: We introduce RareTrap, a framework for estimating the probability of severe behaviors in black box large language models (LLMs). A key challenge for pro

Quasi Linear Kernel Attention with Infinite Capacity

arXiv:2609.35349v1 Announce Type: new Abstract: The evaluation cost of transformers with softmax attention scales quadratically with sequence length. Kernel attention addresses this by replacing softm

Quoting Anthropic Frontier Red Team

model-releasessimon-willison

We evaluate several models on 100 tasks from the [internal Binary Exploitation benchmark] (selected at random), and find that GLM-5.3 develops full control flow hijacks in 4% of the trials; Claude Myt

RAGWarrant: Evidence-Preserving Governance for RAG Policy Promotion Under Quality, Cost, Latency, and Risk Constraints

arXiv:2609.34179v1 Announce Type: cross Abstract: Retrieval-augmented generation systems are extensively instrumented with metrics, benchmarks, traces, and automated judges, but these tools do not dec

Reliability-Aware Sparse Route Memory for Round-Trip Vision-Language Navigation

arXiv:2609.34163v1 Announce Type: new Abstract: Vision-language navigation (VLN) is typically evaluated as a one-way task, although deployed robots may need to return after reaching a goal. We study c

researcharxiv-cs-ro

RemTraceNet: Few-Shot Forensic Detection of Invisible Watermark Attacks

arXiv:2609.31694v1 Announce Type: new Abstract: Removing an invisible watermark and concealing the forensic evidence are distinct objectives: successfully disrupting the embedded watermark does not im

researcharxiv-cs-cv

Local Ai

RoboCompiler: Graph-Native Compilation of Closed-Chain Robots for Consistent Modeling, Control, and Simulation

arXiv:2609.35717v1 Announce Type: new Abstract: Robots with kinematic loops, coupled actuators, and changing contacts require consistent models of configuration, motion, force, and dynamics. Yet these

local-aiarxiv-cs-ro

RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models

arXiv:2609.34384v1 Announce Type: cross Abstract: Vision-Language-Action (VLA) models have shown strong capabilities in robotic manipulation, yet existing benchmarks typically assume that task-relevan

ScreenHaystack: Finding Blind Zones in GUI Grounding

arXiv:2609.32036v1 Announce Type: new Abstract: We introduce ScreenHaystack, a dynamic needle-in-a-haystack benchmark for evaluating spatial reliability in GUI grounding. Instead of testing each targe

Scriboora: Rethinking Human Pose Forecasting

arXiv:2511.15565v3 Announce Type: replace Abstract: Human pose forecasting predicts future human poses from past observations for applications including action recognition, sports analysis, and human-

Tutorials

SeLMRoute: Probabilistic Semantic Evidence for Large Language Model Routing

arXiv:2609.34736v1 Announce Type: cross Abstract: Large language model (LLM) routing aims to select the most suitable model for each incoming query. Most existing routers learn this decision directly

tutorialsarxiv-cs-lg

SPINET: Sheaf Protein Inverse Folding Network

arXiv:2609.34153v1 Announce Type: new Abstract: Proteins change shape as they function, yet most inverse folding models predict amino acid sequences from a single, fixed backbone. A central challenge

StoryEngine: A State-Grounded Agentic Framework for Video Storytelling

arXiv:2609.33627v1 Announce Type: new Abstract: Despite recent progress in agentic multi-shot video generation, producing coherent and consistent long-form stories remains challenging. Existing agenti

STR: Supervised Transcoder Replacement for Reducing Steering Side Effects

arXiv:2609.32519v1 Announce Type: cross Abstract: Model steering can strengthen a target behavior while degrading other useful behaviors. We introduce Supervised Transcoder Replacement (STR) to reduce

Toward Open-World Video Segmentation over Long Horizons

arXiv:2606.15632v3 Announce Type: replace Abstract: Long videos challenge open-world segmentation systems to continually discover objects and preserve their identities through disappearance and reappe

researcharxiv-cs-cv

TT-VidT: Decoupling the Temporal Axis for Efficient Motion-Centric Video Pretraining

arXiv:2609.33419v1 Announce Type: new Abstract: Comparisons in video self-supervised learning often evaluate complete training recipes rather than isolating the method itself: architecture, objective,

researcharxiv-cs-cv

We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream…

We let open models access the outside world during evals on tasks from public benchmarks. Here's what happened: Most went and found upstream commits containing fixes for the tasks they were supposed t

agentsai21-labs--x

What Does a ProcGen Generalization Gap Measure? Action Rules, Convergence, and the Missing Random Floor

arXiv:2609.32532v1 Announce Type: new Abstract: A generalization gap in reinforcement learning (return on training levels minus return on held-out levels) is usually reported without a reference point

When Better Gets Worse: Improvement Fidelity for Self-Improving Agents in Adaptive Worlds

arXiv:2609.32677v1 Announce Type: cross Abstract: Self-improving agents increasingly rely on proxy verifiers to choose policy updates, yet deployment can change the world in which those updates are ev

When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

arXiv:2603.13688v2 Announce Type: replace-cross Abstract: AI systems increasingly assist decision making by producing cheap factor-level assessments of complex inputs, but these assessments can be bia

When Sparse Reward Meets Dense Distillation: Training Dynamics of On-Policy Distillation

arXiv:2609.34849v1 Announce Type: new Abstract: Reinforcement learning with verifiable rewards provides a sparse post-training signal: a single binary outcome evaluates the entire rollout, and every t

When the Score Becomes the Target: Rethinking Metric Validity in Autonomous Driving

arXiv:2609.34440v1 Announce Type: new Abstract: Driving benchmark scores are increasingly used not only for evaluation but also as optimization targets. This raises a fundamental question: do score ga

A Benchmarking Framework for Context-aware XR Interfaces

arXiv:2609.30466v1 Announce Type: cross Abstract: Everyday Extended Reality (XR) systems aim to provide context-aware access to the right functionalities at the right time and place, with minimal manu

researcharxiv-cs-ai

A Survey on Fake Review Detection: From Pre-trained Language Models to Large Language Models

arXiv:2609.30292v1 Announce Type: cross Abstract: Online reviews shape consumer decisions, platform governance, and corporate reputation.Fake reviews compromise this information channel by injecting d

Attribution Bias in Large Language Models

arXiv:2604.05224v2 Announce Type: replace Abstract: As Large Language Models (LLMs) are increasingly used to support search and information retrieval, it is critical that they accurately attribute con

BabelCoder: Agentic Code Translation with Specification Alignment

arXiv:2512.06902v2 Announce Type: replace-cross Abstract: As software systems evolve, developers increasingly work across multiple programming languages and often face the need to migrate code from on

Band-Selection Stability and Semantic Segmentation Performance: A Study on Hyperspectral City

arXiv:2609.31074v1 Announce Type: new Abstract: Resource constraints make high-dimensional hyperspectral imaging challenging in autonomous perception, motivating the use of band selection methods. How

agentsarxiv-cs-cv

Detecting Time Series Anomalies Like an Expert: A Multi-Agent LLM Framework with Specialized Analyzers

arXiv:2605.05725v2 Announce Type: replace Abstract: Time-series anomaly detection often returns scores or intervals, while analysts need to understand the abnormal behavior and the evidence supporting

agentsarxiv-cs-ai

From Reward Signal to Visual Utility: A Controlled Audit of Medical VLM Post-Training

arXiv:2609.31450v1 Announce Type: cross Abstract: Medical vision-language model (VLM) post-training is commonly evaluated through answer accuracy. We examine how changes in accuracy and training objec

Geometric Inconsistency Localization in Multi-View Image Sets

arXiv:2609.31247v1 Announce Type: cross Abstract: Novel view synthesis (NVS) models can produce realistic new views of the same scene from different viewpoints. However, these generated views are not

Guarded Gradient-Based Activation Steering of Shutdown Responses in Qwen3.5-0.8B: A Minimum-Step Policy

arXiv:2609.30326v1 Announce Type: new Abstract: Activation steering changes a model's internal activations during inference without updating its weights, but a useful intervention must determine both

HCOE: Hyperbolic Clinical Ontology Embeddings from Biomedical Language Models

arXiv:2609.30763v1 Announce Type: new Abstract: Biomedical language models (LMs) encode textual semantics but do not explicitly preserve medical code hierarchies. We present Hyperbolic Clinical Ontolo

researcharxiv-cs-ai

Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modeling

model-releasesarxiv-cs-cl

arXiv:2609.30288v1 Announce Type: new Abstract: In Transformer-based masked language models, attention is the primary mechanism for context mixing, but there are other ways to mix data across tokens.

MedHal: a Synthetic Dataset for Medical Hallucination Detection

arXiv:2504.08596v3 Announce Type: replace-cross Abstract: Hallucination, the generation of non factual content by AI systems, poses serious risks in medical contexts, where errors can directly affect

researcharxiv-cs-ai

Missingness-Aware Conformal Prediction Under Cross-Hospital Distribution Shift

arXiv:2609.30781v1 Announce Type: new Abstract: Clinical measurements are recorded for some patients but not others, at rates that differ across hospitals, and marginal conformal coverage does not ens

On the Limits of Univariate Deep Learning for Significant Wave Height Forecasting

arXiv:2609.30688v1 Announce Type: cross Abstract: This study conducts a systematic hyperparameter search across five deep learning architectures, DLinear, LSTM, PatchTST, ResAttLstm, and Mamba2, and n

OpenAI halts frontier-model training amid string of agent misalignment incidents

OpenAI has paused all internal training and inference for its most capable frontier models after a misalignment incident in which an agent exploited a DNS filtering gap during routine research, attemp

agentsars-technica