Search: “evaluation”

TeleResilienceBench: Quantifying Resilience for LLM Reasoning in Telecommunications

arXiv:2605.09929v1 Announce Type: new Abstract: Deploying large language models in telecommunications requires more than task accuracy. In realistic workflows, a model may inherit partially completed

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Agents

Temporal Sampling Frequency Matters: A Capacity-Aware Study of End-to-End Driving Trajectory Prediction

DGX agent

arXiv:2605.10388v1 Announce Type: new Abstract: End to end (E2E) autonomous driving trajectory prediction is often trained with camera frames sampled at the highest available temporal frequency, assum

agentsarxiv-cs-cv

12 May 2026

The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring

arXiv:2605.09225v1 Announce Type: cross Abstract: Jailbreak attacks -- adversarial prompts that bypass LLM alignment through purely linguistic manipulation -- pose a growing operational security threa

The Last Word Often Wins: A Format Confound in Chain-of-Thought Corruption Studies

arXiv:2605.10799v1 Announce Type: cross Abstract: Corruption studies, the primary tool for evaluating chain-of-thought (CoT) faithfulness, identify which chain positions are 'computationally important

TOC-Bench: A Temporal Object Consistency Benchmark for Video Large Language Models

arXiv:2605.09904v1 Announce Type: new Abstract: Video large language models (Video-LLMs) have achieved remarkable progress in general video understanding, yet their ability to maintain temporal object

Trustworthy AI: Ensuring Reliability and Accountability from Models to Agents

arXiv:2605.08964v1 Announce Type: new Abstract: In this thesis, we develop algorithms with theoretical guarantees for ensuring reliability and accountability of Machine Learning (ML) systems. As ML sy

safetyarxiv-cs-lg

When Agents Overtrust Environmental Evidence: An Extensible Agentic Framework for Benchmarking Evidence-Grounding Defects in LLM Agents

arXiv:2605.08828v1 Announce Type: new Abstract: Large language model agents increasingly operate through environment-facing scaffolds that expose files, web pages, APIs, and logs. These observations i

When Independent Sampling Outperforms Agentic Reasoning

arXiv:2605.08478v1 Announce Type: new Abstract: We study how to allocate inference-time compute for competitive programming under fixed budgets. Evaluating 216 Codeforces problems across Divisions 1-3

agentsarxiv-cs-lg

Willful Disobedience: Automatically Detecting Failures in Agentic Traces

arXiv:2603.23806v2 Announce Type: replace-cross Abstract: AI agents are increasingly embedded in real software systems, where they execute multi-step workflows through multi-turn dialogue, tool invoca

agentsarxiv-cs-ai

A Resilience Framework for Bi-Criteria Combinatorial Optimization with Bandit Feedback

arXiv:2503.12285v2 Announce Type: replace-cross Abstract: We study bi-criteria combinatorial optimization under noisy function evaluations. While resilience and black-box offline-to-online reductions

researcharxiv-cs-ai

Adaptive auditing of AI systems with anytime-valid guarantees

arXiv:2605.07002v1 Announce Type: new Abstract: A major bottleneck in characterizing the failure modes of generative AI systems is the cost and time of annotation and evaluation. Consequently, adaptiv

researcharxiv-cs-ai

Advancing Reliable Synthetic Video Detection: Insights from the SAFE Challenge

arXiv:2605.06912v1 Announce Type: new Abstract: The proliferation of generative video technologies has intensified the need for reliable methods to detect and characterize synthetic media. To address

researcharxiv-cs-cv

Agentick: A Unified Benchmark for General Sequential Decision-Making Agents

arXiv:2605.06869v1 Announce Type: new Abstract: AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no u

Amortized Multi-Objective Optimization Across Tasks with Generative Solution Modeling

arXiv:2511.09598v5 Announce Type: replace Abstract: Many real-world applications require solving families of expensive multi-objective optimization problems~(EMOPs) under varying operational condition

Beyond Retrieval: A Multitask Benchmark and Model for Code Search

arXiv:2605.04615v2 Announce Type: replace-cross Abstract: Code search has usually been evaluated as first-stage retrieval, even though production systems rely on broader pipelines with reranking and d

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks

arXiv:2605.07024v1 Announce Type: new Abstract: Large Language Models for code generation frequently produce hallucinations in Fill-in-the-Middle (FIM) tasks -- plausible but incorrect completions suc

Do Joint Audio-Video Generation Models Understand Physics?

arXiv:2605.07061v1 Announce Type: cross Abstract: Joint audio-video generation models are rapidly approaching professional production quality, raising a central question: do they understand audio-visu

Hardware

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

arXiv:2605.07985v1 Announce Type: cross Abstract: Selecting the optimal LLM inference configuration requires evaluation across hardware, serving engines, attention backends, and model architectures, s

hardwarearxiv-cs-ai

Local Ai

GESR: Graph-Based Edge Semantic Reconstruction for Stealthy Communication Detection with Benign-Only Training

arXiv:2605.07536v1 Announce Type: cross Abstract: Detecting stealthy malicious communications from flow logs under benign-only training remains a critical challenge in network security. Malicious comm

local-aiarxiv-cs-lg

Local Ai

HMACE: Heterogeneous Multi-Agent Collaborative Evolution for Combinatorial Optimization

arXiv:2605.07214v1 Announce Type: new Abstract: Large Language Models have recently emerged as a promising paradigm for automated heuristic design for NP-hard combinatorial optimization problems. Desp

local-aiarxiv-cs-ai

IntentGrasp: A Comprehensive Benchmark for Intent Understanding

arXiv:2605.06832v1 Announce Type: cross Abstract: Accurately understanding the intent behind speech, conversation, and writing is crucial to the development of helpful Large Language Model (LLM) assis

LithoBench: Benchmarking Large Multimodal Models for Remote-Sensing Lithology Interpretation

arXiv:2605.07640v1 Announce Type: cross Abstract: Remote sensing lithology interpretation is fundamental to geological surveys, mineral exploration, and regional geological mapping. Unlike general lan

Applications

Manufacturing intelligence with Amazon Nova Multimodal Embeddings

In this post, we build a multimodal retrieval system for aerospace manufacturing documents using Amazon Nova Multimodal Embeddings on Amazon Bedrock and Amazon S3 Vectors. We evaluate the system on 26

applicationsaws-ml-blog

MAS-Algorithm: A Workflow for Solving Algorithmic Programming Problems with a Multi-Agent System

arXiv:2605.05949v2 Announce Type: replace Abstract: Algorithmic problem solving serves as a rigorous testbed for evaluating structured reasoning in AI coding systems, as it directly reflects a model's

MathlibPR: Pull Request Merge-Readiness Benchmark for Formal Mathematical Libraries

arXiv:2605.07147v1 Announce Type: cross Abstract: The ecosystem of Lean and Mathlib has become the de facto standard for large language model (LLM) assisted formal reasoning with remarkable successes

mathsf{VISTA}: Decentralized Machine Learning in Adversary Dominated Environments

arXiv:2605.07841v1 Announce Type: cross Abstract: Decentralized machine learning often relies on outsourcing computations, such as gradient evaluations, to untrusted worker nodes. Existing robust aggr

agentsarxiv-cs-ai

MedAction: Towards Active Multi-turn Clinical Diagnostic LLMs

arXiv:2605.07305v1 Announce Type: cross Abstract: Most existing LLM diagnoses are evaluated on static, single-turn settings where complete patient information is provided upfront, an oversimplificatio

Applications

Multimodal synthesis of MRI and tabular data with diffusion in a joint latent space via cross-attention

arXiv:2605.06699v1 Announce Type: cross Abstract: We propose a multimodal latent diffusion model that jointly synthesizes volumetric magnetic resonance imaging (MRI) and tabular clinical data within a

applicationsarxiv-cs-ai

Overview of the TREC 2025 RAGTIME Track

arXiv:2602.10024v2 Announce Type: replace-cross Abstract: The principal goal of the RAG TREC Instrument for Multilingual Evaluation (RAGTIME) track at TREC is to study report generation from multiling

researcharxiv-cs-cl

Physics-Informed Reduced-Order Operator Learning for Hyperelasticity in Continuum Micromechanics

arXiv:2605.07738v1 Announce Type: cross Abstract: Physics-informed operator learning is an attractive candidate for surrogate modeling of microstructures, especially in multiscale finite-element simul

researcharxiv-cs-lg

ProcObject-10K: Benchmarking Object-Centric Procedural Understanding in Instructional Videos

arXiv:2512.03479v2 Announce Type: replace Abstract: Procedural activities are fundamentally driven by object state transitions, yet existing instructional video benchmarks remain action-centric and ca

R-GTD: A Geometric Analysis of Gradient Temporal-Difference Learning in Singular Regimes

arXiv:2601.20599v2 Announce Type: replace-cross Abstract: Gradient temporal-difference (GTD) learning algorithms are widely used for off-policy policy evaluation with function approximation. However,

Self-Programmed Execution for Language-Model Agents

arXiv:2605.06898v1 Announce Type: new Abstract: At the heart of existing language model agents is a fixed orchestrator program responsible for the state transition between consecutive turns. This pape

ShellfishNet: A Domain-Specific Benchmark for Visual Recognition of Marine Molluscs

arXiv:2605.07338v1 Announce Type: new Abstract: The decline of global shellfish biodiversity poses a severe threat to coastal ecosystems. Although artificial intelligence (AI) technologies show potent

TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning

arXiv:2605.07943v1 Announce Type: cross Abstract: Active vision -- where a policy controls its own gaze during manipulation -- has emerged as a key capability for imitation learning, with multiple ind

The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

arXiv:2605.07186v1 Announce Type: cross Abstract: Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect t

Trajectory as the Teacher: Few-Step Discrete Flow Matching via Energy-Navigated Distillation

arXiv:2605.07924v1 Announce Type: cross Abstract: Discrete flow matching generates text by iteratively transforming noise tokens into coherent language, but may require hundreds of forward passes. Dis

TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models

arXiv:2601.18744v2 Announce Type: replace Abstract: Time series are ubiquitous in real-world scenarios and crucial for applications ranging from energy management to traffic control. Consequently, the

Zero-Shot Imagined Speech Decoding via Imagined-to-Listened MEG Mapping

arXiv:2605.08075v1 Announce Type: new Abstract: Decoding imagined speech from non-invasive brain recordings is challenging because imagined datasets are scarce and difficult to align temporally across

safetyarxiv-cs-lg

Agentic coding is a form of machine learning. Generated code is best treated as a blackbox artifact whose behavior and generalization should…

agentsfrancois-chollet--x

Agentic coding is a form of machine learning. Generated code is best treated as a blackbox artifact whose behavior and generalization should be managed via empirical evaluation, like with any ML model

9 May 2026

Huh.

model-releasesethan-mollick--x

Huh. We evaluated an early version of Claude Mythos Preview for risk assessment during a limited window in March 2026. We estimated a 50%-time-horizon of at least 16hrs (95% CI 8.5hrs to 55hrs) on our

9 May 2026

Applications

Thresholds (like “can a robot make a good coffee?”), are not actually very good benchmarks because you can’t track progress towards a goal.

applicationsethan-mollick--x

Ethan Mollick argues that threshold-based benchmarks—binary evaluations of whether AI systems can accomplish specific tasks (such as making good coffee)—are inadequate measures of AI progress because

9 May 2026

Industry

PlayStation sees AI as a ‘powerful tool’ to help make games

As part of an earnings presentation on Friday, Sony shared how it's thinking about AI at the company, including many details about how it's evaluating AI as part of making PlayStation games. Generativ

industrythe-verge-ai

8 May 2026

Adaptive Policy Selection and Fine-Tuning under Interaction Budgets for Offline-to-Online Reinforcement Learning

arXiv:2605.05123v1 Announce Type: new Abstract: In offline-to-online reinforcement learning (O2O-RL), policies are first safely trained offline using previously collected datasets and then further fin

safetyarxiv-cs-lg

Cognitive Twins: Investigating Personalized Thinking Model Building and Its Performance Enhancement with Human-in-the-Loop

arXiv:2605.04761v1 Announce Type: new Abstract: This paper presents the Personalized Thinking Model (PTM), a hierarchical and interpretable learner representation designed for AI supported education.

Collision-Aware Object-Goal Visual Navigation via Two-Stage Deep Reinforcement Learning

arXiv:2502.13498v2 Announce Type: replace-cross Abstract: Object-goal visual navigation aims to reach a specific target object using egocentric visual observations. Recent deep reinforcement learning

agentsarxiv-cs-cv

Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation

arXiv:2512.14954v2 Announce Type: replace Abstract: Computing next-token likelihood ratios between two language models (LMs) is a standard task in training paradigms such as knowledge distillation. Si

researcharxiv-cs-cl

Gaze4HRI: Zero-shot Benchmarking Gaze Estimation Neural-Networks for Human-Robot Interaction

arXiv:2605.04770v1 Announce Type: new Abstract: While zero-shot appearance-based 3D gaze estimation offers significant cost-efficiency by directly mapping RGB images to gaze vectors, its reliability i