Search: “benchmark”

HERO'S JOURNEY: Testing Complex Rule Induction with Text Games

arXiv:2606.02556v1 Announce Type: new Abstract: We introduce HERO'S JOURNEY, a benchmark for rule induction in goal-directed episodic tasks, where agents must infer hidden rules from demonstrations an

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions

DGX agent

arXiv:2606.01703v1 Announce Type: cross Abstract: We address the challenge of generating high-fidelity, long-form soundtracks that remain coherent across scene transitions. Existing AI music systems a

model-releasesarxiv-cs-ai

2 Jun 2026

Learning from Saturated Data: Signals Beyond Correctness for LLM Training

arXiv:2606.01436v1 Announce Type: new Abstract: The growing capabilities of large language models (LLMs) have led to the saturation of many benchmarks and training datasets used to improve them. Motiv

researcharxiv-cs-cl

Make a Video Call with LLM: A Measurement Campaign over Six Mainstream Apps

arXiv:2510.00481v2 Announce Type: replace-cross Abstract: In 2025, Large Language Model (LLM) services have launched a new feature -- AI video chat -- allowing users to interact with AI agents via rea

Measurement-Driven Early Warning of Reliability Breakdown in 5G NSA Railway Networks

arXiv:2511.08851v5 Announce Type: replace-cross Abstract: This paper presents a measurement-driven study of early warning for reliability breakdown events in 5G non-standalone (NSA) railway networks.

MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue

arXiv:2606.00622v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) demonstrate remarkable visual understanding, yet their reliability in interactive settings is severely undermin

Princeton365: A Diverse Dataset with Accurate Camera Pose

arXiv:2506.09035v2 Announce Type: replace Abstract: We introduce Princeton365, a large-scale diverse dataset of 365 videos with accurate camera pose. Our dataset bridges the gap between accuracy and d

Rethinking Amortized Neural Representations for High-Resolution Terrain Elevation Data

arXiv:2606.00404v1 Announce Type: new Abstract: Implicit neural representations (INRs) model a signal as a continuous coordinate-to-value function. For terrain elevation data, this supports analytic d

Safety

Silent Failures in Federated Personalization of Foundation Models

arXiv:2606.00947v1 Announce Type: cross Abstract: Foundation models are increasingly personalized on decentralized private data through federated learning and are now deployed at scale under growing r

safetyarxiv-cs-ai

Agents

The quarq agent is built on LangGraph! LangGraph makes it easy to build complex memory systems (quarq is now at the top of the LongMemEval l…

The Quarq agent is constructed using LangGraph, a framework that simplifies the development of complex memory systems. Quarq has achieved a top ranking on the LongMemEval benchmark, demonstrating the

agentsharrison-chase--x

Topological Ignorability for Structural Causal Effects Beyond Means

arXiv:2606.01184v1 Announce Type: cross Abstract: Many interventions alter the structure of an outcome distribution rather than its mean: they can split a population into disconnected regimes, create

CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law

arXiv:2605.30497v1 Announce Type: new Abstract: RAG-based legal assistants have been growing in popularity, but LLM hallucinations remain a key issue and potentially undermines justice. While benchmar

From Leaky Thoughts to Private Reasoning: Controlling What LRMs Say to Themselves

arXiv:2602.24210v2 Announce Type: replace-cross Abstract: Large reasoning models (LRMs) produce reasoning traces (RTs) that often contain sensitive information. These leaky thoughts are difficult to c

researcharxiv-cs-ai

Safety

Healthcare Mechanisms from Policy-as-Code Search under Strategic Provider Response

arXiv:2605.30680v1 Announce Type: new Abstract: Healthcare mechanisms are inseparable from the strategic provider response they induce: existing healthcare AI benchmarks hold this response fixed and s

safetyarxiv-cs-ai

Hedging on the Frontier: Learning New Tasks with Few Samples

arXiv:2605.30997v1 Announce Type: cross Abstract: When a learner faces a new task with few samples, it must leverage any available side information. In practice, this often comes in the form of model

Linear Ordering Problem: Time for a Change

arXiv:2605.31051v1 Announce Type: cross Abstract: The Linear Ordering Problem (LOP) is a fundamental combinatorial optimization problem with important applications in areas such as economics, social c

MAVEN: Improving Generalization in Agentic Tool Calling

arXiv:2605.30738v1 Announce Type: new Abstract: Generalization across agentic tool-calling environments remains a central challenge for reliable agentic reasoning systems. Although large language mode

MineExplorer: Evaluating Open-World Exploration of MLLM Agents in Minecraft

arXiv:2605.30931v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) have shown strong capabilities in perception, reasoning, and action generation. However, their ability to susta

OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries

arXiv:2605.06235v2 Announce Type: replace-cross Abstract: Retrieval benchmarks are increasingly saturating, but we argue that efficient search is far from a solved problem. We identify a class of quer

researcharxiv-cs-ai

SAW-Bench: Learning Situated Awareness in the Real World

arXiv:2602.16682v2 Announce Type: replace Abstract: A core aspect of human perception is situated awareness, the ability to relate ourselves to the surrounding physical environment and reason over pos

Agents

Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA

arXiv:2605.29277v1 Announce Type: cross Abstract: We present Code-QA-Bench, a fully automated framework for synthesizing repository-level code understanding benchmarks that separates genuine code comp

agentsarxiv-cs-ai

Data filtering methods for training language models

arXiv:2605.29807v1 Announce Type: cross Abstract: Data quality is a critical factor in the effectiveness of machine learning models. Label errors, present even in widely used benchmarks, introduce noi

researcharxiv-cs-ai

Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning

arXiv:2508.19202v3 Announce Type: replace Abstract: Scientific problem solving poses unique challenges for LLMs, requiring both deep domain knowledge and the ability to apply such knowledge through co

DirectorBench: Diagnosing Long-Form Video Generation with Personalized Multi-Agent Evaluation

arXiv:2605.30090v1 Announce Type: new Abstract: Long-form video generation is rapidly moving from short, single-scene synthesis toward minute-long, multi-shot creation with narrative structure, cinema

Entity-Collision: A Stratified Protocol for Attributing Retrieval Lift in Agent Memory

arXiv:2605.29630v1 Announce Type: cross Abstract: End-to-end agent-memory benchmarks report a single hit@k per retriever, confounding lexical leakage (uncontrolled query/gold/distractor entity overlap

GTA: Generating Long-Horizon Tasks for Web Agents at Scale

arXiv:2605.29218v1 Announce Type: new Abstract: Web agents, which couple language models with browsing and tool-use capabilities, show promise as open web assistants. Yet progress is increasingly limi

GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

arXiv:2605.29532v1 Announce Type: cross Abstract: Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an a

How Coding Agents Fail Their Users: A Large-Scale Analysis of Developer-Agent Misalignment in 20,574 Real-World Sessions

arXiv:2605.29442v1 Announce Type: cross Abstract: AI coding agents increasingly act directly within software environments, yet existing analyses of their failures rely on benchmark trajectories that m

Internal Representation, Not Clinical Knowledge: Where Apparent LLM Triage Failures Originate

arXiv:2605.29889v1 Announce Type: cross Abstract: Patient-voiced clinical-triage benchmarks report high under-triage rates for consumer LLMs for constrained multiple-choice output, yet the same cases

Multimodal LLMs See Sentiment

arXiv:2508.16873v3 Announce Type: replace Abstract: Understanding how visual content conveys sentiment is increasingly important in a digital landscape dominated by imagery. However, sentiment percept

Prioritize the Process, Not Just the Outcome: Rewarding Latent Thought Trajectories Improves Reasoning in Looped Language Models

arXiv:2602.10520v3 Announce Type: replace Abstract: Looped Language Models (LoopLMs) perform multi-step latent reasoning prior to token generation and outperform conventional LLMs on reasoning benchma

RoboWits: Unexpected Challenges for Robotic Creative Problem Solving

arXiv:2605.30326v1 Announce Type: cross Abstract: The ability to reason, adapt, and creatively solve problems under unexpected challenges is essential for robots operating in real-world environments.

SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

arXiv:2605.30329v1 Announce Type: new Abstract: Autonomous AI research agents aim to accelerate scientific discovery by automating the research pipeline, from hypothesis generation to peer review. How

The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure

arXiv:2605.29087v1 Announce Type: new Abstract: Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers. Under sustained

VideoFDB: Evaluating Full-Duplex Vision-Speech Capabilities in Conversational Agents

arXiv:2605.30256v1 Announce Type: cross Abstract: Natural human conversation is full-duplex and audio-visual: people simultaneously speak and listen while continuously interpreting and producing nonve

Analyzing Quality-Latency-Resource Trade-offs in a Technical Documentation RAG Assistant Using LoRA Adaptation

arXiv:2605.28222v1 Announce Type: new Abstract: We study quality-latency-resource trade-offs in a documentation-grounded retrieval-augmented generation (RAG) system that uses Low-Rank Adaptation (LoRA

Assessing Factual Music Comprehension in Large Audio Language Models

arXiv:2511.05550v2 Announce Type: replace-cross Abstract: Large audio language models (LALMs) leverage multimodal representations to generate open-ended answers to natural language queries about audio

Beyond Model Ranking: Predictability-Aligned Evaluation for Time Series Forecasting

arXiv:2509.23074v3 Announce Type: replace-cross Abstract: In the era of increasingly complex AI models for time series forecasting, progress is often measured by marginal improvements on benchmark lea

BREAKING: Anthropic just dropped Opus 4.8—and it is a MONSTER We've been testing for about a week @every and our verdict is they could've ju…

BREAKING: Anthropic just dropped Opus 4.8—and it is a MONSTER We've been testing for about a week @every and our verdict is they could've just called it Opus 5, it's that good. Here's our vibe check:

model-releasesthariq--x

Build a test suite that grows with your agent with dataset management in Amazon Bedrock AgentCore

model-releasesaws-ml-blog

Agent evaluation is most powerful when you combine fast-moving online signals with stable offline baselines. To understand whether your agent is truly improving over time, you need a fixed benchmark a

Claude Opus 4.8 is now available in Cursor. On CursorBench, it's able to work much more efficiently than Opus 4.7. We've also found it to be…

Claude Opus 4.8 is now available as a model option in the Cursor code editor. According to Cursor's benchmarking, Opus 4.8 demonstrates improved efficiency compared to its predecessor Opus 4.7. The up

model-releasescursor--x

FLORO: A Multimodal Geospatial Foundation Model for Ecological Remote Sensing Across Sensors and Scales

arXiv:2605.28174v1 Announce Type: cross Abstract: Foundation models offer a promising route to transferable remote sensing representations, but many current approaches depend on very large pretraining

Safety

From Affect to Complex Behavior: Advancing Multimodal Human-Centered AI at the 10th ABAW Workshop & Competition

arXiv:2605.27451v1 Announce Type: new Abstract: The 10th Affective & Behavior Analysis in-the-Wild (ABAW) Workshop and Competition, held at CVPR 2026, continues to advance research on modelling, analy

safetyarxiv-cs-cv

Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows

arXiv:2605.27922v1 Announce Type: new Abstract: LLM agents are increasingly deployed as executable systems that use tools, modify workspaces, and produce concrete artifacts. In such workflows, perform

Tutorials

Learning to Assess the Reliability of Number-of-Runs Estimation in Stochastic Optimization

arXiv:2605.28309v1 Announce Type: new Abstract: In large-scale benchmarking of stochastic optimization algorithms, the key challenge is no longer whether repeated runs are needed for reliability, but

tutorialsarxiv-cs-lg

LiveBrowseComp: Are Search Agents Searching, or Just Verifying What They Already Know?

arXiv:2605.28721v1 Announce Type: new Abstract: Are LLM-based search agents genuinely searching, or using the web to verify what they already know? We study this question on BrowseComp with three diag

MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generation

arXiv:2605.28035v1 Announce Type: new Abstract: In recent years, Multi-Talker Audio-Video Generation (MTAVG) models have shown promising performance on fundamental metrics such as lip-sync and audio-v

Pressure-Testing Deception Probes in LLMs: Scaling, Robustness, and the Geometry of Deceptive Representations

arXiv:2605.27958v1 Announce Type: cross Abstract: Linear probes trained on LLM activations are increasingly proposed as deception-detection metrics, yet report AUROC exceeding 0.96 on clean benchmarks