Search: “evaluation”

MagicSim: A Unified Infrastructure for Executable Embodied Interaction

arXiv:2606.17511v2 Announce Type: replace-cross Abstract: Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

DGX agent

arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However,

model-releasesarxiv-cs-ai

1 Sept 2026

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606.14747v2 Announce Type: replace-cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

arXiv:2608.29286v1 Announce Type: new Abstract: As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructio

Applications

Not All Fallbacks Are Failures: Understanding and Recovering from Fallbacks in Mobile Voice Assistants

arXiv:2608.30738v1 Announce Type: new Abstract: Robust understanding of user input is a core requirement for voice assistants deployed in real-world environments. In practice, these systems encounter

applicationsarxiv-cs-cl

Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelines

arXiv:2608.29589v1 Announce Type: new Abstract: Text-to-image (T2I) safety guardrails fail to generalize equitably to non-standard dialects. Evaluating 23,080 paired prompts across five English dialec

On the Recoverability of Private Information Unlearning in Large Language Models

arXiv:2608.29943v1 Announce Type: cross Abstract: Large language models (LLMs) can memorize sensitive information, raising serious privacy concerns. Machine unlearning offers a potential solution to r

Predicting Future Organ Dysfunction in ICU Patients Using Temporal Convolutional Networks on MIMIC-IV Data

arXiv:2608.29301v1 Announce Type: new Abstract: Predicting future organ dysfunction in Intensive Care Unit (ICU) patients is critical for early clinical intervention, yet existing machine learning app

RealOOB: A Definition-Consistent Real-World Oriented Occlusion Boundary Benchmark

model-releasesarxiv-cs-cv

arXiv:2608.30820v1 Announce Type: new Abstract: Occlusion boundaries (OBs) are pixel-level image boundaries corresponding to surface visibility discontinuities caused by occlusion. Through precise bou

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

arXiv:2608.29109v1 Announce Type: new Abstract: Large language models often answer structurally unanswerable questions, such as computing cot(-540{eg}) or evaluating (1).startswith('1'), instead of ab

safetyarxiv-cs-cl

ShardMemo: Scope-Before-Routing for Agentic Memory Retrieval

arXiv:2601.21545v2 Announce Type: replace Abstract: Agentic systems accumulate persistent memory across sessions, tools, and tasks, and a later request must retrieve from it under two distinct constra

agentsarxiv-cs-ai

SimGuide: Typed Multi-Context User Representations for Preference-Conditioned Agent Planning

arXiv:2608.24888v2 Announce Type: replace Abstract: Agents that act on a user's behalf must plan differently for different users, and increasingly do so from some structured representation of user con

SpatialTrust: A Benchmark for Environmental Risk Recognition in Secure Authentication

model-releasesarxiv-cs-cv

arXiv:2608.29489v1 Announce Type: cross Abstract: Visual environmental risk recognition plays an important role in secure authentication, where a user's surroundings may reveal sensitive information o

State of Health Estimation using Convolutional and Bidirectional LSTM Neural Networks tuned by Bayesian Optimization

arXiv:2608.30593v1 Announce Type: new Abstract: In this research, a novel framework is proposed for the SOH estimation, which employs a hybrid deep learning architecture of a concatenation of a Convol

researcharxiv-cs-lg

Subtraction-Based Tumor Segmentation and Lesion-Centered pCR Prediction for the MAMA-MIA Challenge

arXiv:2608.29162v1 Announce Type: cross Abstract: We describe the submission of team FME to the MAMA-MIA Challenge, which evaluated primary tumor segmentation and prediction of pathological complete r

The Race between Agentic AI Capabilities and Data Quality Control in Online Surveys

arXiv:2608.28597v1 Announce Type: new Abstract: Online surveys are a foundational data collection instrument in a variety of fields, with attention checks serving as critical guardians of response qua

agentsarxiv-cs-ai

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length

agentsarxiv-cs-lg

VibeJam: A User Study Platform for Web Development with Agents

arXiv:2608.29889v1 Announce Type: new Abstract: Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

arXiv:2608.30480v1 Announce Type: new Abstract: Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lac

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

arXiv:2608.28062v2 Announce Type: replace Abstract: Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search

Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts

arXiv:2608.29446v1 Announce Type: new Abstract: Judgments about psychological distress are socially situated: what counts as concerning hinges on community norms around emotional expression, vulnerabi

A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

arXiv:2608.28063v1 Announce Type: new Abstract: Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) ob

safetyarxiv-cs-cv

Local Ai

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

arXiv:2608.28458v1 Announce Type: new Abstract: Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, a

local-aiarxiv-cs-cl

Industry

AWS recognized as a Leader in The Forrester Wave: AI Infrastructure Solutions, Q4 2025

We're excited to share that AWS has been recognized as a Leader in The Forrester Wave: AI Infrastructure Solutions, Q4 2025. In this evaluation of 13 providers, AWS received the highest score in the S

industryaws-ml-blog

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

arXiv:2608.27729v1 Announce Type: new Abstract: Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students

Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoders

arXiv:2608.28316v1 Announce Type: new Abstract: Static importance scores compress visual evidence into a single ranking, but the value of remaining evidence can change after one cue has been observed.

Applications

Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring

arXiv:2608.28237v1 Announce Type: new Abstract: Predictive Process Monitoring (PPM) models are increasingly deployed in dynamic environments where concept drift causes the underlying process distribut

applicationsarxiv-cs-lg

Entity-Memory Graph Retrieval Improves Evidence Coverage in Long-Conversation Question Answering

arXiv:2608.27925v1 Announce Type: new Abstract: Entity-Memory graph retrieval keeps dialogue turns as verbatim Memory nodes, links repeated mentions through shared Entities, and connects adjacent Memo

FedEHR-Agents: Federated Agentic Optimization for Automated EHR Modeling

model-releasesarxiv-cs-lg

arXiv:2608.27856v1 Announce Type: new Abstract: Recent advances in large language models are enabling autonomous clinical agents to perform increasingly complex electronic health record (EHR) modeling

How Proper Scoring Rules Shape LLM Forecasting

arXiv:2608.28482v1 Announce Type: new Abstract: This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training

safetyarxiv-cs-lg

Image Augmentation as Test Generation for Deep Learning-Based Image Retrieval Systems

arXiv:2608.27502v1 Announce Type: cross Abstract: Ensuring the reliability of deep learning-based image retrieval systems is a software engineering challenge. This paper presents a dual contribution:

Lossy Event Compression: From Event Stream Distortion to Task Performance

arXiv:2608.28429v1 Announce Type: new Abstract: Event cameras generate asynchronous, sparse data streams with microsecond temporal resolution, but in moderate-to-high motion scenes they can produce as

NumBench: Diagnosing Counting Failures in Text-to-Image Models

model-releasesarxiv-cs-cv

arXiv:2608.28206v1 Announce Type: new Abstract: Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We int

One year in a forest: Analyzing the challenges of autonomous navigation in subarctic environments

arXiv:2608.27628v1 Announce Type: new Abstract: Subarctic regions have the potential to see increased deployment of autonomous robots in applications including forestry, mining, and environmental moni

agentsarxiv-cs-ro

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

arXiv:2608.28378v1 Announce Type: new Abstract: Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally comp

QUORUM: QUality-Optimized Routing Using Multiple annotators

arXiv:2608.27974v1 Announce Type: new Abstract: Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large

Real-Time Musculoskeletal Surrogates for Pediatric Cerebral Palsy: a Credibility Pilot

arXiv:2608.28371v1 Announce Type: new Abstract: Real-time musculoskeletal (MSK) surrogates could support personalized rehabilitation for children with cerebral palsy (CP), but their credibility depend

Remote Human and Robot Interaction for Greenhouse Gardening Using Virtual Reality

arXiv:2608.27545v1 Announce Type: new Abstract: This study evaluates the effectiveness of remote human-robot interaction using virtual reality for leaf inspection and soil moisture assessment in a gre

researcharxiv-cs-ro

REPLICANT: Learning Policies for Evading and Hardening Malware Detectors

arXiv:2608.28499v1 Announce Type: new Abstract: To determine the real-world effectiveness of machine learning based malware detection, it is vital to evaluate its robustness against highly capable adv

safetyarxiv-cs-lg

Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis

arXiv:2608.27471v1 Announce Type: cross Abstract: Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political d

Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires

arXiv:2510.25384v2 Announce Type: replace Abstract: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced p

Steering Multimodal Large Language Models Decoding for Context-Aware Safety

arXiv:2509.19212v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisi

safetyarxiv-cs-cl

Synth-JDoc: Synthesizing a Japanese Document Image Dataset for OCR with Diverse Layouts and Embedded Images

arXiv:2608.28248v1 Announce Type: cross Abstract: The ability of Large Vision Language Models (LVLMs) to read text within document images is crucial, as it enables various applications such as Documen

What Makes Agent Memory Useful for Reliable Unanswerable Question Handling?

arXiv:2608.27924v1 Announce Type: new Abstract: Reliable handling of unanswerable questions (UAQs) is critical for trustworthy LLM-based agents. Although memory is widely used in agent systems, its ro

agentsarxiv-cs-cl

When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI

arXiv:2608.28518v1 Announce Type: cross Abstract: We investigate whether automatic speech recognition (ASR) errors in user input can lead to unsafe outputs from Embodied AI (EAI) models. We find that

safetyarxiv-cs-cl

Tutorials

1/ An equally consistent description is: Multiple model instances encountered persistent shared state, inherited tools and discoveries from …

1/ An equally consistent description is: Multiple model instances encountered persistent shared state, inherited tools and discoveries from earlier runs, and optimized against common evaluation incent

tutorialsyann-lecun--x

30 Aug 2026

A Task-Centric Ontology and Deterministic Domain Rules as a Verifiable Core for AI-Assisted Chemistry Problem Solving

arXiv:2608.26164v1 Announce Type: new Abstract: Large language models can interpret natural-language chemistry questions, but their internal reasoning is difficult to inspect, constrain, and validate.

researcharxiv-cs-ai

28 Aug 2026

Accelerating Scientific Research with Gemini in the Real-World

arXiv:2608.26701v1 Announce Type: new Abstract: We present an extension and comprehensive real-world validation of Co-Scientist, a Gemini-based multi-agent system designed to accelerate end-to-end sci