Search: “evaluation”

Learning from Many Voices: Literary MT Using Multi-Reference Human and Synthetic Data

arXiv:2412.18707v3 Announce Type: replace Abstract: Multiple valid translations of a single literary work naturally exist. We investigate strategies for leveraging these multi-reference datasets to im

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Learning to Follow In-Context Watermark Instructions via Self-Distillation

DGX agent

arXiv:2608.29030v1 Announce Type: new Abstract: In-context watermarking (ICW) prepends an instruction to a query asking the model to embed a statistically detectable signal in its response. It thus eq

model-releasesarxiv-cs-ai

1 Sept 2026

LLM Judges as Raters: A Pre-Registered Audit of Severity, Halo, Reliability, and Version Instability in LLM Essay Scoring on Public Corpora

arXiv:2608.29517v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as essay graders in learning analytics, evaluated almost exclusively with agreement statistics. Educa

Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs

arXiv:2511.18931v2 Announce Type: replace-cross Abstract: Modern large language models increasingly integrate internal web-based retrieval to provide real-time answers, yet it remains unclear how effe

researcharxiv-cs-ai

Looking Again: Measuring Sycophancy in the Reasoning Chains of Multimodal Models Under Pressure

arXiv:2608.28623v1 Announce Type: cross Abstract: Large multimodal reasoning models (LMRMs) are getting increasingly capable, primarily through generating explicit chain-of-thought reasoning before an

Machine Learning-Enhanced Tabu Search for Tactical Wireless Network Design

arXiv:2608.28627v1 Announce Type: new Abstract: Designing high-performance tactical wireless networks under realistic operational constraints gives rise to challenging combinatorial optimization probl

MagicSim: A Unified Infrastructure for Executable Embodied Interaction

arXiv:2606.17511v2 Announce Type: replace-cross Abstract: Robot learning and embodied agents now require simulation to serve as a shared execution substrate linking control, skills, and planning, not

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents

arXiv:2508.13186v2 Announce Type: replace-cross Abstract: AI agents with advanced reasoning and tool-use capabilities have demonstrated impressive performance in web browsing for deep search. However,

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606.14747v2 Announce Type: replace-cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context

MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputs

arXiv:2608.29286v1 Announce Type: new Abstract: As Multimodal Large Language Models (MLLMs) evolve into sophisticated interactive assistants, their reliability depends not only on following instructio

Applications

Not All Fallbacks Are Failures: Understanding and Recovering from Fallbacks in Mobile Voice Assistants

arXiv:2608.30738v1 Announce Type: new Abstract: Robust understanding of user input is a core requirement for voice assistants deployed in real-world environments. In practice, these systems encounter

applicationsarxiv-cs-cl

Not Safe for All: Auditing the Dialect Penalty in Text-to-Image Safety Pipelines

arXiv:2608.29589v1 Announce Type: new Abstract: Text-to-image (T2I) safety guardrails fail to generalize equitably to non-standard dialects. Evaluating 23,080 paired prompts across five English dialec

On the Recoverability of Private Information Unlearning in Large Language Models

arXiv:2608.29943v1 Announce Type: cross Abstract: Large language models (LLMs) can memorize sensitive information, raising serious privacy concerns. Machine unlearning offers a potential solution to r

Predicting Future Organ Dysfunction in ICU Patients Using Temporal Convolutional Networks on MIMIC-IV Data

arXiv:2608.29301v1 Announce Type: new Abstract: Predicting future organ dysfunction in Intensive Care Unit (ICU) patients is critical for early clinical intervention, yet existing machine learning app

RealOOB: A Definition-Consistent Real-World Oriented Occlusion Boundary Benchmark

model-releasesarxiv-cs-cv

arXiv:2608.30820v1 Announce Type: new Abstract: Occlusion boundaries (OBs) are pixel-level image boundaries corresponding to surface visibility discontinuities caused by occlusion. Through precise bou

Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

arXiv:2608.29109v1 Announce Type: new Abstract: Large language models often answer structurally unanswerable questions, such as computing cot(-540{eg}) or evaluating (1).startswith('1'), instead of ab

safetyarxiv-cs-cl

ShardMemo: Scope-Before-Routing for Agentic Memory Retrieval

arXiv:2601.21545v2 Announce Type: replace Abstract: Agentic systems accumulate persistent memory across sessions, tools, and tasks, and a later request must retrieve from it under two distinct constra

agentsarxiv-cs-ai

SimGuide: Typed Multi-Context User Representations for Preference-Conditioned Agent Planning

arXiv:2608.24888v2 Announce Type: replace Abstract: Agents that act on a user's behalf must plan differently for different users, and increasingly do so from some structured representation of user con

SpatialTrust: A Benchmark for Environmental Risk Recognition in Secure Authentication

model-releasesarxiv-cs-cv

arXiv:2608.29489v1 Announce Type: cross Abstract: Visual environmental risk recognition plays an important role in secure authentication, where a user's surroundings may reveal sensitive information o

State of Health Estimation using Convolutional and Bidirectional LSTM Neural Networks tuned by Bayesian Optimization

arXiv:2608.30593v1 Announce Type: new Abstract: In this research, a novel framework is proposed for the SOH estimation, which employs a hybrid deep learning architecture of a concatenation of a Convol

researcharxiv-cs-lg

Subtraction-Based Tumor Segmentation and Lesion-Centered pCR Prediction for the MAMA-MIA Challenge

arXiv:2608.29162v1 Announce Type: cross Abstract: We describe the submission of team FME to the MAMA-MIA Challenge, which evaluated primary tumor segmentation and prediction of pathological complete r

The Race between Agentic AI Capabilities and Data Quality Control in Online Surveys

arXiv:2608.28597v1 Announce Type: new Abstract: Online surveys are a foundational data collection instrument in a variety of fields, with attention checks serving as critical guardians of response qua

agentsarxiv-cs-ai

Universal Transformers for Circuit Computations: Perfect Length Generalization in Tiny Transformers

arXiv:2608.31067v1 Announce Type: new Abstract: Learning generalizable algorithmic computations remains a challenge for neural networks, as reflected in persistent failures on compositional and length

agentsarxiv-cs-lg

VibeJam: A User Study Platform for Web Development with Agents

arXiv:2608.29889v1 Announce Type: new Abstract: Programming with AI is increasingly agentic, users prompt LLMs to directly edit their code and review the changes, with adoption growing especially for

VisER: Visual Evidence and Reliance for Object Hallucination Detection in LVLMs

arXiv:2608.30480v1 Announce Type: new Abstract: Object hallucination remains a persistent reliability issue in large vision-language models, where generated object mentions may sound plausible but lac

WeAgent-MMSearch: Native Text-Vision Interaction for Multimodal Search Agents

arXiv:2608.28062v2 Announce Type: replace Abstract: Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search

Whose Assessment of Distress? Community Perspectives and LLM Alignment on Well-Being Posts

arXiv:2608.29446v1 Announce Type: new Abstract: Judgments about psychological distress are socially situated: what counts as concerning hinges on community norms around emotional expression, vulnerabi

A Controlled Audit of Architectural Complexity in Uncertainty-Aware Multi-Organ Ultrasound Classification

arXiv:2608.28063v1 Announce Type: new Abstract: Multi-organ ultrasound classifiers increasingly combine attention, mixture-of-experts routing, uncertainty gating, and evidential deep learning (EDL) ob

safetyarxiv-cs-cv

Local Ai

Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents

arXiv:2608.28458v1 Announce Type: new Abstract: Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, a

local-aiarxiv-cs-cl

Industry

AWS recognized as a Leader in The Forrester Wave: AI Infrastructure Solutions, Q4 2025

We're excited to share that AWS has been recognized as a Leader in The Forrester Wave: AI Infrastructure Solutions, Q4 2025. In this evaluation of 13 providers, AWS received the highest score in the S

industryaws-ml-blog

Below the Noise Floor: Bimodal Seed Collapse and Distinct Failure Modes in Small-Model Knowledge Distillation

arXiv:2608.27729v1 Announce Type: new Abstract: Function routing -- selecting the correct API call from a fixed catalog given a natural-language request -- is a deployment problem where small students

Conditional Visual Evidence Utility: State-Dependent Rank Reversals in Frozen Vision-Language Encoders

arXiv:2608.28316v1 Announce Type: new Abstract: Static importance scores compress visual evidence into a single ranking, but the value of remaining evidence can change after one cue has been observed.

Applications

Efficient Online Continual Foundation Model Fine-Tuning for Predictive Process Monitoring

arXiv:2608.28237v1 Announce Type: new Abstract: Predictive Process Monitoring (PPM) models are increasingly deployed in dynamic environments where concept drift causes the underlying process distribut

applicationsarxiv-cs-lg

Entity-Memory Graph Retrieval Improves Evidence Coverage in Long-Conversation Question Answering

arXiv:2608.27925v1 Announce Type: new Abstract: Entity-Memory graph retrieval keeps dialogue turns as verbatim Memory nodes, links repeated mentions through shared Entities, and connects adjacent Memo

FedEHR-Agents: Federated Agentic Optimization for Automated EHR Modeling

model-releasesarxiv-cs-lg

arXiv:2608.27856v1 Announce Type: new Abstract: Recent advances in large language models are enabling autonomous clinical agents to perform increasingly complex electronic health record (EHR) modeling

How Proper Scoring Rules Shape LLM Forecasting

arXiv:2608.28482v1 Announce Type: new Abstract: This paper evaluates how reward function choice shapes the performance and behavior of LLM forecasters. We compare five proper scoring rules as training

safetyarxiv-cs-lg

Image Augmentation as Test Generation for Deep Learning-Based Image Retrieval Systems

arXiv:2608.27502v1 Announce Type: cross Abstract: Ensuring the reliability of deep learning-based image retrieval systems is a software engineering challenge. This paper presents a dual contribution:

Lossy Event Compression: From Event Stream Distortion to Task Performance

arXiv:2608.28429v1 Announce Type: new Abstract: Event cameras generate asynchronous, sparse data streams with microsecond temporal resolution, but in moderate-to-high motion scenes they can produce as

NumBench: Diagnosing Counting Failures in Text-to-Image Models

model-releasesarxiv-cs-cv

arXiv:2608.28206v1 Announce Type: new Abstract: Text-to-image (T2I) models often generate the wrong number of objects, yet existing benchmarks are too small or weakly controlled to explain why. We int

One year in a forest: Analyzing the challenges of autonomous navigation in subarctic environments

arXiv:2608.27628v1 Announce Type: new Abstract: Subarctic regions have the potential to see increased deployment of autonomous robots in applications including forestry, mining, and environmental moni

agentsarxiv-cs-ro

PersonaForge: Realistic Multi-Turn User Simulation for Agentic Systems

arXiv:2608.28378v1 Announce Type: new Abstract: Large language models are increasingly used as agentic workflow executors, yet existing training data and benchmarks largely assume informationally comp

QUORUM: QUality-Optimized Routing Using Multiple annotators

arXiv:2608.27974v1 Announce Type: new Abstract: Data annotation remains a central bottleneck in natural language processing, requiring human effort to obtain high-quality labels at scale. While Large

Real-Time Musculoskeletal Surrogates for Pediatric Cerebral Palsy: a Credibility Pilot

arXiv:2608.28371v1 Announce Type: new Abstract: Real-time musculoskeletal (MSK) surrogates could support personalized rehabilitation for children with cerebral palsy (CP), but their credibility depend

Remote Human and Robot Interaction for Greenhouse Gardening Using Virtual Reality

arXiv:2608.27545v1 Announce Type: new Abstract: This study evaluates the effectiveness of remote human-robot interaction using virtual reality for leaf inspection and soil moisture assessment in a gre

researcharxiv-cs-ro

REPLICANT: Learning Policies for Evading and Hardening Malware Detectors

arXiv:2608.28499v1 Announce Type: new Abstract: To determine the real-world effectiveness of machine learning based malware detection, it is vital to evaluate its robustness against highly capable adv

safetyarxiv-cs-lg

Retrieving Relations, Detecting Fallacies: A RAG Approach to Political Debate Analysis

arXiv:2608.27471v1 Announce Type: cross Abstract: Fallacies are arguments that employ invalid reasoning, making their automatic detection critical in sensitive contexts such as high-stakes political d

Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires

arXiv:2510.25384v2 Announce Type: replace Abstract: Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced p

Steering Multimodal Large Language Models Decoding for Context-Aware Safety

arXiv:2509.19212v2 Announce Type: replace Abstract: Multimodal Large Language Models (MLLMs) are increasingly deployed in real-world applications, yet their ability to make context-aware safety decisi

safetyarxiv-cs-cl