Search: “evaluation”

Bayesian Learning to Select Fair Classifiers

arXiv:2610.20514v1 Announce Type: new Abstract: Evaluations of classifiers (AI or human or agents) often assume a batch setting in which every agent produces labels on every example in a test set, whi

Big-Big-SycoBench: A Benchmark to Detect Sycophancy Through Model Responses to Valid vs. Invalid Arguments

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2610.15634v1 Announce Type: new Abstract: Sycophancy in large language models risks amplifying unsupported claims or beliefs across the knowledge ecosystem as these systems increasingly contribu

model-releasesarxiv-cs-ai

15 Oct 2026

Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents

arXiv:2610.18052v1 Announce Type: new Abstract: Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users i

Can You Trust Your Preference Models? Stackelberg Games for Alignment

arXiv:2610.15008v1 Announce Type: new Abstract: Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policie

CASI: Credit Assignment via Set Intervention for Agentic Retrieval

arXiv:2610.20293v1 Announce Type: cross Abstract: Large language models increasingly generate multiple retrieval actions jointly, while downstream systems often provide only one reward for the resulti

CWE-bench: Refocusing Agentic Security from Exploitation to Defense

arXiv:2610.17020v1 Announce Type: cross Abstract: Cybersecurity evaluations of AI agents have focused largely on offense, yet a working exploit does not show that an agent can make software safe. An a

Do Factuality Methods Make LLMs More Factual? Reproducibility and Fair Comparison

arXiv:2610.18415v1 Announce Type: cross Abstract: Large Language Models (LLMs) are aligned to human preferences, after pre-training and supervised fine-tuning steps. However, standard RL alignment ste

Dungeon and Visibility World Generator

arXiv:2610.19523v1 Announce Type: new Abstract: Robotic exploration, coverage, and multi-agent deployment algorithms require reproducible environments with controllable geometry and visibility. This p

agentsarxiv-cs-ro

EvoWeb: A Benchmark for Coding Agents on Evolving Web Applications

arXiv:2610.16402v1 Announce Type: new Abstract: Web application development often proceeds through successive requests that extend an existing implementation. Yet web-generation benchmarks primarily e

From Human Corrections to Learned Segmentations: A Critical Mathematical Review of Information, Learning, and Human Effort

arXiv:2610.20396v1 Announce Type: new Abstract: When does a human correction become useful learning, and what establishes better segmentation at matched human effort? This critical narrative review fo

How Do LLMs Understand Legal Philosophy? Diagnosing Legal-Philosophical Reasoning in LLMs

arXiv:2610.19427v1 Announce Type: new Abstract: Legal reasoning is a fundamental capability for large language models (LLMs) to complete legal AI tasks such as legal judgment prediction. In legal prac

Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

arXiv:2610.18497v1 Announce Type: new Abstract: Tool-call judges evaluate argument values proposed by LLM agents. We find that they accept a correct value less often when their input holds no referenc

KiToke: Distribution-Aware Token Compression for Video Large Language Models

arXiv:2604.03414v2 Announce Type: replace Abstract: Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the lar

LED-BENCH: A Benchmark for Large-format Engineering Drawing Understanding

arXiv:2610.19571v1 Announce Type: new Abstract: Real-world engineering drawings are often large-format and information-dense. Answering practical engineering questions therefore requires both fine-gra

Research

Lost in Implicit Routing: How LLM Agents Fail to Load Relevant Skills at Scale

arXiv:2610.18522v1 Announce Type: new Abstract: Reusable skills let LLM agents load task-specific instructions on demand, but they make execution depend on implicit routing over skill metadata. We tes

researcharxiv-cs-ai

Manaca-1B-Instruct-2: Reproducible Safety Alignment for a Native Brazilian-Portuguese LLM. Negative DPO Results, a Controlled Composition Scan, and a Silent Failure Mode of the LLM Judge

arXiv:2610.20570v1 Announce Type: new Abstract: Brazilian Portuguese remains underserved by open-source language models, and the few that are safety-aligned rarely document what did not work or valida

Measuring and Reducing Unusable Replies of LLM Agents

arXiv:2610.19659v1 Announce Type: new Abstract: People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faith

agentsarxiv-cs-cl

MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue

arXiv:2610.20172v1 Announce Type: cross Abstract: Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel w

Applications

Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications

arXiv:2610.18612v1 Announce Type: cross Abstract: System One models such as Jev change how software makes decisions by mapping open-ended semantic contexts into structured judgments that directly driv

applicationsarxiv-cs-ai

PRIMEdit: Probability Redistribution for Instance-Aware Multi-Object Video Editing With a New Benchmark Dataset

arXiv:2412.12877v3 Announce Type: replace Abstract: Recent Artificial Intelligence (AI)-based video editing has enabled users to edit videos through simple text prompts, significantly simplifying the

Provable Transformer Context Compression for In-Context Reinforcement Learning

model-releasesarxiv-cs-lg

arXiv:2610.20004v1 Announce Type: new Abstract: Transformers can perform in-context reinforcement learning without parameter updates. However, existing theoretical constructions often require context

Query Your Way Out: Benchmarking Multi-Turn Knowledge Reasoning through Generalized Puzzle Solving

arXiv:2610.16133v1 Announce Type: cross Abstract: Puzzles offer a natural testbed for Large Language Models (LLMs) reasoning because they pose verifiable goals under partial information and require mu

Refine Actions If Needed: Adaptive Action Chunk Refinement for Robot Manipulation

arXiv:2610.19765v1 Announce Type: new Abstract: Computationally lightweight manipulation policies predict executable actions with low inference latency, but prediction errors can limit task success. I

safetyarxiv-cs-ro

ROAR: Unifying Runs across Heterogeneous AI-Driven Research Systems

arXiv:2610.03966v2 Announce Type: replace Abstract: Each run of an AI-driven research system (ADRS) is an expensive search over a vast solution space, and dependable evaluation requires many runs, mak

Single-Sample Decoding of a Two-Layer Sound-Word Canvas Model for Thai Klon Pad

arXiv:2610.14980v1 Announce Type: new Abstract: We present a generator of Thai klon pad verse, one stanza from a short topic, and evaluate it against human verse on held-out topics. The generator is a

Single-step Retrosynthesis via Reinforcement Learning from Forward Feedback

arXiv:2610.15141v1 Announce Type: new Abstract: Single-step retrosynthesis is a one-to-many inverse problem, yet most models are trained and evaluated against a single recorded reactant set. Reference

Stochastic CHAOS: Why Deterministic Inference Kills, and Distributional Variability Is the Heartbeat of Artifical Cognition

arXiv:2601.07239v2 Announce Type: replace Abstract: Deterministic inference is a comforting ideal in classical software: the same program on the same input should always produce the same output. As la

The Gap Is in the Geometry: A Convergent Meta-Analysis of Multi-Facet Aerial and Remote Sensing Video Understanding

arXiv:2610.19102v1 Announce Type: new Abstract: Aerial and remote sensing video is increasingly evaluated like web video, by posing multiple-choice questions to multimodal large language models (MLLMs

Towards a Data-Centric Understanding of Spatial Reasoning in Multimodal Large Language Models

arXiv:2610.19108v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have been widely applied in embodied agents and robotics for vision tasks. However, MLLMs suffer from limitat

Truthful and Expected-Payment-Optimal Model Routing in Graph-Based Agentic Workflows

arXiv:2610.19316v1 Announce Type: cross Abstract: Assigning routine tasks to cheaper large language model (LLM) providers can reduce generation cost, but switching providers may require processing acc

VibeServe: Can AI Agents Build Bespoke LLM Serving Systems?

arXiv:2605.06068v2 Announce Type: replace Abstract: For years, we have built LLM serving systems like any other critical infrastructure, with a single general-purpose stack hand-tuned over many engine

When Non-Adversarial Noise Hurts Learning: Benchmarking Skill Self-Evolution in Agents

model-releasesarxiv-cs-lg

arXiv:2610.19910v1 Announce Type: new Abstract: Large language model agents increasingly improve by distilling experience into reusable skills, but errors in that experience can persist in learned rul

Who's Behind the Text? Robust Model Authorship Attribution

arXiv:2610.19583v1 Announce Type: cross Abstract: As large language models (LLMs) are increasingly used to generate text, their misuse has become a growing societal problem, with implications for acad

XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

arXiv:2605.05662v2 Announce Type: replace-cross Abstract: Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. More

A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding

arXiv:2610.15666v1 Announce Type: new Abstract: Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interac

Adversarial Robustness of Camera-Radar Sensor Fusion for Autonomous Driving

arXiv:2610.16012v1 Announce Type: cross Abstract: Autonomous driving (AD) relies on multi-sensor perception to interpret the surrounding environment. However, the robustness of camera--radar fusion un

agentsarxiv-cs-cv

Applications

AIRRI: Towards a Human-Readable, Machine-Resistant Reading Interface

arXiv:2610.18724v1 Announce Type: new Abstract: The rapid advancement of image-to-text extraction systems and large language models (LLMs) poses new challenges for educational assessment by making ass

applicationsarxiv-cs-cv

Research

All-electron machine-learned orbital-free density functional for periodic CHNO systems

arXiv:2610.17975v1 Announce Type: cross Abstract: We introduce STRUCTURES-P, a machine-learned all-electron orbital-free density functional for periodic CHNO systems. Electrostatics are evaluated anal

researcharxiv-cs-lg

AlphaPilot: Multi-Turn Tool-Using LLM Agents for Adaptive Alpha Mining with Reinforcement Learning

arXiv:2610.15488v1 Announce Type: cross Abstract: Formulaic alpha mining seeks interpretable expressions that rank assets by future returns. The task is challenging because it demands simultaneous pro

agentsarxiv-cs-lg

Bayesian Optimization under Uncertainty for Training a Scale Parameter in Stochastic Models

model-releasesarxiv-cs-lg

arXiv:2510.06439v2 Announce Type: replace Abstract: Hyperparameter tuning is a challenging problem especially when the system itself involves uncertainty. Due to noisy function evaluations, optimizati

ChemExplain: A Multimodal Benchmark for Explanatory Chemical Reasoning

arXiv:2610.17183v1 Announce Type: cross Abstract: Current chemistry benchmarks primarily evaluate outcome prediction or reaction pathway generation, but rarely test whether a model can explain why an

EgoOmniWorld: Unifying Egocentric Understanding via Ego-Action-World Interactions

arXiv:2610.18499v1 Announce Type: new Abstract: Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their ti

EgoProbe: Benchmarking Long-Range and Exact Memory in Streaming Egocentric Videos

arXiv:2610.17128v1 Announce Type: new Abstract: Egocentric video provides a continuous first-person record of everyday activities, making it a natural foundation for wearable AI assistants. Unlike off

ForeTac: Predictive Contact Guidance for Generative Robot Policies

arXiv:2610.18118v1 Announce Type: new Abstract: Visually similar actions can produce contact loss, jamming, slippage, or irreversible damage, while tactile feedback may arrive after a critical transit

safetyarxiv-cs-ro

From Classification to Cross-Modal Understanding: Leveraging Vision-Language Models for Fine-Grained Renal Pathology

arXiv:2511.11984v2 Announce Type: replace Abstract: Fine-grained glomerular subtyping is essential for kidney biopsy interpretation, yet it presents a harder setting than the broad organ- or disease-l

Game2Real-Bench: Do Toy Games Reveal and Transfer to Real-World Visual Reasoning?

arXiv:2610.18834v1 Announce Type: new Abstract: Visual reasoning is an important capability for vision-language models (VLMs) to understand visual evidence, spatial structure, constraints, and state c

Research

Information-based Unified Design Framework for NMR Magnetic Tracking System

arXiv:2610.17873v1 Announce Type: new Abstract: Nuclear magnetic resonance (NMR) magnetic tracking offers a promising alternative to magnetic resonance imaging (MRI)-based localization for medical rob

researcharxiv-cs-ro

Interaction-as-a-Judge

arXiv:2610.17895v1 Announce Type: new Abstract: Coding agents are tasked with producing sophisticated interactive applications such as websites, desktop applications, and games. While humans assess th