Search: “evaluation”

VeriTrip: A Verifiable Benchmark for Travel Planning Agents over Unstructured Web Corpora

arXiv:2605.28683v1 Announce Type: new Abstract: Existing benchmarks have laid the foundation for travel planning agents by establishing API-centric paradigms. However, as the capabilities of Autonomou

28 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

When Think-with-Image Meets Safety: What Determines Multimodal Jailbreak Robustness?

DGX agent

arXiv:2605.27932v1 Announce Type: cross Abstract: Think-with-image reasoning is emerging as a new inference paradigm for large vision-language models, but its safety implications remain poorly underst

safetyarxiv-cs-ai

28 May 2026

Local Ai

AIDG: A Formal Decomposition of Information Extraction and Containment Asymmetries in Multi-Turn LLM Dialogue

arXiv:2602.17443v2 Announce Type: replace Abstract: Multi-turn LLM evaluation is typically reported as a single win-rate scalar, conflating distinct capabilities. We introduce AIDG (Adversarial Inform

local-aiarxiv-cs-cl

Benchmark Leakage Trap: Can We Trust LLM-based Recommendation?

arXiv:2602.13626v3 Announce Type: replace Abstract: The expanding integration of Large Language Models (LLMs) into recommender systems poses critical challenges to evaluation reliability. This paper i

Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

arXiv:2605.26562v1 Announce Type: new Abstract: While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift towar

ChartAct: A Benchmark for Dynamic Chart Understanding

model-releasesarxiv-cs-cv

arXiv:2605.26994v1 Announce Type: new Abstract: Charts are widely used to present complex data for analysis and decision making. Existing chart understanding benchmarks mainly focus on static charts,

Composition Collapse: Stable Factual Knowledge Does Not Imply Compositional Reasoning

arXiv:2605.26789v1 Announce Type: new Abstract: Post-training is routinely evaluated through aggregate benchmark scores that treat multi-hop reasoning as a single capability -- as if a model that answ

Counterfactual Credit Policy Optimization for Multi-Agent Collaboration

arXiv:2603.21563v2 Announce Type: replace Abstract: Collaborative multi-agent large language models (LLMs) can solve complex reasoning tasks by decomposing roles, but reinforcement learning for such s

Drive-P2D: A Progressive Perception-to-Decision Benchmark for VLMs in Autonomous Driving

arXiv:2601.14702v2 Announce Type: replace Abstract: Autonomous driving requires reliable perception and safe decision-making in complex scenarios. Recent vision-language models (VLMs) demonstrate reas

EdgeFlow: Edge-Map Augmented VLM-Based Flowchart Processing for Industrial Requirements Engineering

arXiv:2605.27332v1 Announce Type: cross Abstract: Flowcharts are widely used in industrial requirements, but usually remain embedded as static images. Vision Language Models (VLMs) show promise in the

HTMLCure: Turning Browser Experience into State Guided Repair for Interactive HTML

arXiv:2605.26807v1 Announce Type: cross Abstract: LLMs can now produce full HTML pages, but many of those pages are only superficially correct: they render once, then fail under scroll, hover, click,

InfoSynth: Information-Guided Benchmark Synthesis for LLMs

arXiv:2601.00575v2 Announce Type: replace Abstract: Large language models (LLMs) have demonstrated significant advancements in reasoning and code generation, but efficiently creating new benchmarks to

It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

arXiv:2605.26731v1 Announce Type: new Abstract: A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models n

MATCHA: Matching Text via Contrastive Semantic Alignment

arXiv:2605.27345v1 Announce Type: new Abstract: Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g.,

safetyarxiv-cs-cl

OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning

arXiv:2505.17163v2 Announce Type: replace-cross Abstract: Recent advancements in multimodal slow-thinking systems have demonstrated remarkable performance across various visual reasoning tasks. Howeve

Tutorials

Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

arXiv:2605.26840v1 Announce Type: new Abstract: Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of language models. However, for tasks such as

tutorialsarxiv-cs-cl

ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

arXiv:2601.21008v3 Announce Type: replace-cross Abstract: Operations Research practitioners debug infeasible models through an iterative process: inspecting Irreducible Infeasible Subsystems ( IIS), i

Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts

arXiv:2602.03545v2 Announce Type: replace Abstract: Evaluating AI systems that interact with humans requires understanding their behavior across diverse user populations, but collecting representative

agentsarxiv-cs-ai

PRBench: A Standardized Probabilistic Robustness Benchmark

model-releasesarxiv-cs-cv

arXiv:2511.01724v3 Announce Type: replace Abstract: Deep learning models are notoriously vulnerable to imperceptible perturbations. Most existing research centers on adversarial robustness (AR), which

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

arXiv:2605.26548v1 Announce Type: cross Abstract: Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. E

SPEAR: Code-Augmented Agentic Prompt Optimization

arXiv:2605.26275v1 Announce Type: new Abstract: Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed

agentsarxiv-cs-cl

The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance

arXiv:2601.07085v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based conversational AI systems present a challenge to human cognition that current frameworks for understanding mi

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

arXiv:2605.26418v1 Announce Type: cross Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every wor

A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

arXiv:2605.25652v1 Announce Type: new Abstract: Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling

AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

arXiv:2605.25707v1 Announce Type: new Abstract: Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digita

AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora

arXiv:2605.25382v1 Announce Type: new Abstract: Evidence construction systems--chunk retrieval, agent memory, knowledge-graph traversal, and thematic indexing--are evaluated on separate benchmarks wit

Local Ai

AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimization

arXiv:2605.25658v1 Announce Type: cross Abstract: Expensive optimization tasks are ubiquitous in real-world applications, demanding highly specialized solvers. While LLM-driven automated solver genera

local-aiarxiv-cs-ai

Bayesian Distributional Models of Executive Functioning

arXiv:2510.00387v3 Announce Type: replace Abstract: This study uses controlled simulations with known ground-truth parameters to evaluate how Distributional Latent Variable Models (DLVM) and Bayesian

Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

arXiv:2605.24423v1 Announce Type: new Abstract: In-Context Reinforcement Learning (ICRL) has enabled foundation agents to adapt instantaneously to novel tasks, yet its efficacy in Ad-Hoc Teamwork (AHT

Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows

arXiv:2605.24219v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed as autonomous agents that reason, use tools, and act over multiple steps. Yet most hallucination

DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations

arXiv:2512.19097v3 Announce Type: replace-cross Abstract: Intracranial EEG (iEEG) provides direct, millisecond-scale recordings of human neural activity, but reusable representation learning is diffic

Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation

arXiv:2605.25781v1 Announce Type: new Abstract: Evaluating structured-information extraction from historical documents at scale requires high-precision ground-truth annotations, yet traditional manual

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

arXiv:2605.25773v1 Announce Type: cross Abstract: Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a

Explore Before You Solve: The Speed--Depth Trade-off in Epistemic Agents for ARC-AGI-3

arXiv:2605.25931v1 Announce Type: new Abstract: We systematically investigate all 25 public ARC-AGI-3 games and find that every one is reachable through non-intelligent strategies: 10 in a single blin

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

arXiv:2605.23970v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such

safetyarxiv-cs-cl

Research

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers

arXiv:2603.05143v3 Announce Type: replace Abstract: Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoni

researcharxiv-cs-cl

Fine-Tuning Language Models to Know What They Know

arXiv:2602.02605v2 Announce Type: replace-cross Abstract: Evaluating true metacognition in Large Language Models (LLMs) is difficult due to biases and heuristics. This paper presents a framework to me

FLOATBench: A Dataset and Benchmark for Floating Offshore Wind Turbine Tower Fatigue

arXiv:2605.25717v1 Announce Type: new Abstract: Most of the world's offshore wind resource lies in waters too deep for fixed-bottom foundations, making floating offshore wind turbines (FOWTs) essentia

FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

arXiv:2507.07644v4 Announce Type: replace Abstract: We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in struct

From Knowledge to Inference: Formalizing Specialized Public Health Reasoning on GlobalHealthAtlas

arXiv:2602.00491v2 Announce Type: replace Abstract: Public health reasoning requires population level inference grounded in scientific evidence, expert consensus, and safety constraints. However, it r

safetyarxiv-cs-cl

How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

arXiv:2605.24660v1 Announce Type: cross Abstract: Before an LLM agent can use a tool, a retrieval system must decide which candidate tools to show to the agent. How long should that shortlist be? Show

Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications

arXiv:2605.24883v1 Announce Type: new Abstract: The widespread integration of Large Language Models (LLMs) necessitates rigorous and systematic safety evaluation. Existing paradigms either rely on con

Research

Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

arXiv:2603.12983v3 Announce Type: replace-cross Abstract: Error Span Detection (ESD) is a crucial subtask in Machine Translation (MT) evaluation, aiming to identify the location and severity of transl

researcharxiv-cs-ai

Research

Length Generalization with Log-Depth Recurrent Units

arXiv:2605.26035v1 Announce Type: new Abstract: Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are

researcharxiv-cs-lg

Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence

arXiv:2605.23952v1 Announce Type: new Abstract: Artificial agents now generate behavior rich enough to invite trust, surprise, and concern, yet our evaluation tools still privilege capability scores o

Measuring the Depth of LLM Unlearning via Activation Patching

arXiv:2605.24614v1 Announce Type: cross Abstract: Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether target kn

Meta-Engineering Harnesses for AI-Native Software Production: A Contract-Driven Adversarial Verification Architecture with Early Deployment Report

arXiv:2605.25665v1 Announce Type: cross Abstract: AI-native software development is often evaluated at the level of individual models, prompts, or generated artifacts. This framing is insufficient for

agentsarxiv-cs-ai

Path Following Control System of Line-of-Sight Guidance for Robotic Dolphin with Multi-Link Mechanism in Underwater Simulator

arXiv:2605.25401v1 Announce Type: new Abstract: Biomimetic autonomous underwater vehicle (BAUV) with multi-link mechanism is widely used in aquatic life observation and environmental surveys due to it

agentsarxiv-cs-ro