Search: “evaluation”

RedDebate: Safer Responses Through Multi-Agent Red Teaming Debates

arXiv:2506.11083v3 Announce Type: replace Abstract: We introduce RedDebate, a novel multi-agent debate framework that provides the foundation for Large Language Models (LLMs) to identify and mitigate

safetyarxiv-cs-cl

RenoBench: A Citation Parsing Benchmark

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

DGX agent

arXiv:2603.25640v2 Announce Type: replace-cross Abstract: Accurate parsing of citations is necessary for machine-readable scholarly infrastructure. But, despite sustained interest in this problem, exi

model-releasesarxiv-cs-cl

2 Jun 2026

RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenes

arXiv:2606.00828v1 Announce Type: new Abstract: Vision-Language Models (VLMs) have shown strong visual understanding and are increasingly deployed in embodied AI systems, where reliable perception und

Spectral Audit of In-Context Operator Networks

arXiv:2606.02427v1 Announce Type: cross Abstract: Existing evaluations of neural operators and in-context operator learning rely primarily on prediction error, but accurate output prediction does not

local-aiarxiv-cs-lg

Tutorials

Strong Stochastic Flow Maps

arXiv:2606.01086v1 Announce Type: cross Abstract: Flow and diffusion models generate high-quality samples in many modalities; however, many network evaluations are required during inference due to num

tutorialsarxiv-cs-ai

The Assistant as a Privileged Persona: A canonical reference in cross-persona self-recognition

model-releasesarxiv-cs-lg

arXiv:2606.00545v1 Announce Type: new Abstract: Post-trained language models can recognize their own outputs from a sentence or two out of context. In a companion paper itep{jack2026twomodes} we showe

TLG: Temporal-Logic Grounding for Video Question Answering via Source-Annotation Reconstruction and Category-Targeted Reasoning

arXiv:2606.01591v1 Announce Type: new Abstract: The TimeLogic Challenge evaluates formal temporal-logic reasoning over video - 16 operators (before, after, until, since, always, co-occur, ordering, ..

Tracking the Behavioral Trajectories of Adapting Agents

arXiv:2606.02536v1 Announce Type: new Abstract: Text files such as skill files, memory files, and behavioral configuration files play a central role in defining how modern agents act. Through edits by

TVIR: Building Deep Research Agents Towards Text--Visual Interleaved Report Generation

arXiv:2606.02320v1 Announce Type: new Abstract: Deep Research Agents have shown strong capability in multi-step information retrieval, reasoning, and long-form report generation, but existing benchmar

Two-Fidelity Best-Action Identification for Stochastic Minimax Tree

arXiv:2606.01708v1 Announce Type: cross Abstract: We study fixed-confidence best-action identification (BAI) in stochastic minimax trees. This problem is increasingly relevant in modern AI planning, w

local-aiarxiv-cs-ai

What to Format and How: A Benchmark and Workflow Approach for Document Formatting

arXiv:2606.01936v1 Announce Type: new Abstract: Recent advances in large language models (LLMs) have opened up new possibilities for automated document formatting. However, real-world formatting often

Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories

arXiv:2606.02060v1 Announce Type: new Abstract: Deep-research agents solve tasks through long trajectories of search, tool use, evidence inspection, and answer synthesis. Evaluation based on final ans

3DAE: Binaural Quality Assessment for Audio Novel View Synthesis with Spatial Maps and Benchmark

arXiv:2605.30469v1 Announce Type: cross Abstract: 3D audio and novel-view acoustic synthesis models are usually evaluated with global metrics.However, global metrics often hide where and why binaural

Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery

arXiv:2502.15224v2 Announce Type: replace-cross Abstract: Interactive discovery requires agents to maintain and update structured beliefs over many rounds of feedback. Before evaluating agents in nois

BIAS-ID: A Framework for Analyzing Transformation Biases in AI-Generated Image Detectors

arXiv:2605.31153v1 Announce Type: new Abstract: Given the surge of harmful AI-generated imagery online, reliably distinguishing authentic images from generated ones has become an urgent research topic

safetyarxiv-cs-cv

Choosing the Lens: Strategic Perspective Activation in Context-Dependent Argumentation

arXiv:2605.31581v1 Announce Type: new Abstract: The same arguments often need to be evaluated under different external regimes. An agent with influence over the regime has a strategic lever that stand

Conditional Coverage Diagnostics for Conformal Prediction

arXiv:2512.11779v2 Announce Type: replace-cross Abstract: Evaluating conditional coverage remains one of the most persistent challenges in assessing the reliability of predictive systems. Although con

Hardware

Deterministic Inference across Tensor Parallel Sizes That Eliminates Training-Inference Mismatch

arXiv:2511.17826v2 Announce Type: replace-cross Abstract: Deterministic inference is increasingly critical for large language model (LLM) applications such as LLM-as-a-judge evaluation, multi-agent sy

hardwarearxiv-cs-cl

Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

arXiv:2602.00521v2 Announce Type: replace Abstract: While LLM-as-a-Judge is widely used in automated evaluation, existing validation practices primarily operate at the level of observed outputs, offer

safetyarxiv-cs-ai

DTBench: A Synthetic Benchmark for Document-to-Table Extraction

arXiv:2602.13812v3 Announce Type: replace-cross Abstract: Document-to-table (Doc2Table) extraction derives structured tables from unstructured documents under a target schema, enabling reliable and ve

DynaTree: Dynamic Agentic Retrieval Tree for Time-Sensitive News Retrieval

arXiv:2605.31377v1 Announce Type: cross Abstract: Agentic Retrieval-Augmented Generation improves retrieval by integrating planning, tool use, and iterative reasoning, but existing agentic RAG methods

EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs

arXiv:2605.30637v1 Announce Type: new Abstract: Clinical decision-making (CDM) is central to real-world clinical workflows, where clinicians infer diagnoses, select treatments, or anticipate future he

Empirical Characterization of Inference-Time Elicited Probability Transformations in Large Language Models

arXiv:2603.19262v2 Announce Type: replace-cross Abstract: Large language models increasingly rely on inference-time procedures such as chain-of-thought reasoning, self-refinement, retrieval augmentati

researcharxiv-cs-ai

ERGeoBench:A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models

arXiv:2605.31251v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) have shown strong potential as embodied agents, yet embodied geo-localization remains underexplored due to th

EvoGens: A Population-Based Heuristic Search Framework for Scientific Idea Generation

arXiv:2605.30961v1 Announce Type: new Abstract: Generating novel research ideas is fundamental to scientific progress. While Large Language Models (LLMs) show promise in assisting this process, existi

researcharxiv-cs-cl

Forgetting Has Neighbors: Localized Collateral Forgetting in Machine Unlearning

arXiv:2605.31317v1 Announce Type: new Abstract: Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning q

local-aiarxiv-cs-lg

Learning Randomized Reductions

model-releasesarxiv-cs-lg

arXiv:2412.18134v4 Announce Type: replace Abstract: Randomized self-reductions (RSRs) express f(x) using f evaluated at random correlated points, enabling self-correcting programs, instance-hiding pro

LegSegNet: A Public Deep Learning System for Lower Extremity CT Tissue Segmentation and Quantification

arXiv:2605.30829v1 Announce Type: new Abstract: Lower extremity computed tomography (CT) contains clinically relevant information for body composition analysis, sarcopenia assessment, and musculoskele

MultiPriv: Benchmarking Individual-Level Privacy Reasoning in Vision-Language Models

arXiv:2511.16940v3 Announce Type: replace Abstract: Modern Vision-Language Models (VLMs) pose significant individual-level privacy risks by linking fragmented multimodal data to identifiable individua

Applications

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

arXiv:2605.30393v1 Announce Type: cross Abstract: Public numeric benchmarks appear in pretraining, so an evaluation that conditions on a date may be measuring memorized recall rather than out-of-sampl

applicationsarxiv-cs-ai

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

arXiv:2605.31572v1 Announce Type: new Abstract: Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations

Pairwise Reference Alignment as a Model-Level Ordinal Observable

arXiv:2605.30758v1 Announce Type: new Abstract: Pairwise preference data is widely used in language-model evaluation and alignment, often for model ranking, reward modeling, or preference optimization

PReMISE: Policy Rubrics as Measurement Specifications for LLM Judges

arXiv:2605.30803v1 Announce Type: new Abstract: LLM judges are increasingly used to evaluate open-ended responses, but their scores depend strongly on the rubrics that condition them. A vague rubric a

safetyarxiv-cs-ai

Tutorials

Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

arXiv:2605.30557v1 Announce Type: cross Abstract: Spatial reasoning is a fundamental capability for vision-language models (VLMs) deployed in real-world environments. However, visual observations are

tutorialsarxiv-cs-ai

SimulCost: A Cost-Aware Benchmark and Toolkit for Automating Physics Simulations with LLMs

arXiv:2603.20253v2 Announce Type: replace-cross Abstract: Evaluating LLM agents for scientific tasks has focused on token costs while ignoring tool-use costs like simulation time and experimental reso

SurGe: Improved Surface Geometry in Point Maps

arXiv:2605.31577v1 Announce Type: new Abstract: Recent feedforward 3D reconstruction methods predict point maps and estimate global 3D geometry remarkably well. However, their predictions still exhibi

local-aiarxiv-cs-cv

SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence

arXiv:2605.31529v1 Announce Type: new Abstract: True video intelligence demands more than recognizing what is visible: it requires reasoning about why events unfold, predicting what would change under

Target-Side Paraphrase Augmentation for Sign Language Translation with Large Language Models

arXiv:2605.31393v1 Announce Type: cross Abstract: Sign language translation (SLT) remains constrained by limited paired sign-video/text corpora and heavy-tailed target vocabularies. We study target-si

researcharxiv-cs-ai

Triaging Threats to Specialized Guardrails

arXiv:2605.30693v1 Announce Type: cross Abstract: Building robust safety guardrails is essential for deploying Large Language Models across diverse real-world applications. However, this goal remains

Wall-Clock Complexity for Zeroth-Order Optimization with Tunable Oracle Fidelity

arXiv:2605.31346v1 Announce Type: cross Abstract: Zeroth-order (black-box) optimization is applied when gradients are unavailable and objective evaluations rely on expensive simulations. In many such

researcharxiv-cs-lg

World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry

arXiv:2604.01985v2 Announce Type: replace-cross Abstract: General-purpose world models promise scalable policy evaluation, optimization, and planning, yet achieving the required level of robustness re

safetyarxiv-cs-ai

Benchmarking at the Edge of Comprehension

arXiv:2602.14307v3 Announce Type: replace Abstract: As frontier Large Language Models (LLMs) increasingly saturate new benchmarks shortly after they are published, benchmarking itself is at a juncture

local-aiarxiv-cs-ai

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

arXiv:2605.30188v1 Announce Type: cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibr

DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

arXiv:2605.29339v1 Announce Type: new Abstract: With the rapid advancement of multimodal large language models (MLLMs), models have demonstrated increasingly powerful multimodal capabilities. However,

Does The Way You Plan Matter? An Empirical Study of Planning Representations for LLM Web Agents

arXiv:2605.29927v1 Announce Type: cross Abstract: Despite recent advances, LLM-based web agents still struggle with limited exploration, omission of critical steps, and sensitivity to task constraints

E-valuator: Reliable Agent Verifiers with Sequential Hypothesis Testing

arXiv:2512.03109v2 Announce Type: replace-cross Abstract: Agentic AI systems execute a sequence of actions, such as reasoning steps or tool calls, in response to a user prompt. To evaluate the success

Embodied3DBench: Benchmarking Low-Level Embodied Spatial Intelligence of Vision Language Models

arXiv:2605.29074v1 Announce Type: new Abstract: Are current Vision Language Models (VLMs) ready to comprehend and reason about complex embodied interactions in 3D environments? We introduce Embodied3D

EVADE: LLM-Based Explanation Generation and Validation for Error Detection in NLI

arXiv:2511.08949v2 Announce Type: replace Abstract: High-quality datasets are critical for training and evaluating reliable NLP models. In tasks like natural language inference (NLI), human label vari

researcharxiv-cs-cl