Search: “evaluation”

MapTab: Are MLLMs Ready for Multi-Criteria Route Planning in Heterogeneous Graphs?

arXiv:2602.18600v3 Announce Type: replace Abstract: Systematic evaluation of Multimodal Large Language Models (MLLMs) is crucial for advancing Artificial General Intelligence (AGI). However, existing

23 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Provable Joint Decontamination for Benchmarking Multiple Large Language Models

DGX agent

arXiv:2605.21543v1 Announce Type: new Abstract: Benchmark data contamination has become a central challenge in LLM evaluation: when evaluation examples appear in the training data of one or more audit

model-releasesarxiv-cs-lg

23 May 2026

Safety

Harder to Defend: Towards Chinese Toxicity Attacks via Implicit Enhancement and Obfuscation Rewriting

arXiv:2605.22258v1 Announce Type: new Abstract: Large language models (LLMs) require robust toxicity evaluation beyond explicit wording. This setting remains underexplored in Chinese, where toxicity m

safetyarxiv-cs-cl

22 May 2026

VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction

arXiv:2602.13294v3 Announce Type: replace Abstract: Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks r

22 May 2026

EngiAI: A Multi-Agent Framework and Benchmark Suite for LLM-Driven Engineering Design

arXiv:2605.19743v1 Announce Type: new Abstract: Large Language Model (LLM) agents are increasingly applied to engineering design tasks, yet existing evaluation frameworks do not adequately address mul

MetaRA: Metamorphic Robustness Assessment for Multimodal Large Language Model-based Visual Question Answering Systems

arXiv:2605.19307v1 Announce Type: new Abstract: Visual Question Answering (VQA), as the representative multimodal task, serves as a key benchmark for evaluating the reasoning capabilities of Multimoda

Quantifying the Generalization Gap in Seizure Detection: A Large-Scale Empirical Benchmark via the SzCORE Challenge

arXiv:2505.18191v2 Announce Type: replace-cross Abstract: Reliable automatic seizure detection from long-term electroencephalography (EEG) remains an unsolved challenge, as current models often fail t

RoboJailBench: Benchmarking Adversarial Attacks and Defenses in Embodied Robotic Agents

model-releasesarxiv-cs-ro

arXiv:2605.19328v1 Announce Type: cross Abstract: Recent advances in Vision-Language Models (VLMs) facilitate a new class of embodied AI systems, where these models are integrated into physical platfo

Safety

AI Alignment Breaks at the Edge

arXiv:2602.20042v2 Announce Type: replace Abstract: General Alignment has improved average-case helpfulness and safety, but current alignment practice still rewards confident, single-turn responses. T

safetyarxiv-cs-cl

Safety

Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents

arXiv:2605.17830v1 Announce Type: new Abstract: Safety evaluations of memory-equipped LLM agents typically measure within-task safety: whether an agent completes a single scenario safely, often under

safetyarxiv-cs-ai

SCICONVBENCH: Benchmarking LLMs on Multi-Turn Clarification for Task Formulation in Computational Science

arXiv:2605.18630v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed as scientific AI as- sistants, and a growing body of benchmarks evaluates their capabilities acro

Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents

arXiv:2605.16282v1 Announce Type: cross Abstract: The rapid deployment of LLM-based autonomous agents has introduced safety risks that extend far beyond traditional LLM concerns, prompting a prolifera

TOBench: A Task-Oriented Omni-Modal Benchmark for Real-World Tool-Using Agents

arXiv:2605.16909v1 Announce Type: new Abstract: Tool-using agents are increasingly expected to operate across realistic professional workflows, where they must interpret multimodal inputs, coordinate

ToolMATH: A Diagnostic Benchmark for Long-Horizon Tool Use under Systematic Tool-Catalog Constraints

arXiv:2602.21265v2 Announce Type: replace Abstract: We introduce ToolMATH, a math-grounded diagnostic benchmark for evaluating long-horizon tool use under controllable tool-catalog conditions. ToolMAT

WorldArena 2.0: Extending Embodied World Model Benchmarking on Modality, Functionality and Platform

arXiv:2605.17912v1 Announce Type: cross Abstract: World models have emerged as a central paradigm for embodied intelligence, enabling agents to predict action-conditioned future and reason about envir

FINESSE-Bench: A Hierarchical Benchmark Suite for Financial Domain Knowledge and Technical Analysis in Large Language Models

arXiv:2605.15482v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly being applied to financial analysis, reporting, investment decision support, risk management, compliance,

Minerva-Ego: Spatiotemporal Hints for Egocentric Video Understanding

arXiv:2605.15342v1 Announce Type: new Abstract: Video reasoning models are a core component of egocentric and embodied agents. However, standard benchmarks for assessing models provide only evaluation

Agents

TeamTR: Trust-Region Fine-Tuning for Multi-Agent LLM Coordination

arXiv:2605.15207v1 Announce Type: new Abstract: Multi-agent LLM systems have shown promise for complex reasoning, yet recent evaluations reveal they often underperform single-model baselines. We ident

agentsarxiv-cs-lg

VideoVerse: Does Your T2V Generator Have World Model Capability to Synthesize Videos?

arXiv:2510.08398v4 Announce Type: replace Abstract: The recent rapid advancement of Text-to-Video (T2V) generation technologies are engaging the trained models with more world model ability, making th

Research

AI Outperforms Humans in Personalized Image Aesthetics Assessment via LLM-Based Interviews and Semantic Feature Extraction

arXiv:2605.14761v1 Announce Type: new Abstract: Accurately predicting individual aesthetic evaluation for images is a fundamental challenge for AI. Various deep learning (DL)-based models have been pr

researcharxiv-cs-ai

Cattle Trade: A Multi-Agent Benchmark for LLM Bluffing, Bidding, and Bargaining

arXiv:2605.14537v1 Announce Type: new Abstract: We introduce extsc{Cattle Trade, a multi-agent benchmark for evaluating large language models (LLMs) as agents in strategic reasoning under imperfect in

ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents

arXiv:2605.14133v1 Announce Type: new Abstract: Interactive agent benchmarks face a tension between scalable construction and realistic workflow evaluation. Hand-authored tasks are expensive to extend

Agents

In-IDE Toolkit for Developers of AI-Based Features

arXiv:2605.14612v1 Announce Type: cross Abstract: AI-enabled features built on LLMs and agentic workflows are difficult to test, debug, and reproduce, especially for product-focused software engineers

agentsarxiv-cs-ai

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence

arXiv:2605.12882v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) have significantly advanced document understanding, yet current Doc-VQA evaluations score only the final answ

Large Language Models Lack Temporal Awareness of Medical Knowledge

arXiv:2605.13045v1 Announce Type: new Abstract: The existing methods for evaluating the medical knowledge of Large Language Models (LLMs) are largely based on atemporal examination-style benchmarks, w

SpurAudio: A Benchmark for Studying Shortcut Learning in Few-Shot Audio Classification

arXiv:2605.13672v1 Announce Type: new Abstract: Few-shot classification (FSC) is widely used for learning from limited labeled data, yet most evaluations implicitly assume that target concepts are ind

Agents

When Should an AI Workflow Release? Always-Valid Inference for Black-Box Generate-Verify Systems

arXiv:2605.12947v1 Announce Type: cross Abstract: LLM-enabled AI workflows increasingly produce outputs through iterative generate-evaluate-revise loops. Each iteration can improve the candidate, but

agentsarxiv-cs-ai

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

arXiv:2605.11663v1 Announce Type: new Abstract: Authentic school examinations provide a high-validity test bed for evaluating multimodal large language models (MLLMs), yet benchmarks grounded in Japan

13 May 2026

Research

Unpacking the Eye of the Beholder: Social Location, Identity, and the Moving Target of Political Perspectives

arXiv:2605.11166v1 Announce Type: new Abstract: Political and social identities structure how people evaluate political information, a finding decades deep in political science and routinely discarded

researcharxiv-cs-cv

13 May 2026

Benchmarking Sensor-Fault Robustness in Forecasting

arXiv:2605.10822v1 Announce Type: new Abstract: Cyber-physical system (CPS) forecasting models depend on sensor streams with noisy, biased, missing, or temporally misaligned readings, yet standard for

Beyond Self-Play and Scale: A Behavior Benchmark for Generalization in Autonomous Driving

model-releasesarxiv-cs-ro

arXiv:2605.10034v1 Announce Type: new Abstract: Recent Autonomous Driving (AD) works such as GigaFlow and PufferDrive have unlocked Reinforcement Learning (RL) at scale as a training strategy for driv

Research

Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges

arXiv:2605.09702v1 Announce Type: cross Abstract: Multi-judge evaluation is increasingly used to assess LLMs and reward models, and the prevailing heuristic is to curate: keep the most accurate judges

researcharxiv-cs-cl

Computer Use at the Edge of the Statistical Precipice

arXiv:2605.08261v1 Announce Type: cross Abstract: Evaluating Computer Use Agents (CUAs) on interactive environments is fraught with methodological pitfalls that the field has yet to systematically add

Decomposing and Steering Functional Metacognition in Large Language Models

arXiv:2605.08942v1 Announce Type: new Abstract: Large language models (LLMs) increasingly exhibit behaviors suggesting awareness of their evaluation context, often adapting their reasoning strategies

Tutorials

Interactive Benchmarks

arXiv:2603.04737v2 Announce Type: replace Abstract: Existing reasoning evaluation paradigms suffer from different limitations: fixed benchmarks are increasingly saturated and vulnerable to contaminati

tutorialsarxiv-cs-ai

Towards Cross-lingual Values Judgment: A Consensus-Pluralism Perspective

arXiv:2602.17283v2 Announce Type: replace-cross Abstract: As large language models (LLMs) are employed worldwide, existing evaluation paradigms for their multilingual capabilities primarily focus on f

Benchmarking World-Model Learning with Environment-Level Queries

arXiv:2510.19788v4 Announce Type: replace Abstract: World models are central to building AI agents capable of flexible reasoning and planning. Yet current evaluations (i) test only properties measurab

FAME: Forecasting Academic Impact via Continuous-Time Manifold Evolution

arXiv:2605.07208v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used to brainstorm and evaluate research ideas, yet assessing such judgments is fundamentally difficult be

How Far Are VLMs from Privacy Awareness in the Physical World? An Empirical Study

arXiv:2605.05340v2 Announce Type: replace-cross Abstract: As Vision-Language Models (VLMs) are increasingly deployed as autonomous cognitive cores for embodied assistants, evaluating their privacy awa

NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models

arXiv:2605.07051v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown good performance on various science educational benchmarks, demonstrating their potential for use in science and

EvoLM: Self-Evolving Language Models through Co-Evolved Discriminative Rubrics

arXiv:2605.03871v1 Announce Type: new Abstract: Language models encode substantial evaluative knowledge from pretraining, yet current post-training methods rely on external supervision (human annotati

7 May 2026

Detecting Stealth Sycophancy in Mental-Health Dialogue with Dynamic Emotional Signature Graphs

arXiv:2605.03472v1 Announce Type: new Abstract: As conversational AI therapists are increasingly used in psychological support settings, reliable offline evaluation of therapeutic response quality rem

Simulated Students in Tutoring Dialogues: Substance or Illusion?

arXiv:2601.04025v2 Announce Type: replace Abstract: Advances in large language models (LLMs) enable many new innovations in education. However, evaluating the effectiveness of new technology requires

Soft Tournament Equilibrium

arXiv:2604.04328v3 Announce Type: replace-cross Abstract: The evaluation of general-purpose artificial agents, particularly those based on LLMs, presents a significant challenge due to the non-transit

Tutorials

ValueBlindBench: Agreement-Gated Stress Testing of LLM-Judged Investment Rationales Before Returns Are Observable

arXiv:2604.25224v2 Announce Type: replace Abstract: LLM-based financial agents increasingly produce investment rationales before the outcomes needed to evaluate them are observable. This creates a del

tutorialsarxiv-cs-ai

Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies

arXiv:2605.03596v1 Announce Type: cross Abstract: Workspace learning requires AI agents to identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a

Applications

CLEAR: Revealing How Noise and Ambiguity Degrade Reliability in LLMs for Medicine

arXiv:2605.01011v1 Announce Type: new Abstract: Medical large language model (LLM) evaluations rely on simplified, exam-style benchmarks that rarely reflect the ambiguity of real-world medical inquiri

applicationsarxiv-cs-cl

5 May 2026

Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong

arXiv:2501.09775v3 Announce Type: replace Abstract: Multiple Choice Question (MCQ) tests are among the most used methods for evaluating large language models (LLMs). Besides checking the correctness o