Search: “evaluation”

MapAnything: Evaluating Monocular Metric Depth Models for 3D Urban Asset Localization

arXiv:2509.14839v2 Announce Type: replace Abstract: City administrations increasingly rely on comprehensive databases and urban digital twins of city assets, such as traffic signs and trees, as well a

researcharxiv-cs-cv

20 May 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Industry

Multimodal evaluators: MLLM-as-a-judge for image-to-text tasks in Strands Evals

DGX agent

If you’re building visual shopping, image or document understanding, or chart analysis, you need a way to verify whether your model’s response is actually grounded in the source image. A text-only eva

industryaws-ml-blog

20 May 2026

ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

arXiv:2603.06740v2 Announce Type: replace-cross Abstract: Protein language models (pLMs) have shown strong potential for zero-shot prediction of missense variant effects, yet systematic benchmarking o

20 May 2026

Evaluating AI Alignment in LLMs: Output Analysis of Value Priorities Across 75 Models with Human Benchmarking

arXiv:2506.12617v4 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used in human-AI interaction research and practice, yet existing capability and safety benchmarks reve

safetyarxiv-cs-ai

Evaluating Cognitive Age Alignment in Interactive AI Agents

arXiv:2605.17894v1 Announce Type: new Abstract: While agentic AI and its core multimodal large language models (MLLMs) have demonstrated remarkable promise in language and visual reasoning across doma

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

arXiv:2605.17554v1 Announce Type: new Abstract: Frontier deep research agents (DRAs) plan a research task, synthesize across documents, and return a structured deliverable on demand. They are being de

From Pixels to Places: A Systematic Benchmark for Evaluating Image Geolocalization Ability in Large Language Models

arXiv:2508.01608v2 Announce Type: replace Abstract: Image geolocalization, the task of identifying the geographic location depicted in an image, is important for applications in crisis response, digit

Hunt Instead of Wait: Evaluating Deep Data Research on Large Language Models

arXiv:2602.02039v2 Announce Type: replace Abstract: The agency expected of Agentic Large Language Models goes beyond answering correctly, requiring autonomy to set goals and decide what to explore. We

The Range Shrinks, the Threat Remains: Re-evaluating LLM Package Hallucinations on the 2026 Frontier-Model Cohort

model-releasesarxiv-cs-lg

arXiv:2605.17062v1 Announce Type: cross Abstract: Spracklen et al. (USENIX Security '25) showed that code-generating large language models hallucinate package names that do not exist on PyPI or npm at

Trust No Tool: Evaluating and Defending LLM Agents under Untrusted Tool Feedback

arXiv:2605.17453v1 Announce Type: cross Abstract: Tool-using LLM agents increasingly rely on external tools to make consequential decisions, yet most existing agent-security benchmarks and defenses im

Evaluating Chinese Ambiguity Understanding in Large Language Models

arXiv:2605.15635v1 Announce Type: new Abstract: Linguistic ambiguity is critical to the robustness of Large Language Models (LLMs), yet existing research focuses mostly on English, with limited attent

18 May 2026

Large Language Models Could Be Rote Learners

arXiv:2504.08300v5 Announce Type: replace-cross Abstract: Benchmark-based evaluation, e.g., multiple-choice questions (MCQs) and open-ended questions (OEQs), is widely used for evaluating Large Langua

18 May 2026

LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

arXiv:2605.15393v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustl

researcharxiv-cs-lg

18 May 2026

Editor's Choice: Evaluating Abstract Intent in Image Editing through Atomic Entity Analysis

arXiv:2605.14842v1 Announce Type: new Abstract: Humans naturally communicate through abstract concepts like 'mood'. However, current image editing benchmarks focus primarily on explicit, literal comma

15 May 2026

FutureSim: Replaying World Events to Evaluate Adaptive Agents

arXiv:2605.15188v1 Announce Type: cross Abstract: AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently m

15 May 2026

KamonBench: A Grammar-Based Dataset for Evaluating Compositional Factor Recovery in Vision-Language Models

arXiv:2605.13322v1 Announce Type: new Abstract: Kamon (family crests) are an important part of Japanese culture and a natural test case for compositional visual recognition: each crest combines a smal

14 May 2026

LLMs as annotators of credibility assessment in Danish asylum decisions: evaluating classification performance and errors beyond aggregated metrics

arXiv:2605.13412v1 Announce Type: cross Abstract: Off-the-shelf large language models (LLMs) are increasingly used to automate text annotation, yet their effectiveness remains underexplored for underr

14 May 2026

Tools

increasing levels of autonomy: /skill: preset prompts /plan: human-refined inputs /goal: AI-evaluated outputs

This post outlines a framework for progressively automating AI workflows through three levels of increasing autonomy: using preset prompts for consistency, refining inputs with human judgment to impro

toolsswyx--x

13 May 2026

Agent-ValueBench: A Comprehensive Benchmark for Evaluating Agent Values

arXiv:2605.10365v1 Announce Type: new Abstract: Autonomous agents have rapidly matured as task executors and seen widespread deployment via harnesses such as OpenClaw. Safety concerns have rightly dra

Beyond Multiple Choice: Evaluating Steering Vectors for Summarization

arXiv:2505.24859v3 Announce Type: replace-cross Abstract: Steering vectors are a lightweight method for controlling text properties by adding a learned bias to language model activations at inference

safetyarxiv-cs-cl

BiAxisAudit: A Novel Framework to Evaluate LLM Bias Across Prompt Sensitivity and Response-Layer Divergence

arXiv:2605.09041v1 Announce Type: new Abstract: Bias audits of large language models now operate within governance frameworks such as the EU AI Act, making benchmark reliability a security concern in

Can Language Models Analyze Data? Evaluating Large Language Models for Question Answering over Datasets

arXiv:2605.10419v1 Announce Type: cross Abstract: This paper investigates the effectiveness of large language models (LLMs) in answering questions over datasets. We examine their performance in two sc

Agents

Consistency as a Testable Property: Statistical Methods to Evaluate AI Agent Reliability

arXiv:2605.10516v1 Announce Type: new Abstract: This paper establishes a rigorous measurement science for AI agent reliability, providing a foundational framework for quantifying consistency under sem

agentsarxiv-cs-ai

Evaluating Developmental Cognition Capabilities of LLMs

arXiv:2605.08549v1 Announce Type: new Abstract: Conversational AI is increasingly personalized around users' preferences, histories, goals, and knowledge, but much less around how users interpret and

Matching Meaning at Scale: Evaluating Semantic Search for 18th-Century Intellectual History through the Case of Locke

arXiv:2605.09236v1 Announce Type: cross Abstract: While digitized corpora have transformed the study of intellectual transmission, current methods rely heavily on lexical text reuse detection, capturi

Med-StepBench: A Hierarchical Reasoning Framework for Evaluating Hallucinations in Medical Vision-Language Models

arXiv:2605.10002v1 Announce Type: new Abstract: Large vision-language models (VLMs) demonstrate strong performance in medical image understanding, but frequently generate clinically plausible yet inco

Neuroprobe: Evaluating Intracranial Brain Responses to Naturalistic Stimuli

arXiv:2509.21671v2 Announce Type: replace Abstract: High-resolution neural datasets enable foundation models for the next generation of brain-computer interfaces and neurological treatments. The commu

researcharxiv-cs-lg

OPT-BENCH: Evaluating the Iterative Self-Optimization of LLM Agents in Large-Scale Search Spaces

arXiv:2605.08904v1 Announce Type: new Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning and tool use. However, the fundamental cognitive faculties essential

The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space

arXiv:2605.09883v1 Announce Type: cross Abstract: As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores gen

EnvSimBench: A Benchmark for Evaluating and Improving LLM-Based Environment Simulation

arXiv:2605.07247v1 Announce Type: new Abstract: Scalable AI agents training relies on interactive environments that faithfully simulate the consequences of agent actions. Manually crafted environments

11 May 2026

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

arXiv:2605.07073v1 Announce Type: new Abstract: Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls.

11 May 2026

KernelBench-X: A Comprehensive Benchmark for Evaluating LLM-Generated GPU Kernels

model-releasesarxiv-cs-lg

arXiv:2605.04956v1 Announce Type: new Abstract: LLM-based Triton kernel generation has attracted significant interest, yet a fundamental empirical question remains unanswered: where does this capabili

NoisyCausal: A Benchmark for Evaluating Causal Reasoning Under Structured Noise

arXiv:2605.04313v1 Announce Type: new Abstract: Causal reasoning in natural language requires identifying relevant variables, understanding their interactions, and reasoning about effects and interven

ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval

arXiv:2605.03361v2 Announce Type: new Abstract: As multimodal content continues to expand at a rapid pace, audio retrieval has emerged as a key enabling technology for media search, content organizati

StoryAlign: Evaluating and Training Reward Models for Story Generation

arXiv:2605.04831v1 Announce Type: new Abstract: Story generation aims to automatically produce coherent, structured, and engaging narratives. Although large language models (LLMs) have significantly a

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

arXiv:2605.01986v1 Announce Type: new Abstract: What if the twelve jurors of Sidney Lumet's 12 Angry Men (1957) were not men, but large language models? Would the one juror who disagrees still be able

Local Ai

BIT.UA-AAUBS at ArchEHR-QA 2026: Evaluating Open-Source and Proprietary LLMs via Prompting in Low-Resource QA

arXiv:2605.03618v1 Announce Type: new Abstract: This paper presents the joint participation of the BIT.UA and AAUBS groups in the ArchEHR-QA 2026 shared task, which focuses on clinical question answer

local-aiarxiv-cs-cl

Evaluating Generative Models as Interactive Emergent Representations of Human-Like Collaborative Behavior

arXiv:2605.03855v1 Announce Type: new Abstract: Human-AI collaboration requires AI agents to understand human behavior for effective coordination. While advances in foundation models show promising ca

researcharxiv-cs-ro

Evaluating Reasoning Models for Queries with Presuppositions

arXiv:2605.03050v1 Announce Type: new Abstract: Millions of users turn to AI models for their information needs. It is conceivable that a large number of user queries contain assumptions that may be f

researcharxiv-cs-cl

Repurposing and Evaluating the (In)Feasibility of Dataset Poisoning enabled Watermarking for Contrastive Learning

arXiv:2605.01834v1 Announce Type: cross Abstract: Contrastive learning (CL) reduces annotation cost via auto-derived supervisory signals. Since large-scale in-house CL datasets are infeasible, relianc

Creating and Evaluating Figurative Language Dataset for Sindhi

arXiv:2605.01323v1 Announce Type: new Abstract: In this article, we introduce SiNFluD, a novel benchmark dataset for Sindhi figurative language classification. We first collect raw text from various b

Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages

arXiv:2605.02608v1 Announce Type: new Abstract: Transformer-based models achieve state-of-the-art dependency parsing for high-resource languages, yet their advantage over simpler architectures in low-

researcharxiv-cs-cl

From Concept to Capability: Evaluating 3D Gaussian Splatting for Synthetic Scene Editing in Autonomous Driving

arXiv:2605.01995v1 Announce Type: new Abstract: The perception of an Autonomous Driving System (ADS) critically depends on relevant, comprehensive, and diverse datasets to ensure its safety while oper

safetyarxiv-cs-cv

GR-Ben: A General Reasoning Benchmark for Evaluating Process Reward Models

arXiv:2605.01203v1 Announce Type: cross Abstract: Currently, process reward models (PRMs) have exhibited remarkable potential for test-time scaling. Since large language models (LLMs) regularly genera

Pandora's Regret: A Proper Scoring Rule for Evaluating Sequential Search

model-releasesarxiv-cs-lg

arXiv:2605.01936v1 Announce Type: new Abstract: In sequential search, alternatives are tested until the true class is found. Standard proper scoring rules like log loss are local, ignoring the ranking

FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios

arXiv:2605.00706v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly applied in financial scenarios. However, they may produce harmful outputs, including facilitating illegal

4 May 2026

Image Score: Learning and Evaluating Human Preferences for Mercari Search

arXiv:2408.11349v2 Announce Type: replace Abstract: Mercari is the largest C2C e-commerce marketplace in Japan, having more than 20 million active monthly users. Search being the fundamental way to di

researcharxiv-cs-cv

4 May 2026

Linking Behaviour and Perception to Evaluate Meaningful Human Control over Partially Automated Driving