Search: “evaluation”

ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

model-releaseshugging-face

ITBench-AA is a new benchmark developed by Artificial Analysis and IBM that evaluates frontier AI models on agentic enterprise IT tasks, with results showing that current leading models score below 50

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers

DGX agent

arXiv:2605.26731v1 Announce Type: new Abstract: A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models n

model-releasesarxiv-cs-ai

27 May 2026

MATCHA: Matching Text via Contrastive Semantic Alignment

arXiv:2605.27345v1 Announce Type: new Abstract: Reliable evaluation is essential for understanding large language model (LLM) performance, yet today's go-to metrics, namely token-overlap scores (e.g.,

safetyarxiv-cs-cl

OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning

arXiv:2505.17163v2 Announce Type: replace-cross Abstract: Recent advancements in multimodal slow-thinking systems have demonstrated remarkable performance across various visual reasoning tasks. Howeve

Tutorials

Optimising Factual Consistency in Summarisation via Preference Learning from Multiple Imperfect Metrics

arXiv:2605.26840v1 Announce Type: new Abstract: Reinforcement learning with evaluation metrics as rewards is widely used to enhance specific capabilities of language models. However, for tasks such as

tutorialsarxiv-cs-cl

ORLoopBench: Solver-in-the-Loop Benchmarks for Self-Correction and Behavioral Rationality in Operations Research

arXiv:2601.21008v3 Announce Type: replace-cross Abstract: Operations Research practitioners debug infeasible models through an iterative process: inspecting Irreducible Infeasible Subsystems ( IIS), i

Persona Generators: Generating Diverse Synthetic Personas for Arbitrary Contexts

arXiv:2602.03545v2 Announce Type: replace Abstract: Evaluating AI systems that interact with humans requires understanding their behavior across diverse user populations, but collecting representative

agentsarxiv-cs-ai

PRBench: A Standardized Probabilistic Robustness Benchmark

model-releasesarxiv-cs-cv

arXiv:2511.01724v3 Announce Type: replace Abstract: Deep learning models are notoriously vulnerable to imperceptible perturbations. Most existing research centers on adversarial robustness (AR), which

SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?

model-releasesarxiv-cs-lg

arXiv:2605.26548v1 Announce Type: cross Abstract: Large language models (LLMs) now support automated software security tasks, including vulnerability discovery and proof-of-concept (PoC) generation. E

SPEAR: Code-Augmented Agentic Prompt Optimization

arXiv:2605.26275v1 Announce Type: new Abstract: Automatic prompt engineering (APE) rewrites prompts to improve downstream task performance, but existing APE loops treat the optimizer itself as a fixed

agentsarxiv-cs-cl

The AI Cognitive Trojan Horse: How Large Language Models May Bypass Human Epistemic Vigilance

arXiv:2601.07085v2 Announce Type: replace-cross Abstract: Large language model (LLM)-based conversational AI systems present a challenge to human cognition that current frameworks for understanding mi

When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Control

arXiv:2605.26418v1 Announce Type: cross Abstract: A properly calibrated rule-based autoscaler can beat every one of six mainstream deep reinforcement learning (DRL) algorithms on cost across every wor

A Two-Phase Stability Study of LLM Judges and Bar Council Examiners on Thai Bar-Exam Free-Form Essays

model-releasesarxiv-cs-cl

arXiv:2605.25652v1 Announce Type: new Abstract: Free-form legal essay evaluation in NLP treats expert inter-rater stability as a single ceiling number, and treats LLM-judge agreement with that ceiling

AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions

arXiv:2605.25707v1 Announce Type: new Abstract: Autonomous computer use agents that powered by multimodal large language models (MLLMs) are emerging as capable assistants for completing complex digita

AuthTrace: Diagnosing Evidence Construction in Thematically Dense Single-Author Corpora

model-releasesarxiv-cs-cl

arXiv:2605.25382v1 Announce Type: new Abstract: Evidence construction systems--chunk retrieval, agent memory, knowledge-graph traversal, and thematic indexing--are evaluated on separate benchmarks wit

Local Ai

AutoSG: LLM-Driven Solver Generation Solely from Task Prompts for Expensive Optimization

arXiv:2605.25658v1 Announce Type: cross Abstract: Expensive optimization tasks are ubiquitous in real-world applications, demanding highly specialized solvers. While LLM-driven automated solver genera

local-aiarxiv-cs-ai

Bayesian Distributional Models of Executive Functioning

model-releasesarxiv-cs-lg

arXiv:2510.00387v3 Announce Type: replace Abstract: This study uses controlled simulations with known ground-truth parameters to evaluate how Distributional Latent Variable Models (DLVM) and Bayesian

Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork

arXiv:2605.24423v1 Announce Type: new Abstract: In-Context Reinforcement Learning (ICRL) has enabled foundation agents to adapt instantaneously to novel tasks, yet its efficacy in Ad-Hoc Teamwork (AHT

Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows

arXiv:2605.24219v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly deployed as autonomous agents that reason, use tools, and act over multiple steps. Yet most hallucination

DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations

arXiv:2512.19097v3 Announce Type: replace-cross Abstract: Intracranial EEG (iEEG) provides direct, millisecond-scale recordings of human neural activity, but reusable representation learning is diffic

Double Triangle Annotation: A Scalable Human-in-the-Loop Framework for High-Precision Historical Document Annotation

model-releasesarxiv-cs-cl

arXiv:2605.25781v1 Announce Type: new Abstract: Evaluating structured-information extraction from historical documents at scale requires high-precision ground-truth annotations, yet traditional manual

Efficient Benchmarking Is Just Feature Selection and Multiple Regression

arXiv:2605.25773v1 Announce Type: cross Abstract: Efficient benchmarking techniques aim to lower the computational cost of evaluating LLMs by predicting full benchmark scores using only a subset of a

Explore Before You Solve: The Speed--Depth Trade-off in Epistemic Agents for ARC-AGI-3

arXiv:2605.25931v1 Announce Type: new Abstract: We systematically investigate all 25 public ARC-AGI-3 games and find that every one is reachable through non-intelligent strategies: 10 in a single blin

Faithful or Fabricated? A Causal Framework for Rationalization Bias in LLM Judges

arXiv:2605.23970v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as automatic judges for summarization and dialogue evaluation. Prior work has documented biases such

safetyarxiv-cs-cl

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers

arXiv:2603.05143v3 Announce Type: replace Abstract: Understanding reasoning in large language models is complicated by evaluations that conflate multiple reasoning types. We isolate analogical reasoni

researcharxiv-cs-cl

Fine-Tuning Language Models to Know What They Know

arXiv:2602.02605v2 Announce Type: replace-cross Abstract: Evaluating true metacognition in Large Language Models (LLMs) is difficult due to biases and heuristics. This paper presents a framework to me

FLOATBench: A Dataset and Benchmark for Floating Offshore Wind Turbine Tower Fatigue

arXiv:2605.25717v1 Announce Type: new Abstract: Most of the world's offshore wind resource lies in waters too deep for fixed-bottom foundations, making floating offshore wind turbines (FOWTs) essentia

FloorplanQA: A Benchmark for Spatial Reasoning in LLMs using Structured Representations

arXiv:2507.07644v4 Announce Type: replace Abstract: We introduce FloorplanQA, a diagnostic benchmark for evaluating spatial reasoning in large language models (LLMs). FloorplanQA is grounded in struct

From Knowledge to Inference: Formalizing Specialized Public Health Reasoning on GlobalHealthAtlas

arXiv:2602.00491v2 Announce Type: replace Abstract: Public health reasoning requires population level inference grounded in scientific evidence, expert consensus, and safety constraints. However, it r

safetyarxiv-cs-cl

How Many Tools Should an LLM Agent See? A Chance-Corrected Answer

arXiv:2605.24660v1 Announce Type: cross Abstract: Before an LLM agent can use a tool, a retrieval system must decide which candidate tools to show to the agent. How long should that shortlist be? Show

Inverting the Shield: Systematically Generating Safety Tests from Policy Specifications

arXiv:2605.24883v1 Announce Type: new Abstract: The widespread integration of Large Language Models (LLMs) necessitates rigorous and systematic safety evaluation. Existing paradigms either rely on con

Is Human Annotation Necessary? Iterative MBR Distillation for Error Span Detection in Machine Translation

arXiv:2603.12983v3 Announce Type: replace-cross Abstract: Error Span Detection (ESD) is a crucial subtask in Machine Translation (MT) evaluation, aiming to identify the location and severity of transl

researcharxiv-cs-ai

Length Generalization with Log-Depth Recurrent Units

arXiv:2605.26035v1 Announce Type: new Abstract: Length generalization remains a persistent challenge for neural networks: recurrent models tend to suffer from positional biases, while transformers are

researcharxiv-cs-lg

Machine Psychometrics: A Mathematical Psychology of Artificial Intelligence

arXiv:2605.23952v1 Announce Type: new Abstract: Artificial agents now generate behavior rich enough to invite trust, surprise, and concern, yet our evaluation tools still privilege capability scores o

Measuring the Depth of LLM Unlearning via Activation Patching

arXiv:2605.24614v1 Announce Type: cross Abstract: Large language model (LLM) unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether target kn

Meta-Engineering Harnesses for AI-Native Software Production: A Contract-Driven Adversarial Verification Architecture with Early Deployment Report

arXiv:2605.25665v1 Announce Type: cross Abstract: AI-native software development is often evaluated at the level of individual models, prompts, or generated artifacts. This framing is insufficient for

agentsarxiv-cs-ai

Local Ai

Model test: SenseNova U1 vs GPT Image2 vs Nano Banana in Infographic generation

local-air-stablediffusion

A comparative test of three AI image generation models—SenseNova U1, GPT Image 2, and Nano Banana 2—evaluating their performance on infographic generation tasks. GPT Image 2 proved more reliable for e

Path Following Control System of Line-of-Sight Guidance for Robotic Dolphin with Multi-Link Mechanism in Underwater Simulator

arXiv:2605.25401v1 Announce Type: new Abstract: Biomimetic autonomous underwater vehicle (BAUV) with multi-link mechanism is widely used in aquatic life observation and environmental surveys due to it

agentsarxiv-cs-ro

PRIMA: Operational Patterns for Resilient Multi-Agent Research with Verifiable Identity and Convergent Feedback

arXiv:2605.24775v1 Announce Type: new Abstract: Operating LLMs as coordinated multi-agent research systems over multi-hour runs surfaces failure modes that single-shot evaluation cannot: upstream prov

agentsarxiv-cs-ai

Local Ai

Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients

arXiv:2605.25020v1 Announce Type: new Abstract: Chronic dermatologic diseases such as pemphigus require long-term follow-up, generating extensive longitudinal clinical documentation that is difficult

local-aiarxiv-cs-ai

PromptAudit: Auditing Prompt Sensitivity in LLM-Based Vulnerability Detection

arXiv:2605.24171v1 Announce Type: cross Abstract: Large language models are increasingly used for vulnerability detection, yet their reliability under different prompt formulations remains uncharacter

researcharxiv-cs-ai

Rethinking Continual Anomaly Detection on the Edge: Benchmarking Under Realistic Industrial Conditions

model-releasesarxiv-cs-lg

arXiv:2605.24251v1 Announce Type: new Abstract: Continual anomaly detection (CAD) addresses the need for industrial inspection systems to adapt to evolving production conditions, yet existing methods

RoboManipBaselines: A Unified Framework for Imitation Learning in Robotic Manipulation across Real and Simulation Environments

model-releasesarxiv-cs-ro

arXiv:2509.17057v3 Announce Type: replace Abstract: We present RoboManipBaselines, an open-source software framework for imitation learning research in robotic manipulation. The framework supports the

Safety Generalization Under Distribution Shift in Safe Reinforcement Learning: A Diabetes Testbed

arXiv:2601.21094v2 Announce Type: replace-cross Abstract: Safe Reinforcement Learning (RL) algorithms are typically evaluated under fixed training conditions. We investigate whether training-time safe

SMDD-Bench: Can LLMs Solve Real-World Small Molecule Drug Design Tasks?

arXiv:2605.21740v2 Announce Type: replace Abstract: LLM agents have incredible potential for scientific discovery applications. However, the performance of LLM agents on real-world, small molecule dru

Teaching large language models to reason like expert diagnosticians

arXiv:2509.12194v2 Announce Type: replace Abstract: Differential diagnosis is an iterative process that integrates patient information with broader medical knowledge. Clinical case series such as the

Teaching Through Analogies: A Modular Pipeline for Educational Analogy Generation

arXiv:2605.24211v1 Announce Type: cross Abstract: Analogies help learners understand unfamiliar concepts by relating them to known concepts. Despite recent advances, large language models (LLMs) conti

TRACER: A Semantic-Aware Framework for Fine-Grained Contamination Detection in Code LLMs

arXiv:2605.24079v1 Announce Type: cross Abstract: Data contamination is a known threat to the reliability of model evaluation. However, it remains underexplored in code large language models (LLMs), w