Search: “evaluation”

ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation

arXiv:2607.03650v1 Announce Type: cross Abstract: Extracting textual information from scanned medical documents, such as external laboratory reports and manually filled forms, has been a major challen

7 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Research

Measuring What Matters: A Unified Evaluation Framework for GNN Explainability

DGX agent

arXiv:2607.04600v1 Announce Type: new Abstract: Graph eXplainable AI (G-XAI) is increasingly important for making Graph Neural Networks interpretable and accountable. While a growing number of explain

researcharxiv-cs-lg

7 Jul 2026

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

arXiv:2510.17801v2 Announce Type: replace-cross Abstract: Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems

7 Jul 2026

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

arXiv:2510.11098v5 Announce Type: replace-cross Abstract: Recent advances in large audio language models (LALMs) have greatly enhanced multimodal conversational systems. However, existing benchmarks r

7 Jul 2026

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

arXiv:2606.22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic tes

3 Jul 2026

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

arXiv:2511.10687v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled w

local-aiarxiv-cs-ai

3 Jul 2026

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v1 Announce Type: new Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a

researcharxiv-cs-cl

2 Jul 2026

TRIE: An Evaluation Framework for Stochastic PDE Surrogates

model-releasesarxiv-cs-lg

arXiv:2607.00196v1 Announce Type: new Abstract: Many scientific systems exhibit uncertainty from stochastic forcing, unresolved degrees of freedom, or imperfect observations, making reliable surrogate

2 Jul 2026

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606.30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (

researcharxiv-cs-lg

Envisage: Diffusion-Based Rhinoplasty Goal Visualization with Mask-Decomposed Evaluation

arXiv:2606.28628v1 Announce Type: cross Abstract: Localized generative editing needs localized evaluation: full-image identity metrics are structurally confounded under hard-composited edits. We prese

Tutorials

Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

arXiv:2606.29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-check

tutorialsarxiv-cs-ai

Fine-Tuning General-Purpose Large Language Models for Agricultural Applications:A Reproducible Framework and Evaluation Protocol Based on Qwen3-8B

arXiv:2606.28992v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have demonstrated strong abilities in opendomain question answering, information extraction, and text gen

Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

arXiv:2606.28925v1 Announce Type: cross Abstract: Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while o

Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

arXiv:2602.09305v2 Announce Type: replace Abstract: Large Language Models (LLMs) demonstrate transformative potential, yet their reasoning remains inconsistent and unreliable. Reinforcement learning (

safetyarxiv-cs-lg

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

arXiv:2606.29520v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software

The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling

arXiv:2606.29278v1 Announce Type: new Abstract: We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays as the number of required sequential

Industry

AI leaderboard provider Arena says it hit $100M in annualized run-rate revenue eight months after launching AI Evaluations, which offers performance analytics (Marina Temkin/TechCrunch)

Marina Temkin / TechCrunch: AI leaderboard provider Arena says it hit $100M in annualized run-rate revenue eight months after launching AI Evaluations, which offers performance analytics — Just eight

industrytechmeme

29 Jun 2026

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

arXiv:2606.27091v1 Announce Type: cross Abstract: LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that

26 Jun 2026

Tutorials

Scalable Behavior Cloning with Open Data, Training, and Evaluation

arXiv:2606.27375v1 Announce Type: new Abstract: We introduce ABC, a fully open-source stack for manipulation with behavior cloning. At its core is ABC-130K: the largest open-source teleoperation datas

tutorialsarxiv-cs-ro

26 Jun 2026

What We are Missing in Multimodal LLM Evaluation?

arXiv:2606.26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses. While their c

26 Jun 2026

A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

arXiv:2606.25476v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable performance across natural language processing tasks, yet their deployment in high-stakes appl

PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data

arXiv:2507.20068v3 Announce Type: replace Abstract: Off-policy evaluation (OPE) methods estimate the value of a new reinforcement learning (RL) policy prior to deployment. Recent advances have shown t

safetyarxiv-cs-lg

Robustness assessment of large audio language models in multiple-choice evaluation

arXiv:2510.04584v2 Announce Type: replace Abstract: Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. How

SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care

arXiv:2601.16529v3 Announce Type: replace Abstract: Large language models (LLMs) deployed in clinical decision support may acquiesce to patient requests for care that conflicts with evidence-based gui

safetyarxiv-cs-ai

The Evaluation Cost of Task Specialization in Evolutionary Multi-Robot Systems

arXiv:2606.24191v1 Announce Type: new Abstract: Task specialization can improve the efficiency of multi-robot systems (MRSs). Previous works have investigated the emergence of task-specialist robot co

researcharxiv-cs-ro

24 Jun 2026

Agents

Robust Tightly-Coupled Filter-Based Monocular Visual-Inertial State Estimation and Graph-Based Evaluation for Autonomous Drone Racing

arXiv:2603.02742v2 Announce Type: replace Abstract: Autonomous drone racing (ADR) demands state estimation that is simultaneously computationally efficient and resilient to the perceptual degradation

agentsarxiv-cs-ro

23 Jun 2026

When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents

model-releasesarxiv-cs-lg

arXiv:2606.22864v1 Announce Type: new Abstract: Hidden-state probing -- a linear classifier on a frozen vision-language model's internal activations -- has emerged as an attractive evaluation tool for

23 Jun 2026

Industry

Many people think any given ML project is 99% training. In reality, it’s 50% evaluation, 40% data cleaning, 8% integration, and 2% training.…

Many people think any given ML project is 99% training. In reality, it’s 50% evaluation, 40% data cleaning, 8% integration, and 2% training. The first two set the noise floor for learning. No ML magic

industryclem-delangue--x

20 Jun 2026

Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

arXiv:2411.02817v2 Announce Type: replace-cross Abstract: Generative models guided by text prompts are widely evaluated for fidelity and prompt alignment, yet their ability to produce outputs remains

safetyarxiv-cs-ai

10 Jun 2026

PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models

arXiv:2606.10642v1 Announce Type: new Abstract: Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs requir

researcharxiv-cs-lg

10 Jun 2026

Agents

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation

arXiv:2606.10749v1 Announce Type: cross Abstract: Large language model (LLM) agents are rapidly moving from conversational interfaces to software components that plan, invoke tools, maintain memory, a

agentsarxiv-cs-ai

10 Jun 2026

Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese

arXiv:2606.07853v1 Announce Type: cross Abstract: Large Language Models are transforming the support for clinical decision and their application in real scenarios. Yet, most benchmarks are conducted i

9 Jun 2026

Rank Intervals for Leaderboards: A Hierarchical Framework for Model Evaluation

arXiv:2606.08679v1 Announce Type: cross Abstract: Pretrained models are often evaluated on multi-task leaderboards to measure their applicability in diverse contexts. However, current methods for aggr

researcharxiv-cs-lg

9 Jun 2026

Testing the Black Box: Structural Barriers to Independent Evaluation of Consumer-Facing Health LLMs

arXiv:2606.08483v1 Announce Type: new Abstract: Background: Consumer-facing large language models are now a common source of health information, and they interpret and personalize responses rather tha

safetyarxiv-cs-ai

9 Jun 2026

From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning

arXiv:2606.07190v1 Announce Type: new Abstract: Reasoning prefixes shape the future trajectory of LLM problem solving, yet existing process reward models usually evaluate them through local step corre

local-aiarxiv-cs-cl

8 Jun 2026

Off-Policy Evaluation with Strategic Agents via Local Disclosure

arXiv:2606.07308v1 Announce Type: new Abstract: We study off-policy evaluation (OPE) under strategic behavior where decision subjects (or agents) respond to a decision maker's policy by strategically

local-aiarxiv-cs-ai

8 Jun 2026

SWE-IF: Aligning Code Evaluation with Human Preference

arXiv:2510.07315v2 Announce Type: replace-cross Abstract: Large Language Models (LLMs) have catalyzed vibe coding, where users leverage LLMs to generate and iteratively refine code through natural lan

researcharxiv-cs-ai

8 Jun 2026

Efficient Asynchronous Federated Evaluation with Strategy Similarity Awareness for Intent-Based Networking in Industrial Internet of Things

arXiv:2512.20627v2 Announce Type: replace-cross Abstract: Intent-Based Networking (IBN) offers a promising paradigm for intelligent and automated network control in Industrial Internet of Things (IIoT

local-aiarxiv-cs-ai

6 Jun 2026

Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

arXiv:2606.05403v1 Announce Type: cross Abstract: Language models increasingly act as epistemic proxies, synthesizing evidence from multiple sources to inform decisions. Whether they evaluate the qual

6 Jun 2026

Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models

arXiv:2606.05531v1 Announce Type: cross Abstract: Despite the rapid progress of Vision-Language Models (VLMs), the field lacks benchmarks that rigorously diagnose their true reasoning abilities and ch

5 Jun 2026

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

arXiv:2606.05308v1 Announce Type: cross Abstract: With PRECISE, we extended Prediction-Powered Inference to produce bias-corrected estimates of ranking evaluation metrics by combining a small human-la

5 Jun 2026

Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models

arXiv:2510.20042v3 Announce Type: replace Abstract: Generative image models produce striking visuals yet often misrepresent culture. Prior work has examined cultural bias mainly in text-to-image (T2I)

Model Evaluations: Prove Your Routing Policy Actually Works

This article discusses methods and tools for evaluating routing policies in machine learning models, likely covering techniques to validate that model routing decisions are effective and functioning a

safetydigitalocean

Toward Trustworthy Portrait Editing: Evaluation of Demographic Misrepresentation in I2I Models

arXiv:2602.16149v2 Announce Type: replace Abstract: Instruction-guided image-to-image (I2I) editors are increasingly used in consumer and professional visual workflows, where trustworthiness depends n

VGGSounder: Audio-Visual Evaluations for Foundation Models

arXiv:2508.08237v4 Announce Type: replace-cross Abstract: The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound

OpenAI diverges from Trump's AI EO in a new policy paper, proposing cyber risk evaluations for advanced AI systems be mandatory and led by CAISI, not the NSA (Brendan Bordelon/Politico)

Brendan Bordelon / Politico: OpenAI diverges from Trump's AI EO in a new policy paper, proposing cyber risk evaluations for advanced AI systems be mandatory and led by CAISI, not the NSA — OpenAI's ne

safetytechmeme

3 Jun 2026

Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill

arXiv:2606.03980v1 Announce Type: cross Abstract: Reward models (RMs) provide critical feedback signals for LLM post-training, notably in reinforced fine-tuning (RFT) and reinforcement learning (RL) p

3 Jun 2026

A Multi-Domain Red Teaming Framework for Safety, Robustness, and Fairness Evaluation of Medical Large Language Models

arXiv:2606.00027v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed across healthcare, yet existing benchmarks fail to capture model behavior under adversarial or