Search: “evaluation”

arXiv:2607.26000v1 Announce Type: cross Abstract: Tabular Foundation Models (TFMs) have emerged as novel approaches for tabular predictive tasks, demonstrating competitive predictive performance to en

applicationsarxiv-cs-ai

29 Jul 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Safety

Evaluation of forced alignment of code-mixed speech: the case of Hindi-English

DGX agent

arXiv:2607.25581v1 Announce Type: new Abstract: Code-mixed speech poses unique challenges to forced alignment: expanded inventories, orthographic errors, and speaker variation. We evaluate forced alig

safetyarxiv-cs-cl

29 Jul 2026

Applications

Eye Tracking Based Cognitive Evaluation of Automatic Readability Assessment Methods

arXiv:2502.11150v5 Announce Type: replace Abstract: Automatic methods for scoring text readability have been studied for over a century, and are widely used in research and in user-facing applications

applicationsarxiv-cs-cl

29 Jul 2026

Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees

arXiv:2607.21623v1 Announce Type: new Abstract: We present EaaS, a cloud-native reference architecture that operationalizes AI evaluation methods as six stateless Kubernetes microservices: conformal p

safetyarxiv-cs-lg

27 Jul 2026

Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings

arXiv:2607.21962v1 Announce Type: new Abstract: Benchmarks for LLM-agent memory typically generate conversations first and extract answer keys afterwards -- with documented label-error and contaminati

27 Jul 2026

An Evaluation Framework for Structured Audio Captions Validated by Controlled Perturbations

arXiv:2607.21424v1 Announce Type: new Abstract: Recent advancements in automated audio captioning (AAC) have shifted from monolithic sentence generation toward structured formats that explicitly disen

researcharxiv-cs-cl

From Evaluation to Optimisation: Hierarchy-Aware Training Signals for CWE Prediction in Python

arXiv:2607.21069v1 Announce Type: new Abstract: The original ALPHA benchmark introduced a taxonomy-aware penalty for evaluating CWE-level vulnerability prediction in Python and proposed that the penal

Local Ai

Minimum Bayes Risk Decoding for Error Span Detection in Reference-Free Automatic Machine Translation Evaluation

arXiv:2512.07540v4 Announce Type: replace-cross Abstract: Error Span Detection (ESD) extends automatic machine translation (MT) evaluation by localizing translation errors and labeling their severity.

local-aiarxiv-cs-ai

Representing Entity Importance in AI Knowledge Systems: A Dual-Signal Framework of Audience Evaluation and Structural Authority

arXiv:2607.20925v1 Announce Type: new Abstract: AI knowledge systems require representations of entity importance for retrieval, recommendation, evidence selection, and knowledge-intensive reasoning.

Economic Evaluations of Language Models

arXiv:2607.19375v1 Announce Type: cross Abstract: Language models perform economically valuable work, yet they are not currently assessed for how well they perform every economically valuable task. We

23 Jul 2026

Reference-Free Evaluation of Reasoning in Open-Ended Question Answering

arXiv:2607.19678v1 Announce Type: cross Abstract: AI-generated answers in high-stakes domains are often fluent but difficult to verify, especially when they contain multi-step reasoning rather than a

23 Jul 2026

Toward Anthropomorphic Dialogue: A Closed-Loop Framework for Human-Like Chat Generation, Evaluation, and Preference Alignment

arXiv:2607.17191v2 Announce Type: replace Abstract: Human-like private chat requires more than fluent response generation: a system must preserve persona, relationship, memory, bounded knowledge, medi

23 Jul 2026

Evaluation Ability Does Not Imply Optimization Utility: LLM-as-a-Judge Signals in Closed-Loop Table Recognition

arXiv:2607.13347v1 Announce Type: cross Abstract: LLM-as-a-judge is widely used to provide feedback and selection signals in closedloop regeneration, but this use remains insufficiently validated. We

16 Jul 2026

HEDGEHOG: Hierarchical Evaluation of Drug Generators Through Rigorous Filtration

arXiv:2607.13155v1 Announce Type: new Abstract: Generative molecular models can support early drug discovery by proposing new candidate compounds de novo. In practice, useful candidates must balance t

16 Jul 2026

DR-Arena: an Automated Evaluation Framework for Deep Research Agents

arXiv:2601.10504v2 Announce Type: replace Abstract: As Large Language Models (LLMs) increasingly operate as Deep Research (DR) Agents capable of autonomous investigation and information synthesis, rel

safetyarxiv-cs-cl

10 Jul 2026

TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing

arXiv:2510.04100v2 Announce Type: replace-cross Abstract: Topological mapping offers a compact and robust representation for navigation, but progress in the field is hindered by the lack of standardiz

10 Jul 2026

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

arXiv:2607.06713v1 Announce Type: cross Abstract: Large language models are rapidly moving towards closing the development cycle, transitioning from simple assistive companions to autonomous contribut

agentsarxiv-cs-ai

9 Jul 2026

Beyond Static Evaluation: Building Simulation Environments for Scalable Agentic Reinforcement Learning

arXiv:2607.05773v1 Announce Type: new Abstract: As Large Language Models (LLMs) evolve into autonomous agents, traditional static evaluation fails to capture multi-step decision-making. We introduce A

agentsarxiv-cs-ai

8 Jul 2026

Dynamic Evaluation of Classical and Control-Aware Optimal Trajectory Planning in Robot Manipulators

arXiv:2607.05544v1 Announce Type: new Abstract: Trajectory planning strongly influences tracking accuracy, actuator demand, and overall execution behavior in robotic manipulators. Classical planners s

researcharxiv-cs-ro

8 Jul 2026

ClinOCR-Bench: A Comprehensive Clinical Scanned Document Dataset for Optical Character Recognition Model Evaluation

arXiv:2607.03650v1 Announce Type: cross Abstract: Extracting textual information from scanned medical documents, such as external laboratory reports and manually filled forms, has been a major challen

Measuring What Matters: A Unified Evaluation Framework for GNN Explainability

arXiv:2607.04600v1 Announce Type: new Abstract: Graph eXplainable AI (G-XAI) is increasingly important for making Graph Neural Networks interpretable and accountable. While a growing number of explain

researcharxiv-cs-lg

Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

model-releasesarxiv-cs-cv

arXiv:2510.17801v2 Announce Type: replace-cross Abstract: Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a central challenge. Recent embodied systems

VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agents

arXiv:2510.11098v5 Announce Type: replace-cross Abstract: Recent advances in large audio language models (LALMs) have greatly enhanced multimodal conversational systems. However, existing benchmarks r

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

arXiv:2606.22737v2 Announce Type: replace Abstract: Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic tes

3 Jul 2026

Local Ai

Who Gets the Reward & Who Gets the Blame? Evaluation-Aligned Training Signals for Multi-LLM Agents

arXiv:2511.10687v3 Announce Type: replace-cross Abstract: Large Language Models (LLMs) in multi-agent systems (MAS) have shown promise for complex tasks, yet current training methods lack principled w

local-aiarxiv-cs-ai

3 Jul 2026

ALEE: Any-Language Evaluation of Embeddings via English-Centric Minimal Pairs

arXiv:2607.00171v1 Announce Type: new Abstract: Text embeddings are standard for semantic similarity tasks, yet their evaluation remains an open challenge. Current benchmarks are static, cover only a

researcharxiv-cs-cl

2 Jul 2026

TRIE: An Evaluation Framework for Stochastic PDE Surrogates

arXiv:2607.00196v1 Announce Type: new Abstract: Many scientific systems exhibit uncertainty from stochastic forcing, unresolved degrees of freedom, or imperfect observations, making reliable surrogate

2 Jul 2026

CAN We Trust Your Results? A Cross-Dataset Study of Automotive IDS Evaluation

arXiv:2606.30430v1 Announce Type: cross Abstract: The increasing connectivity of modern vehicles has made securing in-vehicle communication networks a critical challenge. Intrusion Detection Systems (

researcharxiv-cs-lg

Envisage: Diffusion-Based Rhinoplasty Goal Visualization with Mask-Decomposed Evaluation

model-releasesarxiv-cs-cv

arXiv:2606.28628v1 Announce Type: cross Abstract: Localized generative editing needs localized evaluation: full-image identity metrics are structurally confounded under hard-composited edits. We prese

Tutorials

Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving

arXiv:2606.29493v1 Announce Type: new Abstract: Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-check

tutorialsarxiv-cs-ai

Fine-Tuning General-Purpose Large Language Models for Agricultural Applications:A Reproducible Framework and Evaluation Protocol Based on Qwen3-8B

arXiv:2606.28992v1 Announce Type: cross Abstract: General-purpose large language models (LLMs) have demonstrated strong abilities in opendomain question answering, information extraction, and text gen

Multi-Agent Routing as Set-Valued Prediction: A WildChat Benchmark and Cost-Aware Evaluation

arXiv:2606.28925v1 Announce Type: cross Abstract: Tool and agent routing from natural-language prompts is naturally a set-valued prediction problem: a single query may require multiple agents, while o

Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation

arXiv:2602.09305v2 Announce Type: replace Abstract: Large Language Models (LLMs) demonstrate transformative potential, yet their reasoning remains inconsistent and unreliable. Reinforcement learning (

safetyarxiv-cs-lg

SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

arXiv:2606.29520v1 Announce Type: cross Abstract: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software

The Complexity Ceiling Benchmark: A Multi-Domain Evaluation of Sequential Reasoning Under Depth Scaling

arXiv:2606.29278v1 Announce Type: new Abstract: We introduce the Complexity Ceiling Benchmark (CCB), a controlled evaluation of how language-model reasoning decays as the number of required sequential

Inherited Circuits, Learned Semantics: How Fine-Tuning Creates Evasion Vulnerabilities Invisible to Standard Evaluation

arXiv:2606.27091v1 Announce Type: cross Abstract: LLMs fine-tuned for security classification are usually evaluated on held-out examples from the same distribution as their training data. We show that

26 Jun 2026

Tutorials

Scalable Behavior Cloning with Open Data, Training, and Evaluation

arXiv:2606.27375v1 Announce Type: new Abstract: We introduce ABC, a fully open-source stack for manipulation with behavior cloning. At its core is ABC-130K: the largest open-source teleoperation datas

tutorialsarxiv-cs-ro

26 Jun 2026

What We are Missing in Multimodal LLM Evaluation?

arXiv:2606.26348v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses. While their c

26 Jun 2026

A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

arXiv:2606.25476v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated remarkable performance across natural language processing tasks, yet their deployment in high-stakes appl

PERRY: Policy Evaluation with Confidence Intervals using Auxiliary Data

arXiv:2507.20068v3 Announce Type: replace Abstract: Off-policy evaluation (OPE) methods estimate the value of a new reinforcement learning (RL) policy prior to deployment. Recent advances have shown t

safetyarxiv-cs-lg

Robustness assessment of large audio language models in multiple-choice evaluation

arXiv:2510.04584v2 Announce Type: replace Abstract: Recent advances in large audio language models (LALMs) have primarily been assessed using a multiple-choice question answering (MCQA) framework. How

SycoEval-EM: Sycophancy Evaluation of Large Language Models in Simulated Clinical Encounters for Emergency Care

arXiv:2601.16529v3 Announce Type: replace Abstract: Large language models (LLMs) deployed in clinical decision support may acquiesce to patient requests for care that conflicts with evidence-based gui

The Evaluation Cost of Task Specialization in Evolutionary Multi-Robot Systems

arXiv:2606.24191v1 Announce Type: new Abstract: Task specialization can improve the efficiency of multi-robot systems (MRSs). Previous works have investigated the emergence of task-specialist robot co

researcharxiv-cs-ro

24 Jun 2026

Robust Tightly-Coupled Filter-Based Monocular Visual-Inertial State Estimation and Graph-Based Evaluation for Autonomous Drone Racing

arXiv:2603.02742v2 Announce Type: replace Abstract: Autonomous drone racing (ADR) demands state estimation that is simultaneously computationally efficient and resilient to the perceptual degradation

agentsarxiv-cs-ro

23 Jun 2026

When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents

arXiv:2606.22864v1 Announce Type: new Abstract: Hidden-state probing -- a linear classifier on a frozen vision-language model's internal activations -- has emerged as an attractive evaluation tool for

23 Jun 2026

Conditional Vendi Score: Prompt-Aware Diversity Evaluation for Generative AI Models and LLMs

arXiv:2411.02817v2 Announce Type: replace-cross Abstract: Generative models guided by text prompts are widely evaluated for fidelity and prompt alignment, yet their ability to produce outputs remains

10 Jun 2026

PhysMetrics.Weather: An Evaluation Framework for Physical Consistency in ML Weather Models

arXiv:2606.10642v1 Announce Type: new Abstract: Machine learning weather prediction (MLWP) models have achieved impressive forecasting performance at a small fraction of the computational costs requir

researcharxiv-cs-lg

10 Jun 2026

Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation