Search: “evaluation”

WebArxiv: A Reproducible Benchmark for Evaluating Multimodal Web Agents on arXiv Tasks

arXiv:2507.00938v4 Announce Type: replace-cross Abstract: Foundation models now enable autonomous agents to interact with real-world websites, but existing benchmarks emphasize general-purpose browsin

25 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Evaluating Feedback Focus and Pedagogical Adaptivity in LLM-Generated Feedback on Student Writing

DGX agent

arXiv:2609.28026v1 Announce Type: cross Abstract: We investigate whether state-of-the-art large language models (LLMs) generate feedback that reflects the pedagogical practices of expert teachers in t

model-releasesarxiv-cs-ai

24 Sept 2026

X2Real: an eXtensive simulation benchmark for real-world generalist policies

model-releasesarxiv-cs-ro

arXiv:2609.27449v1 Announce Type: new Abstract: Generalist robot manipulation policies have developed rapidly, yet their reliable evaluation remains challenging due to fundamental flaws in existing si

24 Sept 2026

A Semiotics-Aware Framework for Evaluating Fidelity and Coverage in Natural Language Generation

arXiv:2609.26527v1 Announce Type: new Abstract: When two texts describe the same expression, standard metrics based on lexical overlap or whole-text similarity may fail to detect meaningful difference

safetyarxiv-cs-cl

BackTrend: Evaluating Scientific Weak-Signal Prediction via Backward Reconstruction

arXiv:2609.24921v1 Announce Type: new Abstract: Scientific weak signals are early, low-visibility research directions that later become central to mature scientific topics, yet existing resources such

Evaluating Accuracy and Probabilistic Reliability of Zero-Shot Time Series Foundation Models

arXiv:2609.25788v1 Announce Type: new Abstract: Time Series Foundation Models (TSFMs) promise a paradigm shift toward zero-shot forecasting by eliminating task-specific training. However, existing wor

Evaluating the Effectiveness of SechKAN on 1D Data

arXiv:2609.25876v1 Announce Type: new Abstract: The connection between the Kolmogorov-Arnold representation theorem (KART) and neural network design has led to the development of Kolmogorov-Arnold Net

A Comparative Framework for Evaluating Foundation Models on Tabular Data: A Case Study in Healthcare

arXiv:2609.22154v1 Announce Type: new Abstract: Tabular data is the most common format in clinical practice, encompassing laboratory results, medication records, diagnostic codes, and patient demograp

applicationsarxiv-cs-lg

ChartJudgeBench: Evaluating LMM Judges for Chart-to-Code Generation

arXiv:2609.24210v1 Announce Type: new Abstract: Building strong chart-to-code systems increasingly relies on reinforcement learning, whose effectiveness depends critically on the quality of the reward

Do LLMs Choose Like Humans? Using Cognitive Theory to Evaluate LLM Decision-Making

arXiv:2609.22225v1 Announce Type: new Abstract: Large language models (LLMs) exhibit a range of human-like decision-making behaviors, but whether these reflect similar underlying mechanisms or surface

Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRI

arXiv:2609.23983v1 Announce Type: new Abstract: Neuroimaging foundation models pretrained on large, predominantly western cohorts are increasingly proposed as general-purpose backbones for brain MRI a

MSI-Bench: Evaluating Multi-Speaker Voice Interaction for Collaborative AI Agents

arXiv:2609.24812v1 Announce Type: new Abstract: Voice provides a natural and immediate interface for AI agents. Many settings in which voice agents could be useful, including meetings, households, and

To Consolidate or not to Consolidate? Evaluating the Impact of Consolidation in Multi-Reference Training using Peer Reviews

arXiv:2609.22805v1 Announce Type: new Abstract: Natural language generation (NLG) tasks span the spectrum of conditional entropy, ranging from highly constrained machine translation to open-ended dial

Evaluating In-Context Learning and Retrieval Strategies for Devanagari Post-OCR Correction

arXiv:2609.21595v1 Announce Type: new Abstract: In-context learning using Large Language Models (LLMs) offers a compelling path to training-free post-OCR correction, yet its effectiveness for Devanaga

21 Sept 2026

Evaluating Financial Sentiment in the Age of AI

arXiv:2609.20198v1 Announce Type: new Abstract: Financial sentiment measures are widely used in empirical finance, but it remains unclear whether general-purpose large language models (LLMs) improve o

applicationsarxiv-cs-lg

18 Sept 2026

Evaluating Out-of-Distribution Robustness in Graph-Based Android Malware Classification: A New Principled Benchmark

arXiv:2508.06734v3 Announce Type: replace-cross Abstract: While graph-based Android malware classifiers report strong benchmark accuracy of over 94%, their performance sharply decreases up to 45% when

18 Sept 2026

Prepared Or Unprepared? Evaluating Healthcare Workforce Readiness for Clinical Adoption of Artificial Intelligence in Nigeria

arXiv:2609.19096v1 Announce Type: cross Abstract: Artificial intelligence (AI) is increasingly integrated into healthcare systems worldwide, yet its successful clinical adoption depends critically on

applicationsarxiv-cs-ai

17 Sept 2026

Can LLMs Follow the Pulse of a Crisis? Evaluating Crisis Sentiment in Bangladesh's July Uprising

arXiv:2609.16997v1 Announce Type: new Abstract: Crisis sentiment analysis is especially challenging for low-resource languages such as Bangla, where language, context, and public reaction shift rapidl

16 Sept 2026

Evaluating Mesh Reconstruction Methods for Crop Phenotyping

arXiv:2609.16926v1 Announce Type: new Abstract: Phenotyping an agricultural crop is crucial for studying its entire life cycle, as it provides vital insights to improve yield and, ultimately, food pro

applicationsarxiv-cs-cv

16 Sept 2026

LSREP: A Longitudinal State-Replay Protocol for Evaluating Conversational Memory, with ICE v2 as an Audited Local-First Architecture

arXiv:2609.16730v1 Announce Type: new Abstract: Conversational memory changes during use, so endpoint question answering alone cannot establish how a persistent state accumulates, ages, or incorporate

applicationsarxiv-cs-ai

16 Sept 2026

Agents

CoArena: Evaluating Computer-Use and Multi-Agent Systems in Real Time

arXiv:2609.14239v1 Announce Type: new Abstract: Static benchmarks for computer-use agents fix a task set at release and score every system against it once. That makes them reproducible, and it lets th

agentsarxiv-cs-lg

15 Sept 2026

RFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State Machines

arXiv:2609.13389v1 Announce Type: new Abstract: Mapping textual specifications into formal representations is essential for ensuring the correctness of protocol designs and implementations. LLM-genera

researcharxiv-cs-cl

15 Sept 2026

ParaRecover: A Process-Level Benchmark for Error Localization and Recovery in Parallel Tool-Use Agents

arXiv:2609.12345v1 Announce Type: new Abstract: Existing agent benchmarks mainly evaluate final task success or tool-call correctness, providing limited insight into whether agents can reliably diagno

14 Sept 2026

PEARL: A Task-Aware Framework for Evaluating Differentially Private Synthetic Educational Data

arXiv:2609.10612v1 Announce Type: cross Abstract: Personalized learning systems rely on real learner data, including performance, behavior, and demographic information, but these data are highly priva

safetyarxiv-cs-lg

11 Sept 2026

DI-Bench: Systematically Generating In-Domain Data Intelligence Benchmarks for Enterprise Agents

arXiv:2609.05776v1 Announce Type: new Abstract: Evaluating enterprise agents on domain-specific benchmarks is critical, yet public benchmarks rarely evaluate whether agents can integrate business know

Evaluating the Scalability and Adversarial Generalization of GRPO-Trained NLI Models

arXiv:2504.18376v3 Announce Type: replace-cross Abstract: Natural Language Inference (NLI) is a central task in natural language understanding with applications in fact-checking, question answering, a

S^3-Bench: Evaluating Speech Interaction Models as Scientific Voice Assistants

arXiv:2609.09852v1 Announce Type: new Abstract: The advance of multimodal large language models (MLLMs) has fundamentally reshaped the paradigm of human-computer interaction, especially speech interac

researcharxiv-cs-cl

Spot-the-shift: Evaluating Grounded Image Difference Captioning of Long-term Changes

arXiv:2609.10356v1 Announce Type: new Abstract: Long-term change understanding from images of the same place revisited over time is a challenging task with applications in map maintenance and urban in

The Emerging AI Paper-Review Arms Race: Adversarial Co-Evolution in Scholarly Publishing

arXiv:2609.07713v1 Announce Type: new Abstract: Generative and agentic AI are reshaping both the production and evaluation of scientific research. These developments are often studied separately, as q

safetyarxiv-cs-ai

Evaluating Vision-Language Models for Image Quality Assessment using Psychophysical Data

arXiv:2603.24578v2 Announce Type: replace Abstract: Psychophysical experiments remain the most reliable approach for perceptual image quality assessment (IQA), yet their cost and limited scalability m

9 Sept 2026

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence

arXiv:2609.05385v1 Announce Type: new Abstract: LLM decision components that can operate within agent workflows often produce action-relevant recommendations or judgements together with explanations.

7 Sept 2026

Evaluating GNNs for Success Prediction in Artist Collaboration Networks

arXiv:2609.02920v1 Announce Type: cross Abstract: As the music industry becomes an increasingly collaborative effort, understanding the underlying structures of the artist network has become a focal p

researcharxiv-cs-lg

4 Sept 2026

Evaluating Graph Neural Networks for Change-Criticality Classification in Maritime Navigation Charts

arXiv:2609.02996v1 Announce Type: cross Abstract: Graph neural networks (GNNs) are a class of neural networks suitable for learning on graph-structured data. Their application to spatial data is a nat

safetyarxiv-cs-ai

4 Sept 2026

Principia: Relational Physics Tests for Video Models

arXiv:2609.04200v1 Announce Type: new Abstract: Evaluating physical reasoning in video models is difficult because absolute motion measurements depend on frame rate, object scale, and camera calibrati

4 Sept 2026

Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

arXiv:2609.01404v1 Announce Type: cross Abstract: Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM di

2 Sept 2026

E-Commerce Bench: Evaluating LLM Agents on Long-Horizon Autonomous Business Operation

arXiv:2608.30730v1 Announce Type: cross Abstract: Long-horizon agentic tasks go beyond chaining short tasks over more interaction turns. Their evolving dynamic environments and long-range dependencies

Evaluating Constrained Iterative Refinement for Scalable Vector Graphics Generation with Off-the-Shelf VLMs

arXiv:2608.28678v1 Announce Type: new Abstract: Scalable Vector Graphics (SVGs) power much of the modern visual ecosystem, yet state-of-the-art generative models focus almost entirely on rasterized im

researcharxiv-cs-cv

LexRubric: A Rubric-Guided Diagnostic Benchmark for Open-Ended Legal Tasks

arXiv:2606.09389v2 Announce Type: replace Abstract: As large language models (LLMs) are increasingly applied to real-world legal tasks, evaluating the reliability of their open-ended legal responses h

Local Ai

Man Made Language Models? Evaluating LLMs' Perpetuation of Masculine Generics Bias

arXiv:2502.10577v2 Announce Type: replace-cross Abstract: Instruct-based large language models (LLMs) have been shown to propagate and even amplify gender bias when prompted with contextually constrai

local-aiarxiv-cs-ai

CultureConverse: A Multilingual Multi-turn Simulation Harness for Culturally Grounded Assistance in East and Southeast Asia

arXiv:2608.28405v1 Announce Type: new Abstract: Current cultural evaluations for large language models (LLMs) often reduce culture to single-turn factual recall via MCQs, failing to capture a common u

31 Aug 2026

SURE-Challenge: Evaluating Speech Evidence Before Speech-LLM Generation

arXiv:2608.27783v1 Announce Type: cross Abstract: Speech LLMs are usually graded after they answer, although an operating system first has to decide whether a waveform should be sent to the model. We

31 Aug 2026

Interpretable, Fairly Evaluated Automated L2 Speaking Assessment that Beats the Single-Human Ceiling and Why Pause Encoding Does Not Change LLM Fluency Scores

arXiv:2608.26137v1 Announce Type: new Abstract: Second-language (L2) English learners can rarely rehearse speaking with a partner. Speaking is also the most anxiety-laden skill. These gaps drive a fas

safetyarxiv-cs-cl

28 Aug 2026

Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia

arXiv:2608.23065v1 Announce Type: cross Abstract: Cultural understanding in video means more than recognizing what is visible; it requires grasping the symbolic and temporal significance of cultural c

DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue

arXiv:2606.17904v2 Announce Type: replace Abstract: Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, recent systems ground these models in pr

researcharxiv-cs-ai

Evaluating Human and LLM-Generated Thematic Analysis in HRI for Vulnerable Populations: A Comparative and Ethical Analysis

arXiv:2608.21420v1 Announce Type: new Abstract: Thematic analysis (TA) has long been regarded as an inherently human, reflexive, and interpretive process. However, the extent to which LLM-generated TA

researcharxiv-cs-ro

Evaluating Multimodal Narrative Understanding of Popular Hollywood Films

arXiv:2608.21430v1 Announce Type: new Abstract: Multimodal language models increasingly show promise for enabling the large-scale computational analysis of film, opening up new avenues for learning ab

Evaluating SAT Solver Metrics as Predictors of Human-Perceived Nonogram Difficulty

arXiv:2608.23300v1 Announce Type: cross Abstract: Algorithmic solver effort is often assumed to align with perceived puzzle difficulty, but this assumption is rarely tested against human solving data.

researcharxiv-cs-ai

KAN-Robust-Bench: A Benchmark for Evaluating the Robustness of Kolmogorov-Arnold Networks

arXiv:2608.21488v1 Announce Type: cross Abstract: While machine learning models have demonstrated strong performance in many domains, these models have shown profound vulnerabilities when they are exp