Search: “evaluation”

arXiv:2606.02909v1 Announce Type: cross Abstract: Gradient observations can substantially improve Gaussian process (GP) surrogates, particularly in high-dimensional settings where function evaluations

local-aiarxiv-cs-lg

3 Jun 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

The Reliability Gap in Benchmark Auditing: Distribution Shift and Scale as Failure Modes of Contamination Detection

DGX agent

arXiv:2606.03305v1 Announce Type: new Abstract: Benchmark contamination, where evaluation examples appear in a model's training data, threatens the validity of LLM assessment. Statistical tools for de

model-releasesarxiv-cs-ai

3 Jun 2026

X-RAY: Mapping LLM Reasoning Capability via Formalized and Calibrated Probes

arXiv:2603.05290v2 Announce Type: replace Abstract: Large language models (LLMs) achieve promising performance, yet their ability to reason remains poorly understood. Existing evaluations largely emph

researcharxiv-cs-ai

3 Jun 2026

Automated Essay Scoring and Language Certification: Assessing Generalizability, Agreement and Validity for French

arXiv:2606.02009v1 Announce Type: new Abstract: In Automated Essay Scoring (AES), benchmarking practices have fostered minimalist evaluation practices, in contrast with the broader-view recommendation

safetyarxiv-cs-cl

Bridging Requirements and Architecture: Multi-Agent Orchestration with External Knowledge and Hierarchical Memory

arXiv:2606.01385v1 Announce Type: cross Abstract: Software architecture design is a critical yet inherently complex and knowledge-intensive phase that requires balancing competing quality attributes a

FATE-VLA:Failue-aware test generation for vision-language-action models

arXiv:2606.02307v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models are increasingly used as generalist robot policies, yet their evaluation still relies largely on static benchmarks t

researcharxiv-cs-ro

Knowledge-Intensive Video Generation

arXiv:2606.01285v1 Announce Type: cross Abstract: Text-to-video generation has advanced rapidly in visual quality, but remains under-evaluated for factuality and practical usefulness. We introduce kno

Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling

arXiv:2606.02578v1 Announce Type: cross Abstract: Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a

OARelatedWork: A Large-Scale Dataset of Related Work Sections with Full-texts from Open Access Sources

arXiv:2405.01930v2 Announce Type: replace Abstract: This paper introduces OARelatedWork: a dataset for related work generation from open-access sources. It is the first large-scale multi-document summ

PolySpeech-100: A Large-Scale Benchmark for Speech Understanding Across 100+ Languages and Dialects

arXiv:2606.01016v1 Announce Type: cross Abstract: While End-to-End (E2E) Speech-Large Language Models (Speech-LLMs) are rapidly evolving, their evaluation methodologies remain limited to the era of si

Quantifying the Salience of Geo-Cultural Values for Pluralistic Safety Alignment

arXiv:2606.00369v1 Announce Type: cross Abstract: Safe global deployment of AI models requires alignment with human values that vary across cultures. Yet rater pools in safety evaluation datasets rema

safetyarxiv-cs-lg

Tempora: Characterising the Time-Contingent Utility of Online Test-Time Adaptation

arXiv:2602.06136v2 Announce Type: replace-cross Abstract: Test-time adaptation (TTA) offers a compelling remedy for machine learning (ML) models that degrade under domain shifts, improving generalisat

researcharxiv-cs-cv

Tutorials

Tractable Shapley Values and Interactions via Tensor Networks

arXiv:2510.22138v3 Announce Type: replace Abstract: We show how to replace the O(2^n) coalition enumeration over n features behind Shapley values and Shapley-style interaction indices with a few-evalu

tutorialsarxiv-cs-lg

WAXAL-NET: Finetuned Edge ASR Across 19 African Languages

arXiv:2606.02375v1 Announce Type: new Abstract: We evaluate whether compact domain-specialized ASR models can outperform massively multilingual foundation models for conversational African speech acro

researcharxiv-cs-cl

Bounded Behavioral Indistinguishability for Black-Box LLM Distillation

arXiv:2605.30448v1 Announce Type: cross Abstract: Black-box LLM distillation is usually evaluated as an output-matching problem: a student is considered successful when its responses are semantically

Chem-PerturBridge: a harmonized compendium of small molecule perturbation transcriptomic effects

arXiv:2605.31522v1 Announce Type: new Abstract: Large perturbation models require training data encompassing chemical, cellular, and assay diversity. Current transcriptomic resources for small-molecul

researcharxiv-cs-lg

Local Ai

GPU Forecasters: Language Models as Selective Surrogates for Kernel Runtime Optimization

arXiv:2605.31464v1 Announce Type: cross Abstract: GPU kernels are the workhorse of modern deep learning, and optimizing them (via evolutionary search or coding agents) usually requires repeated measur

local-aiarxiv-cs-ai

Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models

arXiv:2508.08204v2 Announce Type: replace-cross Abstract: There has been much recent interest in evaluating large language models for uncertainty calibration to facilitate model control and modulate u

SCOPE: Selective Conformal Optimized Pairwise LLM Judging

arXiv:2602.13110v3 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and bias

The Illusion of Generalization in Tabular Language Models

model-releasesarxiv-cs-lg

arXiv:2602.04031v2 Announce Type: replace Abstract: Tabular Language Models (TLMs) have been claimed to achieve strong generalization for tabular prediction. We conduct a systematic re-evaluation of T

TUX: Measuring Human--AI Tacit Understanding

arXiv:2605.30930v1 Announce Type: cross Abstract: As large language models (LLMs) increasingly act as collaborative partners, human--AI alignment is often evaluated through explicit task success, accu

CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving

arXiv:2605.29935v1 Announce Type: cross Abstract: Autonomous driving systems are commonly trained and evaluated within limited geographic regions, which hinders their scalability when deployed in new

Agents

Estimating the Empowerment of Language Model Agents

arXiv:2509.22504v3 Announce Type: replace Abstract: As language model (LM) agents become increasingly capable and adopted in real-world applications, there is a growing need for scalable evaluation fr

agentsarxiv-cs-ai

MedCase-Structured: A Text-to-FHIR Dataset for Benchmarking Diagnostic Reasoning in Clinically Realistic EHR Settings

arXiv:2605.30295v1 Announce Type: cross Abstract: Large language models (LLMs) show promise for clinical reasoning and decision support, but evaluation in realistic, electronic health record-congruent

researcharxiv-cs-ai

Agents

SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?

arXiv:2605.30104v1 Announce Type: new Abstract: Widely used language-model benchmarks are increasingly saturated, with frontier systems often receiving near-tied scores that standard metrics cannot re

agentsarxiv-cs-cl

Who can we trust? LLM-as-a-jury for Comparative Assessment

arXiv:2602.16610v2 Announce Type: replace-cross Abstract: Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise

Local Ai

A Paired Testing Protocol for Batch-Conditioned Refusal Robustness in LLM Serving

arXiv:2605.27763v1 Announce Type: new Abstract: Safety evaluations of language models often treat serving configuration as fixed background infrastructure, but batch condition is an untested treatment

local-aiarxiv-cs-lg

Adaptive Reservoir Computing for Multi-Scenario Chaotic System Forecasting

arXiv:2605.28145v1 Announce Type: new Abstract: We present an adaptive reservoir computing framework for the CTF-4-Science Lorenz benchmark, which evaluates machine learning models across twelve disti

ATLAS: All-round Testing of Long-context Abilities across Scales

arXiv:2605.28079v1 Announce Type: new Abstract: Long-context language models now advertise context windows up to millions of tokens, yet evaluations typically report a single length or a narrow task f

Bayesian Deployment Approval for Learned Landing Controllers under Finite Rollout Validation

arXiv:2605.27720v1 Announce Type: new Abstract: Reinforcement learning and data-driven autonomous controllers are commonly evaluated using cumulative reward and empirical success frequency under finit

safetyarxiv-cs-lg

Bayesian Optimization Parameter Tuning Framework for a Lyapunov Based Path Following Controller

model-releasesarxiv-cs-ro

arXiv:2512.12649v2 Announce Type: replace Abstract: Parameter tuning in real-world experiments is constrained by the limited evaluation budget available on hardware. The path-following controller stud

ChildEval: When large language models meet children's personalities

arXiv:2605.27805v1 Announce Type: cross Abstract: While LLMs enable personalized chatbots, their effectiveness in child-centered personalization remains unclear, as systematic evaluation of child-spec

Colosseum V2: Benchmarking Generalization for Vision Language Action Models

model-releasesarxiv-cs-ro

arXiv:2605.27759v1 Announce Type: new Abstract: Vision-Language-Action (VLA) models demonstrate promising generalization in robotic manipulation, driven by advances in large-scale vision and language

HRBench: Benchmarking and Understanding Thinking-Mode Switch Strategies in Hybrid-Reasoning LLMs

arXiv:2605.28398v1 Announce Type: new Abstract: Hybrid-reasoning large language models (LLMs) expose explicit controls over reasoning effort, allowing users or systems to trade off answer quality agai

researcharxiv-cs-ai

OR-Space: A Full-Lifecycle Workspace Benchmark for Industrial Optimization Agents

arXiv:2605.28158v1 Announce Type: new Abstract: Large language model (LLM) agents are increasingly used to assist with operations research (OR) modeling, yet existing OR-oriented benchmarks often redu

PetroBench: A Benchmark for Large Language Models in Petroleum Engineering

arXiv:2605.28032v1 Announce Type: new Abstract: Large Language Models are increasingly applied in the petroleum industry, highlighting the need for a domain-specific evaluation framework. This study d

Refining Multidimensional Video Reward Models via Disentangled Influence Functions

arXiv:2605.28203v1 Announce Type: new Abstract: As Text-to-Video (T2V) generation models continue to evolve, the complexity of video evaluation necessitates a fine-grained assessment across various ax

safetyarxiv-cs-lg

Applications

ResearchLoop: An Evidence-Gated Control Plane for AI-Assisted Research

arXiv:2605.28282v1 Announce Type: new Abstract: AI-assisted research compresses ideation, implementation, evaluation, and manuscript writing into a single interactive loop. This compression is useful,

applicationsarxiv-cs-ai

The Abstraction Gap in Vision-Language Causal Reasoning

arXiv:2605.28779v1 Announce Type: new Abstract: Vision-language models (VLMs) generate fluent causal explanations, but current evaluations cannot distinguish linguistic plausibility from faithful caus

BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

arXiv:2603.03194v2 Announce Type: replace Abstract: Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software en

FAB-Bench: A Framework for Adaptive RAG Benchmarking in Semiconductor Manufacturing

arXiv:2605.26476v1 Announce Type: new Abstract: Retrieval-Augmented Generation (RAG) has become critical for knowledge-intensive applications, yet evaluating its performance in vertical domains remain

Self-Improvement Imitation with Biologically Guided Search for Protein Design Under Oracle Budgets

arXiv:2605.26690v1 Announce Type: cross Abstract: Protein sequence optimization under tight oracle budgets requires methods that explore vast combinatorial spaces while making each evaluation informat

VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

arXiv:2605.26144v1 Announce Type: cross Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents. Unlike

QUIET: A Multi-Blank Cascaded Story Cloze Benchmark for LLM Creative Generation Capability

arXiv:2605.25955v1 Announce Type: cross Abstract: Large language models (LLMs) face a dual challenge in creative capability evaluation: existing benchmarks (e.g., Story Cloze Test, HellaSwag) measure

SimuWoB: Simulating Real-World Mobile Apps for Fast and Faithful GUI Agent Benchmarking

arXiv:2605.25160v1 Announce Type: new Abstract: Mobile GUI agents powered by large language models have progressed rapidly, creating urgent needs for realistic and comprehensive evaluation. Existing b

Stop Comparing LLM Agents Without Disclosing the Harness

arXiv:2605.23950v1 Announce Type: new Abstract: This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely

The Age of Curiosity Meets the Age of AI: Benchmarking Child Safety in Large Language Models

arXiv:2605.25510v1 Announce Type: new Abstract: Children increasingly have access to Large Language Models (LLMs), which may expose them to responses that are developmentally inappropriate or require

The LSCD Benchmark: a Testbed for Diachronic Word Meaning Tasks

arXiv:2404.00176v3 Announce Type: replace Abstract: Lexical Semantic Change Detection (LSCD) is a complex, lemma-level task, which is usually operationalized based on two subsequently applied usage-le