Search: “evaluation”

arXiv:2604.16009v1 Announce Type: new Abstract: Metacognition, the ability to monitor and regulate one's own reasoning, remains under-evaluated in AI benchmarking. We introduce MEDLEY-BENCH, a benchma

arXiv:2511.10262v3 Announce Type: replace-cross Abstract: Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experienc

arXiv:2604.13413v1 Announce Type: new Abstract: Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remai

arXiv:2507.22359v4 Announce Type: replace Abstract: Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical chall

arXiv:2604.09706v1 Announce Type: cross Abstract: Recent AI media detectors report near-perfect performance under clean laboratory evaluation, yet their robustness under realistic deployment condition

arXiv:2604.10127v1 Announce Type: cross Abstract: The rapid advancement of AIGC-based video generation has underscored the critical need for comprehensive evaluation frameworks that go beyond traditio

arXiv:2604.07775v1 Announce Type: cross Abstract: Collaboration and information sharing empower Multi-Agent Systems (MAS) but also introduce a critical security risk known as Agent Cascading Injection

arXiv:2604.07429v1 Announce Type: new Abstract: Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse f

arXiv:2604.08499v1 Announce Type: cross Abstract: Prompt injection attacks pose serious security risks across a wide range of real-world applications. While receiving increasing attention, the communi

arXiv:2510.26241v5 Announce Type: replace-cross Abstract: Modern vision-language models (VLMs) excel at many multimodal tasks, yet their grasp of temporal information in video remains weak and has not

arXiv:2610.21881v1 Announce Type: new Abstract: Training language models for formal theorem proving requires large numbers of verifier calls. Slow checks and timeouts often leave GPUs waiting and with

arXiv:2610.20690v1 Announce Type: cross Abstract: The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In

arXiv:2610.15714v1 Announce Type: new Abstract: Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions und

arXiv:2610.17016v1 Announce Type: new Abstract: Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional cla

arXiv:2610.17908v1 Announce Type: cross Abstract: In manga translation, visual context outside speech-bubble text, including speaker identities, referents, and scene tone, is depicted in page images,

arXiv:2610.17986v1 Announce Type: new Abstract: A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability. One prominent example is ILIAS,

arXiv:2610.10848v1 Announce Type: new Abstract: Accurate evaluation is central to selecting which LLM to deploy, yet testing a candidate on live traffic exposes real users to an unvetted model. Teams

arXiv:2610.11129v1 Announce Type: new Abstract: Game commentary is an open-ended generation task requiring multimodal perception, strategic reasoning, and contextual knowledge. Existing AI-Generated G

arXiv:2610.11736v1 Announce Type: new Abstract: Modern video generators can realize increasingly complex visual narratives, positioning the prompt enhancer (PE) as a critical bridge from concise user

arXiv:2610.09529v1 Announce Type: new Abstract: There are more than 2,000 listed companies on the UK's London Stock Exchange, divided into 11 sectors who are required to communicate their financial re

arXiv:2610.10506v1 Announce Type: cross Abstract: Many of the questions now put to large language models have no correct answer to score against: what a policy is worth, which option a user should cho

arXiv:2610.08303v1 Announce Type: new Abstract: Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures

arXiv:2610.03939v1 Announce Type: new Abstract: Many inference workloads evaluate a trained tree ensemble on row groups that share feature values: discrete-time survival models expand each patient int

arXiv:2610.05533v1 Announce Type: new Abstract: Harness search keeps a change to the prompts, reasoning switches, token budgets or parsers around a frozen model if the change raises a score. Such a ga

arXiv:2610.02775v1 Announce Type: new Abstract: Mental health stigma has profoundly harmful impacts but its complexity makes it difficult to evaluate. Stigma may involve explicit derogation, but also

arXiv:2609.39072v1 Announce Type: cross Abstract: Emotion recognition in conversation has been widely studied, but applying Large Language Models (LLMs) to continuous dimensional emotion evaluation in

arXiv:2609.39429v1 Announce Type: cross Abstract: Uncertainty Quantification (UQ) is a key requirement for trustworthy AI in high-stakes medical image analysis. In this work, we evaluate UQ in a multi

arXiv:2609.36987v1 Announce Type: new Abstract: Graph databases are increasingly queried through natural language, yet every existing benchmark evaluates isolated single-turn queries rather than the m

arXiv:2609.37294v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly used for software engineering tasks, yet their stochastic behavior challenges the validity, reproducibil

arXiv:2609.36316v1 Announce Type: cross Abstract: Evaluation awareness (EA) can cause large language models (LLMs) to behave differently during audits than in deployment, yet measuring and accounting

arXiv:2609.31759v1 Announce Type: cross Abstract: Domain Generalization (DG) aims to develop models that remain robust to conditions unseen during training. While DG has been systematically studied in

arXiv:2609.31692v1 Announce Type: new Abstract: Explainable Artificial Intelligence (XAI) has become essential in medical image analysis to ensure transparency of deep learning (DL)-based diagnostic s

arXiv:2609.32743v1 Announce Type: new Abstract: Electroencephalography (EEG) foundation models (FMs) promise transferable neural representations, yet their advantages over strong supervised baselines

arXiv:2609.31696v1 Announce Type: cross Abstract: Offline evaluation is the dominant experimental paradigm in recommender systems research, enabling reproducible and cost-effective comparisons on hist

arXiv:2609.31655v1 Announce Type: new Abstract: Generative priors reconstruct accelerated 3D MRI well but pay heavily at deployment: tens of network evaluations per volume, and protocol-specific hyper

arXiv:2601.04301v3 Announce Type: replace Abstract: As frontier AI systems are pretrained on web-scale data, test set contamination has become a critical concern for accurately assessing their capabil

arXiv:2609.33872v1 Announce Type: cross Abstract: Robotic manipulation policies are advancing rapidly with increasing reliance on vision-language models for end-to-end decision making. However, reliab

arXiv:2609.33761v1 Announce Type: new Abstract: Generative models for multivariate weather series are routinely evaluated with pooled distributional metrics computed after marginal calibration. We sho

arXiv:2609.31473v1 Announce Type: new Abstract: We introduce Kaggle Game Arena, an open and ever-expanding platform to evaluate large language models (LLMs) through competitive games. Different from s

arXiv:2609.30571v1 Announce Type: new Abstract: Language models are often evaluated on curated benchmarks that underrepresent the complexity of enterprise deployments. We introduce HARDEN, a constrain

arXiv:2609.31082v1 Announce Type: new Abstract: Knowing where species occur is fundamental for biodiversity research and conservation. Species distribution models (SDMs) link species observations to e

arXiv:2609.30074v1 Announce Type: cross Abstract: Evaluations of LLM systems routinely average over small prompt sets and report models as a ranked table. We ask how much confidence such a table deser

arXiv:2609.29140v1 Announce Type: new Abstract: Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requ

arXiv:2609.27139v1 Announce Type: cross Abstract: Surgical procedures follow a phase-to-step hierarchy, yet the video-language models used to recognize them are evaluated with flat per-level metrics t

arXiv:2609.26853v1 Announce Type: cross Abstract: While Large Language Models (LLMs) have achieved remarkable results across various benchmarks, their alignment with normative values often results in

arXiv:2609.27867v1 Announce Type: new Abstract: A forecasting benchmark reports which method won. We show that the answer is set by the evaluator's choices before any model is fitted. We benchmark 24