Search: “evaluation”

arXiv:2605.03858v1 Announce Type: new Abstract: Multi-constraint instruction following requires verifying whether a response satisfies multiple individual requirements, yet LLM judges are often assess

arXiv:2605.01783v1 Announce Type: new Abstract: Procedural Content Generation (PCG) enables game content to be created algorithmically without direct manual level-design effort, but it introduces a se

arXiv:2605.00873v1 Announce Type: cross Abstract: The rapid advancement of photorealistic Text-to-Video (T2V) generation brings in an urgent need for up-to-date evaluation methods. Existing benchmarks

arXiv:2603.01865v3 Announce Type: replace Abstract: LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be avera

arXiv:2604.27542v1 Announce Type: new Abstract: Conventionally, Automatic Speech Recognition (ASR) systems are evaluated on their ability to correctly recognize each word contained in a speech signal.

arXiv:2604.26460v1 Announce Type: new Abstract: Stylistic personalization - making LLMs write in a specific individual's style, rather than merely adapting to task preferences - lacks evaluation groun

arXiv:2602.11224v3 Announce Type: replace-cross Abstract: We present Agent-Diff, a novel benchmarking framework for evaluating agentic Large Language Models (LLMs) on real-world productivity software

arXiv:2604.25119v1 Announce Type: new Abstract: Auditing the fine-tunes of open-weight generative models for harmful specialization has become a new governance challenge for model hosting platforms. T

arXiv:2604.24018v1 Announce Type: new Abstract: This paper studies the problem of robot performance evaluation, focusing on how to obtain accurate and efficient estimates of real-world behavior under

arXiv:2604.24001v1 Announce Type: new Abstract: The evaluation of generated reports remains a critical challenge in Computed Tomography (CT) report generation, due to the large volume of text, the div

arXiv:2604.23178v1 Announce Type: new Abstract: LLM-as-a-Judge has become the dominant paradigm for evaluating language model outputs, yet LLM judges exhibit systematic biases that compromise evaluati

arXiv:2604.23099v1 Announce Type: cross Abstract: Evaluating generative AI models is increasingly resource-intensive due to slow inference, expensive raters, and a rapidly growing landscape of models

arXiv:2604.24052v1 Announce Type: cross Abstract: Video-to-text summarization remains underexplored in terms of comprehensive evaluation methods. Traditional n-gram overlap-based metrics and recent la

arXiv:2603.04603v2 Announce Type: replace-cross Abstract: We present a risk-aware formalism for evaluating system trajectories in the presence of uncertain interactions between the system and its envi

arXiv:2604.23102v1 Announce Type: new Abstract: Standard evaluations of Bayesian deep learning methods assume that metric estimates are reliable, but we show this assumption fails under data scarcity.

arXiv:2604.22774v1 Announce Type: cross Abstract: Accurate transcription of handwritten mathematics is crucial for educational AI systems, yet current benchmarks fail to evaluate this capability prope

arXiv:2604.22119v1 Announce Type: new Abstract: As reasoning capacity and deployment scope grow in tandem, large language models (LLMs) gain the capacity to engage in behaviors that serve their own ob

arXiv:2510.15313v2 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly applied to creative domains, yet their performance in classical Chinese poetry generation and evaluati

arXiv:2604.21090v1 Announce Type: cross Abstract: AI governance programmes increasingly rely on natural language prompts to constrain and direct AI agent behaviour. These prompts function as executabl

arXiv:2503.16416v2 Announce Type: replace Abstract: LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environ

Today we open sourced many of OpenAI's monitorability evaluations. We hope that the research community and other model developers can build upon them and use them to evaluate the monitorability of the

arXiv:2604.21769v1 Announce Type: new Abstract: LLM leaderboards are widely used to compare models and guide deployment decisions. However, leaderboard rankings are shaped by evaluation priorities set

arXiv:2604.19281v1 Announce Type: cross Abstract: The use of Large Language Models (LLMs) to support patients in addressing medical questions is becoming increasingly prevalent. However, most of the m

arXiv:2601.13729v2 Announce Type: replace Abstract: In recent years, the non-deterministic properties of language models have garnered considerable attention and have shown a significant influence on

arXiv:2505.22176v3 Announce Type: replace Abstract: Evaluating tables qualitatively and quantitatively poses a significant challenge, as standard metrics often overlook subtle structural and content-l

arXiv:2604.02368v4 Announce Type: replace Abstract: As Large Language Models (LLMs) exhibit plateauing performance on conventional benchmarks, a pivotal challenge persists: evaluating their proficienc

arXiv:2510.09275v2 Announce Type: replace Abstract: Medical diagnostics is a high-stakes and complex domain that is critical to patient care. However, current evaluations of large language models (LLM

arXiv:2512.02636v3 Announce Type: replace-cross Abstract: Log-likelihood evaluation enables important capabilities in generative models, including model comparison, certain fine-tuning objectives, and

arXiv:2602.05523v2 Announce Type: replace-cross Abstract: Agentic large language models (LLMs) are increasingly evaluated on cybersecurity tasks using capture-the-flag (CTF) benchmarks, yet existing p

arXiv:2604.16009v1 Announce Type: new Abstract: Metacognition, the ability to monitor and regulate one's own reasoning, remains under-evaluated in AI benchmarking. We introduce MEDLEY-BENCH, a benchma

arXiv:2511.10262v3 Announce Type: replace-cross Abstract: Full-Duplex Speech Language Models (FD-SLMs) enable real-time, overlapping conversational interactions, offering a more dynamic user experienc

arXiv:2604.13413v1 Announce Type: new Abstract: Diffusion language models (DLMs) have emerged as a promising paradigm for large language models (LLMs), yet the non-deterministic behavior of DLMs remai

arXiv:2507.22359v4 Announce Type: replace Abstract: Although large language models (LLMs) have shown exceptional capabilities across a wide range of tasks, reliable evaluation remains a critical chall

arXiv:2604.09706v1 Announce Type: cross Abstract: Recent AI media detectors report near-perfect performance under clean laboratory evaluation, yet their robustness under realistic deployment condition

arXiv:2604.10127v1 Announce Type: cross Abstract: The rapid advancement of AIGC-based video generation has underscored the critical need for comprehensive evaluation frameworks that go beyond traditio

arXiv:2604.07775v1 Announce Type: cross Abstract: Collaboration and information sharing empower Multi-Agent Systems (MAS) but also introduce a critical security risk known as Agent Cascading Injection

arXiv:2604.07429v1 Announce Type: new Abstract: Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse f

arXiv:2604.08499v1 Announce Type: cross Abstract: Prompt injection attacks pose serious security risks across a wide range of real-world applications. While receiving increasing attention, the communi

arXiv:2510.26241v5 Announce Type: replace-cross Abstract: Modern vision-language models (VLMs) excel at many multimodal tasks, yet their grasp of temporal information in video remains weak and has not

arXiv:2610.21881v1 Announce Type: new Abstract: Training language models for formal theorem proving requires large numbers of verifier calls. Slow checks and timeouts often leave GPUs waiting and with

I evaluate new models by giving them actual work tasks that have already been completed by a trusted model and then comparing the results. My typical workflow is to plan tasks/requests in Gemini and c

arXiv:2610.20690v1 Announce Type: cross Abstract: The present work is concerned with how to measure, at scale, the multi-dimensional impact of generative AI models on the individuals who use them. In

arXiv:2610.15714v1 Announce Type: new Abstract: Healthcare agents must transform longitudinal clinical evidence into reliable actions through heterogeneous tools and stateful workflows, conditions und

arXiv:2610.17016v1 Announce Type: new Abstract: Evaluating model uncertainty on real-world data is challenging because each instance provides only a single observed label, leaving true conditional cla

arXiv:2610.17908v1 Announce Type: cross Abstract: In manga translation, visual context outside speech-bubble text, including speaker identities, referents, and scene tone, is depicted in page images,

arXiv:2610.17986v1 Announce Type: new Abstract: A common practice in instance retrieval is to grow the gallery size to improve a benchmark's discrimination capability. One prominent example is ILIAS,