Knowledge catalogue
arXiv:2605.14517v1 Announce Type: cross Abstract: Holistic evaluation scores capture overall output quality but do not distinguish whether a model reproduced the structural form of a user's request fr
arXiv:2502.00270v3 Announce Type: replace-cross Abstract: The performance of an LLM depends heavily on the relevance of its training data to the downstream evaluation task. However, in practice, the d
arXiv:2605.14865v1 Announce Type: new Abstract: AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, an
arXiv:2605.09544v1 Announce Type: new Abstract: Tool-integrated reasoning has emerged as a promising paradigm for enhancing large language models with external computation, retrieval, and execution ca
arXiv:2605.09936v1 Announce Type: new Abstract: We present Urban-ImageNet, a large-scale multi-modal dataset and evaluation benchmark for urban space perception from user-generated social media imager
arXiv:2510.00436v2 Announce Type: replace Abstract: Automated approaches to answer patient-posed health questions are rising, but selecting among systems requires reliable evaluation. The current gold
arXiv:2604.03675v2 Announce Type: replace Abstract: Agentic search enables language models to solve knowledge-intensive tasks by adaptively acquiring external evidence over multiple steps. Reinforceme
arXiv:2605.07446v1 Announce Type: new Abstract: We report the construction of a Korean evaluation-annotated corpus, hereafter called 'Evaluation Annotated Dataset (EVAD)', and its use in Aspect-Based
arXiv:2605.04410v1 Announce Type: new Abstract: The evaluation of explainable AI (XAI) methods is affected by a lack of standardization. Metrics are inconsistently defined, incompletely reported, and
arXiv:2605.04135v1 Announce Type: cross Abstract: Readers of applied-domain LLM capability evaluations want to know what AI systems can currently do. That literature answers a related, but consequenti
Top skill to learn today: AI Agent Evaluation. Anyone can build AI agents now but the difference is in the quality that's only possible via proper evals. Wrote some thoughts on evaluating production A
arXiv:2605.04972v1 Announce Type: new Abstract: Aligning large language models with expert judgment is especially difficult in subjective evaluation tasks, where experts may disagree, rely on tacit cr
arXiv:2605.02050v1 Announce Type: cross Abstract: This work establishes a foundational framework for standardizing AI evaluation RCTs (sometimes called human uplift studies). Drawing on established ex
arXiv:2605.02122v1 Announce Type: new Abstract: Human evaluation remains the primary standard for assessing modern AI systems, yet annotator disagreement, bias, and variability make system rankings fr
arXiv:2605.00227v1 Announce Type: new Abstract: There are growing concerns about the risks posed by AI companion applications designed for emotional engagement. Existing safety evaluations often rely
arXiv:2604.26645v1 Announce Type: new Abstract: AI-for-Science (AI4Science) is increasingly transforming scientific discovery by embedding machine learning models into prediction, simulation, and hypo
arXiv:2604.25580v1 Announce Type: new Abstract: The closure of Perspective API at the end of 2026 discards what has functioned as the de facto standard for automated toxicity measurement in NLP, CSS,
arXiv:2601.21225v2 Announce Type: replace Abstract: Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagge
arXiv:2604.25152v1 Announce Type: cross Abstract: We present MGTEVAL, an extensible platform for systematic evaluation of Machine-Generated Text (MGT) detectors. Despite rapid progress in MGT detectio
arXiv:2602.14914v2 Announce Type: replace Abstract: Off-policy evaluation (OPE) is essential for assessing ranking and recommendation systems without costly online interventions. Self-Normalised Inver
arXiv:2604.23581v1 Announce Type: cross Abstract: Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices l
arXiv:2604.23949v1 Announce Type: new Abstract: Medical and public health experts must make real-time resource decisions, such as expanding hospital bed capacity, based on projected hospitalization tr
arXiv:2604.24525v1 Announce Type: cross Abstract: Automated code review (ACR) bots are increasingly used in industrial software development to assist developers during pull request (PR) review. As ado
arXiv:2604.22152v1 Announce Type: new Abstract: Evaluating robotics policies across thousands of environments and thousands of tasks is infeasible with existing approaches. This motivates the need for
arXiv:2502.17011v2 Announce Type: replace-cross Abstract: Financial bond yield forecasting is challenging due to data scarcity, nonlinear macroeconomic dependencies, and evolving market conditions. In
arXiv:2604.22597v1 Announce Type: new Abstract: Recent advancements in large language models have led to significant improvements across various tasks, including mathematical reasoning, which is used
arXiv:2604.22662v1 Announce Type: cross Abstract: Shapley values are a cornerstone of explainable AI, yet their proliferation into competing formulations has created a fragmented landscape with little
arXiv:2604.20869v1 Announce Type: cross Abstract: Background: More than 80% of U.S. cancer care is delivered in community settings, where survival remains worse than at academic centers. Clinicians mu
arXiv:2604.21481v1 Announce Type: new Abstract: Crowdsourced pairwise evaluation has emerged as a scalable approach for assessing foundation models. However, applying it to Text to Speech(TTS) introdu
arXiv:2604.20273v1 Announce Type: new Abstract: We present ActuBench, a multi-agent LLM pipeline for the automated generation and evaluation of advanced actuarial assessment items aligned with the Int
arXiv:2604.19261v1 Announce Type: new Abstract: The evaluation of narrative quality remains a complex challenge, as it involves subjective factors such as plot, character development, and emotional im
arXiv:2602.11173v2 Announce Type: replace Abstract: Author response (rebuttal) writing is a critical stage of scientific peer review that demands substantial author effort. In practice, authors posses
arXiv:2604.10981v1 Announce Type: new Abstract: ATANT v1.0 (arXiv:2604.06710) defined continuity as a system property with 7 required properties and introduced a 10-checkpoint, LLM-free evaluation met
arXiv:2504.14988v3 Announce Type: replace Abstract: Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal perception capabilities, garnering significant a
arXiv:2604.11246v1 Announce Type: new Abstract: Evaluating the quality of model responses remains challenging in generative tasks with long-form answers, as the expected answers usually contain multip
arXiv:2508.03306v4 Announce Type: replace-cross Abstract: Lowering the numerical precision of model parameters and computations is widely adopted to improve the efficiency of retrieval systems. Howeve
arXiv:2604.10784v1 Announce Type: new Abstract: Recent advances in unified multimodal models (UMMs) have led to a proliferation of architectures capable of understanding, generating, and editing acros
arXiv:2508.16165v2 Announce Type: replace-cross Abstract: Usability evaluation is an essential method to support the design of effective and intuitive user interfaces (UIs). However, it commonly relie
arXiv:2604.02022v2 Announce Type: replace Abstract: Evaluating the safety of LLM-based agents is increasingly important because risks in realistic deployments often emerge over multi-step interactions
arXiv:2512.22416v2 Announce Type: replace Abstract: Hallucinations in Large Language Models (LLMs) pose a significant challenge, generating misleading or unverifiable content that undermines trust and
arXiv:2412.20718v2 Announce Type: replace Abstract: The rapid integration of Large Vision-Language Models (LVLMs) into critical domains necessitates comprehensive moral evaluation to ensure their alig
arXiv:2610.12458v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are rapidly evolving toward continuous audio--visual reasoning, creating an urgent need for evaluations that ex
arXiv:2610.11669v1 Announce Type: new Abstract: Holistic and semantics-aware co-speech gesture generation has advanced rapidly, yet evaluation remains behind: objective metrics do not consistently ref
arXiv:2610.11766v1 Announce Type: cross Abstract: Safety evaluations of large language models commonly summarize harmful-output behavior with attack success rate (ASR). Yet the same non-harmful outcom
arXiv:2610.10368v1 Announce Type: cross Abstract: Adaptive computation aims to improve language-model inference by tailoring execution to each input. For layer programs, oracle evaluations use known a
arXiv:2610.09569v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for emotional support, raising concern that sustained use may draw users away from their real-world r
arXiv:2610.06854v1 Announce Type: cross Abstract: Synthetic data generation is critical for privacy compliance, machine learning augmentation, and software testing. While single-table evaluation is we
arXiv:2610.07722v1 Announce Type: new Abstract: Activation steering provides a lightweight and flexible way to control large language model (LLM) behavior. However, effective steering requires more th
arXiv:2610.07362v1 Announce Type: new Abstract: We evaluate large language models (LLMs) in multi-turn interactions through their time-to-event: the number of interaction steps required to produce an
arXiv:2610.08719v1 Announce Type: new Abstract: Reusing the scores that select a Best-of-N winner can overstate its expected reward. We study evaluation from a fixed matrix of K independent scores per
arXiv:2610.07346v1 Announce Type: new Abstract: Repeated evaluation and differentiation of Signal Temporal Logic (STL) robustness can become a computational bottleneck in robot planning and control. S
arXiv:2610.08677v1 Announce Type: new Abstract: Off-policy evaluation (OPE) for contextual bandit policies becomes challenging when action-level importance weighting incurs excessive variance. Doubly
arXiv:2610.03193v1 Announce Type: new Abstract: The application of machine learning to dermatology has grown substantially in recent years, moving beyond proof-of-concept studies toward potential appl
arXiv:2610.02959v1 Announce Type: new Abstract: Recent text-to-image models have made substantial progress in photorealism, aesthetics, and text-image alignment. Yet visually appealing images can stil
arXiv:2609.38377v1 Announce Type: new Abstract: Evaluating the physical consistency of generated videos remains a fundamental challenge. Existing approaches rely on off-the-shelf vision-language model
arXiv:2609.38463v1 Announce Type: cross Abstract: Autonomous driving planners are typically evaluated using aggregate metrics such as driving score, destination rate, and collision rate, which do not
arXiv:2609.37771v1 Announce Type: new Abstract: Simulated manipulation benchmarks are the standard tool for evaluating vision-language-action (VLA) policies and the acceleration methods that reduce th
arXiv:2609.37322v1 Announce Type: new Abstract: Rubric-based evaluation is widely used to assess LLM-based systems by decomposing response quality into task-specific scoring criteria. However, automat
arXiv:2609.36417v1 Announce Type: new Abstract: Relational in-context learning (ICL) uses labeled support examples and their linked relational context to predict labels for new queries. This creates a
arXiv:2609.35034v1 Announce Type: cross Abstract: Off-policy evaluation, which estimates evaluation policy performance from logged data, is key for recommender ranking policies. However, logged clicks