Knowledge catalogue
arXiv:2605.15185v1 Announce Type: cross Abstract: Generative video models are increasingly studied as implicit world models, yet evaluating whether they produce physically plausible 3D structure and m
arXiv:2605.12925v1 Announce Type: cross Abstract: Evaluation of software engineering (SWE) agents is dominated by a binary signal: whether the final patch passes the tests. This outcome-only view trea
arXiv:2502.15761v3 Announce Type: replace-cross Abstract: The deployment of large language models (LLMs) on extended reality (XR) devices has great potential to advance the field of human-AI interacti
arXiv:2605.13153v1 Announce Type: new Abstract: Temporal Knowledge Graph Reasoning (TKGR) aims at inferring missing (especially future) events from historical data. Current evaluation in TKGR uniforml
arXiv:2605.11601v1 Announce Type: new Abstract: Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early t
arXiv:2510.04265v4 Announce Type: replace-cross Abstract: Pass@k is widely used to report the reasoning performance of LLMs, but it often produces unstable and potentially misleading rankings, especia
arXiv:2605.11209v1 Announce Type: new Abstract: While existing benchmarks demonstrate the near-perfect performance of large language models (LLMs) on various tasks, this apparent saturation often obsc
arXiv:2605.12022v1 Announce Type: new Abstract: Large Language Models (LLMs) achieve strong performance on standard knowledge evaluation benchmarks, yet recent work shows that their knowledge capabili
arXiv:2605.11205v1 Announce Type: new Abstract: Benchmark evaluation across AI and safety-critical domains overwhelmingly relies on simple averaging. We demonstrate that this practice produces substan
arXiv:2605.08522v1 Announce Type: new Abstract: The evaluation of Large Language Models (LLMs) faces a critical challenge in construct validity, where fragmented benchmarks and ad hoc metrics frequent
arXiv:2601.18061v3 Announce Type: replace Abstract: Learning from human feedback~(LHF) assumes that expert judgments, appropriately aggregated, yield valid ground truth for training and evaluating AI
arXiv:2605.10834v1 Announce Type: new Abstract: AI pentesting agents are increasingly credible as offensive security systems, but current benchmarks still provide limited guidance on which will perfor
arXiv:2605.10627v1 Announce Type: cross Abstract: Coreference resolution is typically evaluated using aggregate statistical metrics such as CoNLL-F1, which measure structural overlap between predicted
arXiv:2605.09025v1 Announce Type: new Abstract: Federated learning enables hospitals to collaboratively train segmentation models without sharing patient data. However, current evaluation protocols re
arXiv:2605.07084v1 Announce Type: new Abstract: Automatic speech recognition (ASR) evaluation compares system output to ground truth transcripts, with Word Error Rate (WER) quantifying the distance be
arXiv:2605.07342v1 Announce Type: cross Abstract: Compile-pass rate is the dominant evaluation signal for LLM code generation, yet for multi-component domain-specific artifacts it can be actively misl
arXiv:2605.07796v1 Announce Type: new Abstract: SQL dialects vary in syntax, types, and functions across database engines. Text-to-SQL benchmarks, however, predominantly support only SQLite. This crea
arXiv:2605.07477v1 Announce Type: new Abstract: Recent text-guided image editing (TIE) models have achieved remarkable progress, however, many edited results still suffer from artifacts, unintended mo
arXiv:2601.22725v3 Announce Type: replace Abstract: Recent advances in diffusion models have significantly elevated the visual fidelity of Virtual Try-On (VTON) systems, yet reliable evaluation remain
arXiv:2604.06132v2 Announce Type: replace Abstract: Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing
arXiv:2605.02199v1 Announce Type: new Abstract: Long-term LLM agents must compress streams of past interactions into persistent memory before future queries are known. Existing evaluations usually mea
arXiv:2605.01945v1 Announce Type: new Abstract: Tandem mass spectrometry provides a high-throughput framework for identifying and quantifying proteins in complex biological samples. In computational p
arXiv:2605.02624v1 Announce Type: new Abstract: There is growing interest in exploring user simulation as an alternative to gathering and scoring real user-chatbot interactions for AI chatbot evaluati
arXiv:2605.00645v1 Announce Type: new Abstract: Clinical time-series forecasting is increasingly studied for decision support, yet standard aggregate metrics can obscure whether a model is actually us
arXiv:2508.05469v3 Announce Type: replace Abstract: We evaluate artificial intelligence (AI) systems without ground truth by exploiting a link between strategic gaming and information loss. Building o
arXiv:2507.14719v2 Announce Type: replace Abstract: As large language models (LLMs) become increasingly integrated into real-world applications, scalable and rigorous safety evaluation is essential. T
arXiv:2604.25361v1 Announce Type: new Abstract: Video generation models have developed rapidly in recent years, where generating natural human motion plays a pivotal role. However, accurately evaluati
arXiv:2604.24824v1 Announce Type: new Abstract: This article philosophically examines how shifts in assumptions regarding the existence and non-existence of the true target (TT) give rise to new persp
arXiv:2603.21362v2 Announce Type: replace Abstract: LLM-as-Judge evaluation fails agent tasks because a fixed rubric cannot capture what matters for this task: code debugging demands Correctness and E
arXiv:2604.24158v1 Announce Type: new Abstract: Evaluating nuanced conversational travel recommendations is challenging when human annotations are costly and standard metrics ignore stakeholder-centri
arXiv:2604.20898v1 Announce Type: new Abstract: Wrist function is essential in performing activities of daily living (ADLs). However, there is limited experimental evidence on the functional impact of
arXiv:2604.20967v1 Announce Type: new Abstract: Upper-limb exoskeletons (ULEs) have the potential to restore functional independence in individuals with severe motor impairments; however, the clinical
arXiv:2506.01937v2 Announce Type: replace Abstract: Reward models are used throughout the post-training of language models to capture nuanced signals from preference data and provide a training target
arXiv:2604.21211v1 Announce Type: new Abstract: Current text anonymization evaluation relies on span-based metrics that fail to capture what an adversary could actually infer, and assumes a single dat
arXiv:2604.21093v1 Announce Type: cross Abstract: We introduce TravelFraudBench (TFG), a configurable benchmark for evaluating graph neural networks (GNNs) on fraud ring detection in travel platform g
arXiv:2604.18919v1 Announce Type: new Abstract: Analyzing topics extracted from text data in relation to external outcomes is important across fields such as computational social science and organizat
arXiv:2512.15907v2 Announce Type: replace Abstract: Evaluating the quality of tables generated by large language models (LLMs) remains an open challenge: existing metrics either flatten tables into te
arXiv:2604.18227v1 Announce Type: new Abstract: Feature selection is a fundamental machine learning and data mining task, involved with discriminating redundant features from informative ones. It is a
arXiv:2510.18109v4 Announce Type: replace-cross Abstract: Evaluating the usefulness of data before purchase is essential when obtaining data for high-quality machine learning models, yet both model bu
arXiv:2604.17530v1 Announce Type: cross Abstract: Posture is a critical factor for beginning instrumental learners. Most students receive instruction only once a week, and during the intervals between
arXiv:2604.03920v2 Announce Type: replace Abstract: LLM-based social simulations can generate believable community interactions, enabling ``policy wind tunnels'' where governance interventions are tes
arXiv:2604.14513v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly used in scientific peer review, assisting with drafting, rewriting, expansion, and refinement. However, ex
arXiv:2604.14634v1 Announce Type: new Abstract: Multiple choice evaluation is widely used for benchmarking large language models, yet near ceiling accuracy in low option settings can be sustained by s
arXiv:2507.13942v2 Announce Type: replace Abstract: Forecasting future events is a fundamental capability for general-purpose systems that plan or act across different levels of abstraction. Yet, eval
arXiv:2604.10014v1 Announce Type: cross Abstract: This paper provides a comprehensive evaluation of demographic and linguistic biases in omnimodal language models that process text, images, audio, and
arXiv:2511.14774v3 Announce Type: replace-cross Abstract: Evaluating cross-lingual knowledge transfer in large language models is challenging, as correct answers in a target language may arise either
arXiv:2604.11529v1 Announce Type: new Abstract: Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (T
arXiv:2604.10511v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used for causal and counterfactual reasoning, yet their reliability in real-world policy evaluation remain
arXiv:2604.06111v2 Announce Type: replace Abstract: Existing Agent benchmarks suffer from two critical limitations: high environment interaction overhead (up to 41% of total evaluation time) and imbal
arXiv:2603.23055v2 Announce Type: replace-cross Abstract: Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, i
arXiv:2604.08294v1 Announce Type: cross Abstract: Action Quality Assessment (AQA) has broad applications in physical therapy, sports coaching, and competitive judging. Although Vision Language Models
arXiv:2604.08126v1 Announce Type: new Abstract: Objective Structured Clinical Examinations (OSCEs) are the standard method for assessing medical students' clinical and communication skills through str
arXiv:2601.22318v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly accessed as remotely hosted services by edge and enterprise clients that cannot run frontier models lo
arXiv:2610.15722v1 Announce Type: new Abstract: Multimodal large language models (MLLMs) are good at video understanding and code generation, but it remains unclear whether they can recover the physic
arXiv:2610.15294v1 Announce Type: new Abstract: Evaluating robot manipulation policies through repeated physical trials is costly. We study whether an action-conditioned world model can serve as a clo
arXiv:2610.17806v1 Announce Type: new Abstract: Collision rate remains a dominant metric for evaluating autonomous driving trajectory safety, but its binary nature obscures substantial safety differen
arXiv:2610.13806v1 Announce Type: cross Abstract: Production recommender systems have feature-lifecycle asymmetry: adding new features must produce gains on some metric, while feature removal must not
arXiv:2610.11447v1 Announce Type: cross Abstract: Large language models (LLMs) are increasingly deployed as coding agents that edit files, run builds and tests, inspect execution results, and repair s
arXiv:2610.10554v1 Announce Type: cross Abstract: 3D Gaussian Splatting has evolved from static reconstruction toward physics-integrated representations meant to predict how scenes change under intera
arXiv:2610.09250v1 Announce Type: new Abstract: Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating