Knowledge catalogue
arXiv:2609.28761v1 Announce Type: cross Abstract: We design, implement, and evaluate KathDB-FAO, a new query evaluation subsystem for our KathDB multimodal DBMS. KathDB-FAO takes as input a query in n
arXiv:2603.10287v2 Announce Type: replace-cross Abstract: LLM-as-a-Judge is a flexible framework for text evaluation, which allows us to obtain scores for the quality of a given text from various pers
arXiv:2609.28605v1 Announce Type: cross Abstract: The Factivity Inference Evaluation 2026 (FIE2026) classifies Chinese context-hypothesis pairs into nine ordered factivity intervals. Its evaluation me
arXiv:2609.25689v1 Announce Type: new Abstract: Recent advances in learning-based robot policies have demonstrated promising progress, yet they are predom- inantly evaluated in static or quasi-static
arXiv:2609.22793v1 Announce Type: new Abstract: LLM-based machine translation evaluation can closely match human judgments, but in practice it remains largely diagnostic, with the signals rarely trans
arXiv:2609.24308v1 Announce Type: new Abstract: Evaluating world models requires assessing both the quality of the worlds they generate and their consistency and responsiveness under exploration, inte
arXiv:2609.23010v1 Announce Type: new Abstract: Iterative text-to-motion generation delivers high-quality and semantically aligned motions but requires multiple network evaluations, resulting in subst
arXiv:2609.21267v1 Announce Type: new Abstract: Production LLM agents are evaluated repeatedly as they evolve, but full agent benchmarks are costly to rerun. We study efficient recurring evaluation fo
On September 21 2026, Yohei reported results from a hard “insect‑identification” benchmark where model accuracy spanned 23 points across the evaluated systems. The Qwen 4B + Glance model scored 0.96,
arXiv:2609.20124v1 Announce Type: cross Abstract: Evaluating live streaming speech synthesis (TTS) requires assessing fine-grained, highly expressive prosody such as emotion, intonation, and energy wh
arXiv:2609.17985v1 Announce Type: new Abstract: AI agents are usually evaluated by whether they complete a task. In interactive service settings, a successful agent can still frustrate users by asking
arXiv:2609.19072v1 Announce Type: new Abstract: Large language models are increasingly used for content moderation, but most evaluations still report aggregate accuracy on individual benchmarks. We in
arXiv:2609.18525v1 Announce Type: new Abstract: Modern evaluation of learning pipelines often reduces to downstream accuracy, leaving open the question of why tasks succeed or fail. TriProbe addresses
arXiv:2609.16023v1 Announce Type: new Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale
arXiv:2609.17112v1 Announce Type: new Abstract: Large video models have exhibited impressive performance on a wide range of visual question answering tasks, owing to the rise of powerful, pretrained t
arXiv:2609.16878v1 Announce Type: cross Abstract: Despite its crucial role in video object removal (VOR), existing evaluation paradigms face two critical limitations: questionable references and a mis
arXiv:2609.13168v1 Announce Type: cross Abstract: Speech AI, any AI system that recognizes, transforms, or generates speech, is built and evaluated across two communities with only a small overlap: te
arXiv:2609.14739v1 Announce Type: cross Abstract: Large language models are increasingly used in high-stakes domains such as law, where systems must ground their reasoning in retrieved evidence and ab
arXiv:2609.13454v1 Announce Type: cross Abstract: Clinical decisions are prospective, but clinical language models are often evaluated on retrospective records that reveal the final diagnosis, treatme
arXiv:2609.15640v1 Announce Type: cross Abstract: Long-text image--text congruence scoring is increasingly important for vision-language systems that must evaluate whether detailed textual description
arXiv:2609.13637v1 Announce Type: new Abstract: Persistent agents need evaluations that distinguish identity facts they can recall from those they express and enact. We introduce PAI-Bench, a provider
arXiv:2609.15794v1 Announce Type: new Abstract: Clinical AI systems are evaluated with instruments built for research settings (reference-based similarity metrics and expert rubric panels) that measur
arXiv:2609.14057v1 Announce Type: new Abstract: Frontier LLMs are increasingly capable of conducting automated research, yet their creativity in this setting has not been systematically evaluated. In
arXiv:2609.15939v1 Announce Type: cross Abstract: Language-model agents increasingly operate over complete software repositories, yet cybersecurity evaluations primarily measure whether they can detec
arXiv:2609.12524v1 Announce Type: cross Abstract: We study the strategic design of a two-echelon spare-parts inventory network where evaluating each candidate topology requires an expensive inventory
arXiv:2609.11897v1 Announce Type: new Abstract: Causal discovery aims to uncover causal structures from data and is fundamental to scientific reasoning and intervention-based decision making. Its eval
arXiv:2601.08536v3 Announce Type: replace Abstract: Deep Research Agents (DRA) aim to help users search the web, synthesize information, and deliver comprehensive investigative reports. Prior benchmar
arXiv:2609.09678v1 Announce Type: new Abstract: Clinical diagnosis agents must decide not only what test to request next, but also when to diagnose or defer. Existing agent benchmarks largely evaluate
arXiv:2609.07075v1 Announce Type: new Abstract: Retrieval-augmented generation (RAG) is commonly evaluated by whether the final answer is correct. That test is insufficient: an answer can match its re
arXiv:2405.02228v4 Announce Type: replace-cross Abstract: Large language models (LLMs) increasingly generate citation-backed responses, yet citation hallucination remains a major challenge for trustwo
arXiv:2609.07153v1 Announce Type: cross Abstract: Minimal-pair benchmarks such as BLiMP evaluate linguistic knowledge by testing whether language models (LMs) prefer acceptable sentences over minimall
arXiv:2609.07309v1 Announce Type: cross Abstract: Robustness evaluation of large language models (LLMs) remains a critical challenge, particularly in assessing their sensitivity to perturbations in in
arXiv:2609.09250v1 Announce Type: cross Abstract: A verifier for robot policies reads a candidate behavior and returns a score for how well it did, used both to evaluate vision-language-action policie
arXiv:2609.07601v1 Announce Type: cross Abstract: The application of large language models (LLMs) to personalized medical assistants has garnered growing interest. However, existing medical benchmarks
arXiv:2609.08322v1 Announce Type: cross Abstract: Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is r
arXiv:2603.27611v2 Announce Type: replace Abstract: When a cognitive system modifies its own functioning, what exactly does it modify: a low-level rule, a control rule, or the criterion that evaluates
arXiv:2609.06535v1 Announce Type: new Abstract: We introduce 3DHarnessBench, a benchmark that evaluates the agentic ability of frontier vision-language models (VLMs) to recover 3D geometry as Blender
arXiv:2609.07596v1 Announce Type: cross Abstract: Vision-language models are increasingly used in settings where some input modalities may be unavailable, yet we know little about whether they can fai
arXiv:2609.06207v1 Announce Type: new Abstract: Image-to-video generators are often credited with absorbing physical dynamics as implicit world models, a claim the community currently checks with plau
arXiv:2508.11431v3 Announce Type: replace Abstract: An object can disappear from a 3D scene, yet still be detectable. Even after visual removal, modern vision models may infer what was originally pres
arXiv:2609.06232v1 Announce Type: new Abstract: Ground-truth defect masks in industrial inspection datasets are typically reserved for evaluation. This paper repurposes them as spatial supervision sig
arXiv:2609.05009v1 Announce Type: new Abstract: Safety evaluations can mischaracterize deployed behaviour if artificial-intelligence systems respond to being evaluated. We test this possibility in a f
arXiv:2609.04173v1 Announce Type: new Abstract: For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases.
arXiv:2608.29632v1 Announce Type: cross Abstract: Competitive programming is increasingly being used to evaluate the algorithmic reasoning capabilities of large language models (LLMs). However, existi
arXiv:2609.00918v1 Announce Type: new Abstract: Large language models are increasingly used as interactive recommender assistants. Their evaluation should therefore go beyond plausible item recommenda
arXiv:2608.29463v1 Announce Type: cross Abstract: A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status.
arXiv:2608.30568v1 Announce Type: cross Abstract: Background: Population level assessments of predictive artificial intelligence (AI) can conceal performance disparities across subgroups. Fairness eva
arXiv:2608.29711v1 Announce Type: new Abstract: While MLLMs have made significant strides in chart comprehension and video understanding, current evaluations largely isolate these capabilities, leavin
arXiv:2602.00685v2 Announce Type: replace Abstract: Large language models (LLMs) are increasingly used as simulated participants in social science experiments, but their behavior is often unstable and
arXiv:2608.29677v1 Announce Type: cross Abstract: Despite rapid advances in MIS, fair and reproducible comparisons of segmentation models remain challenging due to heterogeneous datasets, inconsistent
arXiv:2608.29278v1 Announce Type: new Abstract: Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readi
arXiv:2508.02208v3 Announce Type: replace-cross Abstract: Evaluating the mathematical capability of Large Language Models (LLMs) is a critical yet challenging frontier. Existing benchmarks fall short,
arXiv:2608.28639v1 Announce Type: new Abstract: Formal theorem proving with large language models remains challenging due to the difficulty of navigating large proof search spaces efficiently. Existin
arXiv:2603.10807v2 Announce Type: replace-cross Abstract: Existing LLM safety evaluations rely on binary attack-success rates and domain-agnostic taxonomies, leaving regulated Banking, Financial Servi
arXiv:2608.26138v1 Announce Type: new Abstract: We introduce the Cross-Platform Fairness Evaluation (CPFE) framework -- a five-axis audit protocol covering discriminative performance, calibration, sta
arXiv:2608.26529v1 Announce Type: new Abstract: In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core i
arXiv:2608.25824v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as evaluators to assess output quality and preference alignment, yet providing reliable guarantees of
arXiv:2608.24665v1 Announce Type: new Abstract: Power outage prediction models are increasingly used in assessments of climate-driven infrastructure risk, yet current evaluation practices obscure whet
arXiv:2608.23956v1 Announce Type: new Abstract: Test-time reasoning methods such as iterative refinement, decomposition, and repeated sampling are often evaluated in isolation, making their gains diff
arXiv:2608.21369v1 Announce Type: cross Abstract: Nigerian Pidgin is one of Africa's most widely spoken languages, yet remains severely underrepresented in language model evaluation. Existing benchmar