Knowledge catalogue
arXiv:2503.15242v3 Announce Type: replace Abstract: We introduce BigO(Bench), a novel coding benchmark designed to evaluate the capabilities of generative language models in understanding and generati
arXiv:2609.25298v1 Announce Type: new Abstract: Cultural evaluation coverage and robustness in language models are difficult to diagnose because pretraining corpora and cultural benchmarks are rarely
arXiv:2609.26193v1 Announce Type: cross Abstract: Active sequential hypothesis testing studies how to identify an unknown hypothesis with a given set of sensing actions. We study this in the setting o
arXiv:2609.25685v1 Announce Type: new Abstract: Contour initialization and numerical stopping can jointly affect the evaluation of active-contour segmentation. We examine their interaction using the o
arXiv:2508.15526v2 Announce Type: replace Abstract: The rapid proliferation of large language models (LLMs) has intensified the requirement for reliable safety evaluation to uncover model vulnerabilit
arXiv:2609.22917v1 Announce Type: new Abstract: Non-technical stakeholders frequently cannot write the SQL needed to extract insights from operational databases. We built and evaluated a Text-to-SQL a
arXiv:2609.22894v1 Announce Type: cross Abstract: Coreset selection picks a representative subset of the labeled training set to make training cheaper. However, it is usually evaluated by downstream a
As part of our efforts to pace the frontier, we’re committed to supporting independent assessments with deep levels of access across training, evaluation, and deployment. That access should enable thi
arXiv:2609.22971v1 Announce Type: new Abstract: Evaluating user experience (UX) on live websites through user testing is expensive, subjective, and difficult to scale. LLM agents offer a promising rou
arXiv:2609.24971v1 Announce Type: new Abstract: Agents today often take real-world actions that depend on long-term memory and context recall over time. However, most current memory benchmarks are bui
arXiv:2609.24219v1 Announce Type: new Abstract: Traditionally, the reliability of news publishers is assessed by expert organisations that evaluate editorial practices, transparency and factual standa
arXiv:2609.22308v1 Announce Type: new Abstract: Coding-agent benchmarks usually evaluate implementation after the target behavior has been specified in text, code, or demonstrations. Existing research
arXiv:2609.24033v1 Announce Type: new Abstract: Reliable action evaluation in contact-rich manipulation requires looking beyond the current observation to future visual and contact consequences. Exist
arXiv:2609.24350v1 Announce Type: cross Abstract: Robotic foundation models achieve impressive performance on standard manipulation benchmarks, yet these evaluations typically assume clean, timely, an
arXiv:2609.22904v1 Announce Type: new Abstract: Triage in the emergency department (ED) is a sequential decision process that unfolds turn by turn. Existing evaluations of large language models (LLMs)
arXiv:2609.22947v1 Announce Type: new Abstract: Reinforcement learning (RL) is vital for optimizing video generation models, with a robust reward model (RM) serving as the cornerstone. However, existi
arXiv:2609.23939v1 Announce Type: new Abstract: Effective communication between users and AI agents is essential for human-AI collaboration. The XY problem is a well-known communication pitfall where
arXiv:2609.22115v1 Announce Type: new Abstract: Zeroth-order optimization (ZOO) estimates updates from function evaluations, making perturbation queries a primary cost. Fixed budgets spend the same nu
arXiv:2609.21879v1 Announce Type: cross Abstract: Vision-Language Models (VLMs) have recently been proposed as promising tools for face recognition, as they can produce natural language explanations a
arXiv:2609.21763v1 Announce Type: new Abstract: A medical model's benchmark score does not establish that the same conclusion holds under a different evaluation. This study tests whether claims about
arXiv:2609.22064v1 Announce Type: new Abstract: Advances in large-scale neural recording have made it possible to collect data across many animals and distributed brain regions, raising the question o
arXiv:2609.21686v1 Announce Type: cross Abstract: Privacy leakage in LLM agents is commonly evaluated within individual components such as memory, retrieval, or tool-use pipelines, which makes it diff
arXiv:2609.21349v1 Announce Type: cross Abstract: Large language models have shown strong potential as role-playing agents for real individuals, yet faithful impersonating remains challenging. Existin
with @Thom_Wolf in sunny italy ☀️chatting about the @huggingface @OpenAI attack, the “tell” that it was an agentic swarm, how misalignment arises in RL, whether agent awareness of evaluation is eviden
arXiv:2609.19640v1 Announce Type: new Abstract: Croissant is the de facto machine-readable descriptor for ML datasets: JSON-LD over schema.org. Since version 1.1 it also carries data use conditions, r
arXiv:2609.19827v1 Announce Type: new Abstract: With the widespread industrial deployment of Large Language Models (LLMs), DeepSearch has emerged as the dominant paradigm for resolving complex user qu
arXiv:2609.19444v1 Announce Type: new Abstract: Fine-grained evaluation of glomerular pathology must distinguish normal glomeruli from abnormalities such as global and segmental glomerulosclerosis, ob
arXiv:2609.20193v1 Announce Type: new Abstract: Retrieval plug-ins supply a deep forecaster with information its lookback window cannot carry. Published evaluations report consistent gains, and each c
arXiv:2609.18122v1 Announce Type: cross Abstract: Learning-enabled robotic manipulation increasingly relies on robot simulators for policy training and evaluation before real-world deployment. Inside
arXiv:2609.18820v1 Announce Type: new Abstract: Agentic workflows now make consequential decisions in regulated settings, and the governance placed around them is almost entirely step-scoped: input-ou
arXiv:2609.17865v1 Announce Type: new Abstract: Frontier models are often evaluated on how they respond to safety information once it is already in context. We study an earlier decision point: whether
arXiv:2609.18651v1 Announce Type: new Abstract: Generalist robot policies offer useful initialization, but refining compact specialists through limited physical interaction requires informative learni
arXiv:2609.19059v1 Announce Type: cross Abstract: Multimodal large language model (MLLM) agents are increasingly used as personal assistants for long-running tasks. Their utility depends on continuity
arXiv:2609.17763v1 Announce Type: new Abstract: Next-day wildfire prediction requires models whose forecasts can be evaluated alongside the assumptions and historical evidence used in their computatio
arXiv:2609.19088v1 Announce Type: new Abstract: Large vision-language models have achieved remarkable progress in multi-modal understanding, yet their capabilities in educational settings remain insuf
arXiv:2505.23126v5 Announce Type: replace Abstract: Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangl
arXiv:2609.17745v1 Announce Type: cross Abstract: A central goal of autonomous reinforcement learning is continuous policy training without external resets. However, existing paradigms largely depend
arXiv:2609.17956v1 Announce Type: new Abstract: Large-scale machine-translation (MT) systems are typically evaluated on random samples from a corpus whose distributional composition is an artifact of
arXiv:2609.18286v1 Announce Type: new Abstract: Chess has long served as a model domain for studying search, expertise, decision-making, and artificial intelligence. The emergence of large language mo
arXiv:2609.16592v1 Announce Type: new Abstract: This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with
arXiv:2609.16251v1 Announce Type: new Abstract: Computer-use agents are increasingly evaluated in realistic desktop environments, but existing benchmarks provide limited coverage of professional engin
arXiv:2609.17043v1 Announce Type: cross Abstract: Multi-hop question answering requires combining information from multiple documents to answer complex questions. These systems have grown increasingly
arXiv:2509.12626v4 Announce Type: replace-cross Abstract: Aligning agentic AI with user intent is critical for delegating complex, socially embedded tasks, yet user preferences are often implicit, evo
arXiv:2609.17081v1 Announce Type: new Abstract: Grounded language-model systems are often evaluated by final answer accuracy, yet a correct answer can be unsupported, drawn from the wrong source, or p
arXiv:2607.18063v2 Announce Type: replace-cross Abstract: LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. We present a 21-scenario benchmark f
arXiv:2609.15478v1 Announce Type: new Abstract: Multimodal agents can create complex videos in software such as Blender by coding without relying on diffusion models. Yet video understanding benchmark
arXiv:2609.14956v1 Announce Type: cross Abstract: Speech Quality Assessment (SQA) is essential for modern speech technologies, and recent non-intrusive SQA predictors increasingly rely on Speech Found
arXiv:2609.15209v1 Announce Type: cross Abstract: Automatic prompt optimization (APO) improves language-model programs by revising prompts from task feedback, yet it typically holds its training data
arXiv:2609.15625v1 Announce Type: cross Abstract: Driven by the escalating demand for privacy-preserving computing, Federated Learning (FL) has witnessed remarkable progress, becoming a cornerstone te
arXiv:2609.13308v1 Announce Type: new Abstract: A companion evaluation found that naming the target part in a manipulation prompt increased action accuracy by 0.32-0.63 across eight vision-language mo
arXiv:2609.13475v1 Announce Type: new Abstract: Existing pipelines for clinical timeline extraction from case reports are evaluated using an expert reference and are limited by imperfect reference ann
arXiv:2609.15743v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems, trained on paired speech-text data, have been improved by leveraging language models (LMs) trained on text-o
arXiv:2609.14992v1 Announce Type: new Abstract: Recently, the rapid development of large language models (LLMs) has reshaped software engineering by enabling autonomous code agents that plan, execute,
arXiv:2609.15695v1 Announce Type: new Abstract: Vision-language models (VLMs) increasingly power consumer-facing AI search, yet evaluating them on the diversity of everyday visual questions remains ch
arXiv:2606.17467v3 Announce Type: replace-cross Abstract: Prompt injection defenses evaluated on synthetic benchmarks do not generalize to real enterprise documents, which are longer, denser, and inte
arXiv:2605.06165v2 Announce Type: replace Abstract: As the widespread adoption of Large Language Models (LLMs) accelerates, token consumption from intermediate reasoning traces increasingly contribute
arXiv:2502.20099v3 Announce Type: replace-cross Abstract: We evaluate methods for causal representation learning (CRL) on a simple, real-world system that satisfies the basic problem setup of CRL. The
arXiv:2609.15396v1 Announce Type: new Abstract: LLM-based agents increasingly rely on persistent skills, i.e., reusable procedural prompts, to adapt without weight updates. Existing skill self-evoluti
arXiv:2609.13838v1 Announce Type: new Abstract: Purpose: To develop and evaluate TotalSynth, a reusable pretrained framework for whole-body synthetic CT (sCT) generation from MRI and cone-beam CT (CBC
arXiv:2609.14762v1 Announce Type: cross Abstract: Cloud-hosted large language models (LLMs) are increasingly used for root cause analysis (RCA) in AIOps pipelines, but they introduce data privacy risk