Knowledge catalogue
arXiv:2604.07788v1 Announce Type: cross Abstract: We introduce PeReGrINE, a benchmark and evaluation framework for personalized review generation grounded in graph-structured user--item evidence. PeRe
arXiv:2610.21767v1 Announce Type: new Abstract: Can AI agents organize scientific investigations that lead to well-supported conclusions? To evaluate this capability, we introduce PhysClosureBench, a
arXiv:2610.18449v1 Announce Type: new Abstract: AI agents for autonomous scientific discovery must decide which experiments to run under a limited budget. Most benchmarks for science agents provide fi
arXiv:2610.20472v1 Announce Type: new Abstract: Cultural attitudes and social norms evolve over time, yet most cultural benchmarks for large language models (LLMs) treat culture as a static attribute
arXiv:2610.14539v1 Announce Type: new Abstract: Real-world business requests often leave key requirements unstated because frontline practitioners struggle to articulate rules they know from experienc
arXiv:2610.18745v1 Announce Type: cross Abstract: Large language models increasingly generate Solidity smart contracts that, once deployed, can never be patched, making a single latent fault permanent
arXiv:2610.17456v1 Announce Type: cross Abstract: Language models are often evaluated alone but increasingly deployed together, routed between, made to debate, ensembled, or collaboratively trained. F
arXiv:2610.18455v1 Announce Type: new Abstract: Rapid advances in artificial intelligence-generated content (AIGC) have significantly promoted using text-to-image (T2I) models to generate images with
arXiv:2610.17354v1 Announce Type: new Abstract: Current evaluation protocols for medical image segmentation remain largely anchored in sample-wise comparisons, e.g., Dice and Hausdorff distance, which
arXiv:2610.16347v1 Announce Type: new Abstract: Forward-Forward learning replaces end-to-end backpropagation with local, layer-wise training objectives. Its variants are evaluated primarily by classif
arXiv:2610.13021v1 Announce Type: new Abstract: Continual learning requires models to accumulate past experience and transfer it to new tasks. Despite substantial research on continual learning, exist
arXiv:2610.12360v1 Announce Type: new Abstract: When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusio
arXiv:2610.08794v1 Announce Type: new Abstract: Large language models rely on subword tokenizers whose quality varies across languages, yet no standardized multi-metric framework exists for broad comp
arXiv:2610.09360v1 Announce Type: cross Abstract: Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such
arXiv:2610.07345v1 Announce Type: cross Abstract: IAM policy analysis typically emphasizes the authorization capabilities encoded in a policy, but security analyst review priority may also depend on t
For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality. Le Chonk ranked #1 among open-weight m
arXiv:2610.02438v1 Announce Type: cross Abstract: Large language models (LLMs) have demonstrated strong performance in code generation, where success depends on both recalling relevant algorithmic kno
arXiv:2610.02432v1 Announce Type: cross Abstract: Large language models (LLMs) in production systems face prompt injections, trojans (backdoors), and manipulation of automatic quality metrics. This th
arXiv:2610.02399v1 Announce Type: new Abstract: Multimodal agents are increasingly used for data visualization tasks but remain limited in autonomous review. Unlike humans, they may fail to recognize
arXiv:2505.13439v2 Announce Type: replace-cross Abstract: Autoregressive (AR) models have recently shown strong performance in image generation, where a critical component is the visual tokenizer (VT)
arXiv:2610.01938v1 Announce Type: cross Abstract: Exam-style accuracy does not establish whether large language models (LLMs) reason well over clinical records. We define clinical reasoning as integra
arXiv:2601.11354v2 Announce Type: replace Abstract: Recent LLM-for-Space systems address mission planning, scheduling, operations support, simulator control, and autonomy, but their evaluations use di
arXiv:2610.00416v1 Announce Type: new Abstract: Evaluating large language models becomes increasingly challenging as their capabilities advance: benchmarks can saturate, public test sets risk contamin
arXiv:2610.01092v1 Announce Type: new Abstract: Video generation models are increasingly being explored as world simulators for embodied planning and learning. To do so effectively, these models must
arXiv:2610.01581v1 Announce Type: new Abstract: Generative AI models are increasingly used for scenario generation in autonomous driving. While they can generate realistic-looking scenarios, they ofte
arXiv:2610.01506v1 Announce Type: new Abstract: As agents evolve from single-tool systems into modular, composite architectures, skills are becoming an important mechanism for capability development a
arXiv:2610.00084v1 Announce Type: new Abstract: Detailed profession-specific system prompts raise token use and estimated cost per response without a consistent accuracy gain. We evaluate Scientific A
arXiv:2609.38406v1 Announce Type: new Abstract: Access to real-world information is often noisy and fragmented. Constructing a coherent narrative from such fragments requires models to reconstruct mis
arXiv:2609.39147v1 Announce Type: new Abstract: Current image-to-video models achieve visual realism and physical plausibility, but reasoning about mental states remains unexplored. Actions are driven
arXiv:2609.38612v1 Announce Type: new Abstract: As natural language drives more applications, language models increasingly run inside programs as decision components: the program sends them the curren
arXiv:2609.39386v1 Announce Type: new Abstract: Pretrained time-series foundation models (TSFMs) are evaluated as forecasters of future values, yet for sparse series many decisions depend only on whic
arXiv:2609.37788v1 Announce Type: cross Abstract: Rubrics support the structured evaluation of language models. We propose a rubric for assessing expressed clinical reasoning in model responses, drawi
arXiv:2609.35581v1 Announce Type: cross Abstract: We introduce QC-Stark, a benchmark for evaluating large language models (LLMs) on 11 quantum computing (QC) tasks, spanning circuit construction, debu
arXiv:2602.09163v2 Announce Type: replace Abstract: Scientific knowledge bases accelerate discovery by curating findings from primary literature into structured, queryable formats for both human resea
arXiv:2609.21562v1 Announce Type: cross Abstract: Coding agents can modify and test code across large software projects. Game development is a domain where agents must implement gameplay rules. A game
arXiv:2609.17842v1 Announce Type: cross Abstract: Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ende
arXiv:2609.16006v1 Announce Type: cross Abstract: Large language models (LLMs) increasingly serve users whose expectations are shaped by their cultural context, yet most cultural evaluations test what
arXiv:2609.16610v1 Announce Type: new Abstract: Zero-shot waypoint navigation requires vision-language models to select, from the current first-person observation, a sequence of spatial actions that i
arXiv:2609.13280v1 Announce Type: new Abstract: Prescription transcription errors are not interchangeable. A model that fabricates a drug and a model that misreads a legible dose pose very different c
arXiv:2609.13948v1 Announce Type: cross Abstract: A central tenet of human cognition is systematicity, the principle that understanding one concept is inherently tied to understanding close variations
Daniel Kokotajlo / @dkokotajlo: OpenAI researcher: top models are becoming so situationally aware humans “are losing the ability to evaluate them” while humans rely more on AI to lead research — Dan S
Dario Amodei / @darioamodei: Dario Amodei says Anthropic is “unilaterally committing” to giving third-party evaluators permanent access to verify its adherence to safety measures — We Must Pace the Fr
arXiv:2609.11872v1 Announce Type: cross Abstract: Continuous glucose monitoring (CGM) provides high-frequency measurements of glucose dynamics and enables short-term glucose forecasting for diabetes m
arXiv:2609.11134v1 Announce Type: new Abstract: Most Deepfake detectors report near-perfect AUC scores on their reference benchmarks. However, a recent ICML position paper argues that these evaluation
arXiv:2609.09560v1 Announce Type: cross Abstract: This study evaluates Vibe Coding, an emerging AI-led conversational programming paradigm that enables developers to generate software through natural-
arXiv:2601.13591v3 Announce Type: replace Abstract: Recent LLM-based data agents aim to automate data science tasks ranging from data analysis to deep learning. However, the open-ended nature of real-
arXiv:2609.09973v1 Announce Type: new Abstract: Evaluating video captioning remains a critical challenge for Visual Large Language Models (VLLMs). Existing metrics primarily rely on matching generated
arXiv:2609.09038v1 Announce Type: new Abstract: Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric cr
We're introducing Q2D-Web (Query2Doc-Web), a benchmark and public leaderboard for evaluating retrieval in agentic RAG systems. Q2D-Web tests how embedding models perform on large-scale web search usin
arXiv:2609.05090v1 Announce Type: new Abstract: Evaluation of medical artificial intelligence agents remains predominantly answer-centric, assessing only the correctness of final outputs while overloo
arXiv:2507.17049v4 Announce Type: replace-cross Abstract: Vision-Language-Action (VLA)-enabled robots integrate visual perception, natural language understanding, and action planning to interpret thei
arXiv:2603.11915v2 Announce Type: replace Abstract: Theory of Mind (ToM)-the ability to reason about the mental states of oneself and others-is a cornerstone of human social intelligence. As Multimoda
arXiv:2609.00549v1 Announce Type: new Abstract: Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps.
arXiv:2608.30333v1 Announce Type: cross Abstract: Next-basket repurchase recommendation is commonly formulated as a ranking task: given a customer's purchase history, the system ranks previously purch
arXiv:2608.28776v1 Announce Type: new Abstract: Multilingual sentence embeddings are increasingly used to estimate semantic similarity across languages, yet their sensitivity to fine-grained translati
arXiv:2608.30690v1 Announce Type: new Abstract: Visual SLAM is commonly evaluated on clean trajectories, although deployment failures are often caused by adverse weather, illumination, blur, and senso
arXiv:2608.28932v1 Announce Type: new Abstract: Voice products increasingly need affective cues that are present in speech but absent from transcripts. We introduce VocalAffectBench, a public, test-on
arXiv:2608.28916v1 Announce Type: new Abstract: Automatic speech recognition (ASR) systems are commonly evaluated with word error rate (WER), yet many voice workflows depend on exact written values fo
arXiv:2608.28080v1 Announce Type: new Abstract: Image-conditioned 3D generation has advanced rapidly, yet existing evaluation protocols largely judge rendered-view plausibility and semantic alignment,
arXiv:2608.26180v1 Announce Type: cross Abstract: Shopping assistants are shifting from ranked product lists toward structured decision support, where systems must synthesize shopper context, product