Knowledge catalogue
arXiv:2609.23201v1 Announce Type: new Abstract: Benchmark scores increasingly influence the development, marketing, and selection of large language models (LLMs). Yet an overall score is interpretable
arXiv:2602.18307v2 Announce Type: replace-cross Abstract: Large language models have achieved striking results in interactive theorem proving, particularly in Lean. However, most benchmarks for LLM-ba
arXiv:2609.24124v1 Announce Type: new Abstract: Active perception and manipulation are crucial for robots to interact with complex scenes. Existing benchmarks struggle to evaluate how robots effective
arXiv:2609.23490v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly execute multi-step workflows through tool use and interaction with users and environments. However, curre
arXiv:2609.23067v1 Announce Type: new Abstract: Referring surgical video instrument segmentation (RSVIS) aims at segmenting the instrument in a surgical video, given a textual description. Despite rec
arXiv:2609.23003v1 Announce Type: new Abstract: We present M3GA-Wild, the first benchmark for multi-modal, multi-session ground-to-aerial place recognition in forests. M3GA-Wild unifies and extends ex
arXiv:2609.22220v1 Announce Type: new Abstract: Benchmarks for LLM-generated GPU kernels decide correctness with a few random inputs and a loose floating-point tolerance, and their verdicts now feed l
arXiv:2609.23055v1 Announce Type: new Abstract: Discrete diffusion models now match autoregressive language models on several benchmarks, while the question of how best to train them has received far
arXiv:2609.24613v1 Announce Type: cross Abstract: Prior case retrieval (PCR) aims to identify precedent cases relevant to a given query case. Existing PCR benchmarks and methods predominantly operate
arXiv:2609.22038v1 Announce Type: new Abstract: We introduce QuranicMMLU, a benchmark for evaluating generative AI on Quranic Arabic across multiple dimensions of linguistic complexity. Existing Quran
arXiv:2508.06734v3 Announce Type: replace-cross Abstract: While graph-based Android malware classifiers report strong benchmark accuracy of over 94%, their performance sharply decreases up to 45% when
arXiv:2609.19154v1 Announce Type: new Abstract: While Large Language Models (LLMs) achieve high accuracy on established Classical Chinese Poetry benchmarks, it remains challenging to distinguish trans
We just released the first version of our Creative Writing benchmark, comparing 24 LLMs against human writers across 475 writing prompts. Creative writing is subjective, so the rankings aren't meant t
arXiv:2603.28662v2 Announce Type: replace-cross Abstract: Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn c
arXiv:2603.03884v2 Announce Type: replace-cross Abstract: Topic localization aims to identify spans of text that express a given topic defined by a name and description. To study this task, we introdu
arXiv:2603.09908v2 Announce Type: replace Abstract: Existing aerial-robotics benchmarks target vehicles from hundreds of grams to several kilograms and typically expose only high-level state data. The
arXiv:2505.23126v5 Announce Type: replace Abstract: Although many benchmarks evaluate the reasoning abilities of Large Language Models (LLMs) within domains such as mathematics, coding, or data wrangl
arXiv:2609.18396v1 Announce Type: cross Abstract: Virtual sensing, the estimation of hard-to-measure quantities from available sensor measurements, is a critical enabler for control and monitoring in
arXiv:2609.17985v1 Announce Type: new Abstract: AI agents are usually evaluated by whether they complete a task. In interactive service settings, a successful agent can still frustrate users by asking
arXiv:2609.18949v1 Announce Type: cross Abstract: We introduce StableEval Arena, a cost-aware benchmark framework for evaluating agentic AI systems on stablecoin peg-risk prediction. StableEval Arena
arXiv:2601.18493v2 Announce Type: replace Abstract: Vision--language models (VLMs) show promise for disaster-response remote sensing, but existing benchmarks mainly emphasize scene-level or damage-cen
arXiv:2603.12264v2 Announce Type: replace Abstract: Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natura
arXiv:2603.14456v2 Announce Type: replace Abstract: Persian poses unique audio understanding challenges through its classical poetry, traditional music, and pervasive code-switching, none of which is
TensorRT Edge‑LLM ran Qwen 3.6‑27B on a single NVIDIA Jetson AGX Thor and completed the MLPerf Edge Agentic benchmark in 24 min 36 s—52.33 tokens/s—which is 6.4× faster than the llama.cpp reference (2
arXiv:2609.16486v1 Announce Type: new Abstract: Rapid advancements in vision-language models have propelled Referring Remote Sensing Image Segmentation (RRSIS) to the forefront of Earth observation. H
EDIT: Sorry for the unclear title. This model is UkisAI's Swift-Qwen3.8-27B, not a new version of BottleCap AI's 3.6-ThinkingCap. All credit goes to UkisAI for making great fine-tune, and I made this
arXiv:2512.24565v4 Announce Type: replace Abstract: Large Language Models (LLMs) are increasingly serving as autonomous agents, and their utilization of external tools via the Model Context Protocol (
arXiv:2609.14473v1 Announce Type: new Abstract: Personal AI assistants hold the potential to evolve from digital interfaces into embodied companions capable of guiding users through complex physical a
Some food for thought when designing benchmarks... So, here's a little computer-use (visual) comparison between GPT-5.6 Astra and Qwen3.8 Max. The task here was to recreate the image in the center usi
arXiv:2609.12350v1 Announce Type: new Abstract: Close-range first-person field images are important for mobile maize phenotyping because many plant-level traits depend on in-canopy structures that are
arXiv:2609.13003v1 Announce Type: new Abstract: Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-l
arXiv:2609.12884v1 Announce Type: new Abstract: A medical claim's correctness often depends not on the claim alone, but on the clinical structure around it. A claim may require a lab reference range,
arXiv:2512.14691v3 Announce Type: replace Abstract: Modern multimodal generative models can synthesize visually compelling images and videos, but it remains unclear whether this visual fluency reflect
arXiv:2609.12345v1 Announce Type: new Abstract: Existing agent benchmarks mainly evaluate final task success or tool-call correctness, providing limited insight into whether agents can reliably diagno
A few people here mentioned interest in a way to test their custom Pi setups, so I figured I’d drop this here: RoastMyHarness The basic idea is a small engine that sets up an environment to run DeepSW
arXiv:2609.11234v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used in peer review at major AI conferences, yet novelty remains a persistent weak point. Existing benchma
arXiv:2609.10962v1 Announce Type: cross Abstract: Studies of the Model Context Protocol (MCP) server ecosystem draw their samples in ways that quietly select for servers that work: reference sets, pop
arXiv:2609.11242v1 Announce Type: new Abstract: Video generation has advanced to produce visually compelling and temporally coherent results. Yet, whether these models can genuinely think with video--
arXiv:2603.26934v2 Announce Type: replace Abstract: Recent advances in photorealistic avatar generation have enabled highly realistic talking-head avatars, raising security concerns regarding identity
arXiv:2609.09754v1 Announce Type: new Abstract: As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning e
arXiv:2609.10469v1 Announce Type: new Abstract: Automated plant disease diagnosis is increasingly deployed on farmer-held devices in regions where agronomic expertise is scarce and network connectivit
arXiv:2609.09410v1 Announce Type: new Abstract: While autonomous agents have made significant strides in 'deep research' by iteratively navigating the open web to synthesize information, real-world pr
arXiv:2609.08275v1 Announce Type: new Abstract: Recent multi-shot audio-video generators can produce increasingly coherent and cinematic outputs, but coherence does not imply the ability to execute ed
arXiv:2609.05947v1 Announce Type: new Abstract: Peer review is central to quality control in science. However, existing evaluations of AI-assisted peer review mainly focus on the overall quality of ge
arXiv:2609.07022v1 Announce Type: cross Abstract: Reasoning over privacy-constrained records requires combining structured attributes with evidence from free-text narratives. We introduce CIPHER (Cros
arXiv:2609.07603v1 Announce Type: new Abstract: Financial scenarios are diverse and complex, spanning varying data conditions, tool configurations, and workflows. Yet existing CUA, Computer-Using Agen
arXiv:2609.06831v1 Announce Type: new Abstract: AI systems are used worldwide, but they struggle to serve the needs of culturally diverse populations. Prior work on cultural understanding evaluates AI
arXiv:2609.06149v1 Announce Type: cross Abstract: Script-based malware remains a prevalent attack technique. These scripts often contain indicators of compromise (IOCs) that provide actionable threat
arXiv:2609.08226v1 Announce Type: new Abstract: Temporal graph learning models the evolution of dynamic systems, where both structural interactions and semantic states change over time. However, exist
arXiv:2609.05418v1 Announce Type: cross Abstract: Machine learning training workloads place unique demands on storage systems, yet most existing benchmarks focus on computational throughput rather tha
arXiv:2609.06928v1 Announce Type: new Abstract: Emotion analysis is a fundamental task in computer vision, but its practical deployment remains constrained by the privacy risks inherent to conventiona
arXiv:2609.07288v1 Announce Type: new Abstract: Large Language Models (LLMs) are increasingly employed to orchestrate robot behavior through natural-language interfaces, yet no benchmark exists to eva
arXiv:2609.08399v1 Announce Type: new Abstract: Universal machine-learning interatomic potentials (u-MLIPs) aim to generalize across diverse configurations. Benchmarks enable reproducible evaluation b
arXiv:2609.08947v1 Announce Type: cross Abstract: Graph-based surrogate models offer a promising route to accelerate computational fluid dynamics (CFD) simulations on unstructured meshes. However, the
arXiv:2511.22341v2 Announce Type: replace Abstract: Previous works identify sensitivity to option order as a key issue in multiple-choice VQA (MC-VQA) evaluation and propose protocols to mitigate this
arXiv:2609.04641v1 Announce Type: new Abstract: Natural-language-to-SQL systems have ad- vanced rapidly on academic benchmarks, yet production enterprise schemas exhibit graph- like, semi-structured,
Really strong benchmark paper on coding agents. Claude Opus 5 running under Claude Code passes 23.9% of the evaluations. An expert human reference scores 82.2%. Here is what the task actually is. The
arXiv:2609.05405v1 Announce Type: new Abstract: Recent advances in wearable sensing enable continuous monitoring of physiological and behavioral signals, yet existing benchmarks rarely evaluate whethe
arXiv:2609.03654v1 Announce Type: cross Abstract: The comparative analysis of banks' financial statements poses significant challenges for automated question answering systems due to their complexity,
arXiv:2609.03553v1 Announce Type: new Abstract: Policy analysis requires more than predicting whether a proposal will pass: it requires identifying who will be affected, how those actors respond, and