Knowledge catalogue
Hey r/LocalLLaMA, We’ve released our full ShapeLearn GGUFs for Qwen 3.8 27B. Blog / Download models TL;DR 3.84 bpw (GPU-5) reaches 99.63% of BF16’s aggregate score of 8 benchmarks, being the most accu
arXiv:2609.06535v1 Announce Type: new Abstract: We introduce 3DHarnessBench, a benchmark that evaluates the agentic ability of frontier vision-language models (VLMs) to recover 3D geometry as Blender
arXiv:2609.06721v1 Announce Type: new Abstract: AI companions are envisioned as always-on assistants that support users in daily life. With this regard, we introduce BuddyVQA, a benchmark for companio
arXiv:2609.03261v1 Announce Type: cross Abstract: A benchmark score credits final answers, but not the route by which an item can be answered. In medical multimodal multiple-choice questions (MCQs), t
arXiv:2609.04167v1 Announce Type: cross Abstract: Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measur
If you are thinking what model will fit best your HW specs and tasks you are doing here is one table with all currently popular models that still can be considered as local. LLM Test Scores Feature De
arXiv:2608.29210v1 Announce Type: new Abstract: Text-to-Image (T2I) models have recently achieved impressive visual fidelity, yet their evaluation remains constrained by benchmarks that are often diff
arXiv:2608.29420v1 Announce Type: new Abstract: Frontier-model leaderboards now rank systems based on economic benchmarks, tests of how well models carry out professional tasks from software engineeri
arXiv:2608.05086v1 Announce Type: new Abstract: Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to tr
arXiv:2601.15307v2 Announce Type: replace-cross Abstract: The rapid development of automated survey generation technology has made it increasingly important to establish a comprehensive benchmark to e
Join the ARC Prize team -- help us build ARC-AGI-4 and ARC-AGI-5 Platform Engineer - Benchmark Lead ARC Prize Foundation is hiring a senior engineer to build our benchmark platform * Expand ARC-AGI-3
arXiv:2610.17841v1 Announce Type: cross Abstract: Evaluating fashion retrieval systems remains challenging due to the limited availability of benchmarks that capture the fine-grained, open-ended queri
Source: whatmodel.sh, an open source AI leaderboard I built (and launched here yesterday). Every point comes from published benchmark results: 22,113 of them across 987 benchmarks, from 1,011 sources,
arXiv:2610.18499v1 Announce Type: new Abstract: Egocentric video understanding spans diverse capabilities, modalities, and evaluation formats, yet these are often studied in isolation despite their ti
arXiv:2610.16152v1 Announce Type: new Abstract: Although post-training significantly enhances model performance, the use of massive undisclosed data raises concerns about benchmark contamination and c
'TypeSafe AI sells Jev as a frontier-class reasoner that cannot hallucinate, built by the co-inventor of ChatGPT - fast, and almost free. We ran it live on 16,379 benchmark requests, measured its late
arXiv:2609.39909v1 Announce Type: cross Abstract: We introduce DoGBENCH (Documentation Generation Benchmark), to our knowledge, the first benchmark for generating and maintaining real user-facing soft
arXiv:2609.39490v1 Announce Type: cross Abstract: Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and
arXiv:2609.39588v1 Announce Type: new Abstract: We push the frontier of large-scale spatial intelligence in Vision-Language Models (VLMs) and introduce the first benchmark that probes geographical lay
arXiv:2609.38480v1 Announce Type: cross Abstract: Most clinical benchmarks evaluate language models (LMs) on diagnosis using complete case descriptions. In clinical practice, however, patients present
arXiv:2510.27065v2 Announce Type: replace Abstract: We present MLPerf Automotive, the first standardized public performance benchmark for evaluating Machine Learning systems that are deployed for AI a
arXiv:2609.09898v1 Announce Type: new Abstract: Large Language Models (LLMs) have shown promise for translating Natural Language (NL) planning descriptions into PDDL problem instances. However, standa
arXiv:2606.21844v2 Announce Type: replace Abstract: As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing B
arXiv:2609.08196v1 Announce Type: new Abstract: This report analyzes Qiushi Engine v0.8 across all 40 test tasks in AstaBench E2E-Bench-Hard, a benchmark that requires autonomous agents to carry a res
arXiv:2609.09895v1 Announce Type: new Abstract: Video-language models and video agents can produce hallucinations that conflict with spatiotemporal evidence. Existing benchmarks mainly evaluate model
arXiv:2609.09396v1 Announce Type: new Abstract: As Vision-Language Models (VLMs) advance toward physical deployment, the focus has remained on action-oriented Embodied AI evaluated on subject-centric
arXiv:2608.23898v1 Announce Type: new Abstract: We introduce BenchBench-Protocol, a benchmark for large language models of 149 protocol-modification tasks recovered from modifications that scientists
arXiv:2608.18280v1 Announce Type: cross Abstract: Background. Advances in agentic systems are simultaneously, and rapidly, saturating benchmarks. Despite this often discussed phenomena, benchmark scor
DigitalOcean launched Kimi K3 on day 0. It’s already one of the most popular models on the platform and across the market: second most likes on Hugging Face, sixth most traffic on OpenCode. Getting a
arXiv:2607.23514v1 Announce Type: cross Abstract: Multimodal automated fact-checking (MAFC) verifies claims by retrieving and reasoning over external evidence. However, most existing static benchmarks
arXiv:2510.26384v2 Announce Type: replace Abstract: The prohibitive cost of evaluating large language models (LLMs) on comprehensive benchmarks necessitates the creation of small yet representative da
arXiv:2610.14443v1 Announce Type: cross Abstract: Model cards, system cards, and safety reports increasingly document LLM safety behaviors, yet safety evaluations typically start from predefined bench
arXiv:2603.14426v2 Announce Type: replace Abstract: Existing text-to-video retrieval benchmarks often permit matching from static appearance, making it difficult to determine whether models capture st
arXiv:2607.08964v3 Announce Type: replace Abstract: AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple
arXiv:2610.19869v1 Announce Type: cross Abstract: We introduce SEA-HELM Audio, a Southeast Asian (SEA) multilingual evaluation framework that extends the text-based SEA-HELM benchmark to assess perfor
arXiv:2610.03994v2 Announce Type: replace-cross Abstract: VeriContest is a benchmark of 1007 competitive-programming problems in Rust, each with a Verus specification, a judge-accepted solution, and a
arXiv:2610.12707v1 Announce Type: cross Abstract: Vision-language reward models are trained on annotated preference pairs, which are expensive to collect, can disagree, and are ranked by benchmark acc
arXiv:2610.12998v1 Announce Type: new Abstract: Creativity benchmarks for large language models (LLMs) vary widely in task and metric design. This raises the question of whether they capture a common
arXiv:2610.13563v1 Announce Type: cross Abstract: Repository-level issue-resolution benchmarks such as SWE-bench evaluate submitted patches primarily based on whether they pass the correctness tests,
arXiv:2610.09360v1 Announce Type: cross Abstract: Real-world documents distribute evidence across text, tables, figures, and captions within complex page layouts. Answering complex questions over such
arXiv:2610.07774v1 Announce Type: new Abstract: Vision-language models (VLMs) face compositional safety risks where harmful intent emerges from the interaction between visual and textual inputs. As mi
arXiv:2610.07274v1 Announce Type: new Abstract: Deterministic benchmark scores show that an agent received credit, but not whether that credit was earned, reported honestly, or would hold on a second
arXiv:2608.25097v2 Announce Type: replace Abstract: Understanding how (multimodal) large language models perform on physics problems requires benchmarks that reflect the difficulty and breadth of expe
Hi everyone :) The amazing Swift finetunes of Qwen3.8 27B generate much fewer tokens at mostly similar benchmark performance to the original model, while repos like HyperQwen (formerly syv-ai/qwen38-2
arXiv:2608.27844v1 Announce Type: new Abstract: Existing evaluations of harmful content detection rely predominantly on static benchmarks, which struggle to reflect the interactive adversarial ecosyst
arXiv:2608.26235v1 Announce Type: new Abstract: Accuracy-only benchmarking of reasoning-capable large language models misses a central deployment question: when do extended thinking tokens earn their
arXiv:2509.25143v2 Announce Type: replace-cross Abstract: Existing medical reasoning benchmarks for vision-language models primarily focus on analyzing a patient's condition based on an image from a s
arXiv:2607.21635v1 Announce Type: new Abstract: Personal agents maintain memories, learned skills, tool configurations, and policy state that evolve with each user. Existing agent benchmarks often eva
arXiv:2604.00024v2 Announce Type: replace Abstract: Large language models are increasingly used for medical guidance, but women's health remains under-evaluated in benchmark design. We present the Wom
arXiv:2605.27898v1 Announce Type: new Abstract: As LLMs are increasingly deployed as agents, reliable assessment of their agentic capabilities has become essential. However, reported benchmark scores
arXiv:2505.19563v4 Announce Type: replace Abstract: Mathematical reasoning has long been a key benchmark for evaluating large language models. Although substantial progress has been made on math word
arXiv:2604.03765v2 Announce Type: replace Abstract: Recent advances in multimodal large language models (MLLMs) have greatly improved image understanding and captioning capabilities. However, existing
arXiv:2604.11502v1 Announce Type: cross Abstract: Contextual causal reasoning is a critical yet challenging capability for Large Language Models (LLMs). Existing benchmarks, however, often evaluate th
arXiv:2604.09029v1 Announce Type: cross Abstract: Large language models have been widely explored as decision-support tools in high-stakes domains due to their contextual understanding and reasoning c
arXiv:2604.09037v1 Announce Type: cross Abstract: Current video benchmarks for multimodal large language models (MLLMs) focus on event recognition, temporal ordering, and long-context recall, but over
arXiv:2610.21789v1 Announce Type: cross Abstract: Existing safety benchmarks rely largely on synthetic prompts with loosely grounded taxonomies and offer little focused analysis of severe crime catego
arXiv:2610.03574v1 Announce Type: new Abstract: We introduce HyperBrowseComp, a multilingual and multimodal browsing benchmark comprising 423 manually authored and human-validated questions across 13
arXiv:2610.02270v1 Announce Type: cross Abstract: Medical vision-language model (VLM) evaluation is sensitive to workflow design, prompting strategy, and benchmark construction, yet most studies treat
arXiv:2609.12822v2 Announce Type: replace Abstract: Blinded physician evaluation has been considered by many to be the gold standard for assessing clinical reasoning in large language models (LLMs). T
For instance whenever a model comes out, what’s the best engine to run it, the best harness and absolute minimum you need to get same or near same re results that the benchmark of that model claims. A