Search: “benchmark”

MMLongEmbed: Benchmarking Multimodal Embedding Models in Long-Context Scenarios

arXiv:2606.14747v2 Announce Type: replace-cross Abstract: Recent advancements have significantly expanded the theoretical context windows of Multimodal Embedding Models (MEMs). However, larger context

1 Sept 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation

DGX agent

arXiv:2508.14273v3 Announce Type: replace Abstract: As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and esse

model-releasesarxiv-cs-cl

1 Sept 2026

Towards Stream Learning on Embedded Systems: Benchmarking the Memory Consumption of Stream Learning Methods

arXiv:2608.30923v1 Announce Type: cross Abstract: Stream learning is commonly evaluated through predictive performance and adaptation to concept drift. However, sustained operation of a stream learner

1 Sept 2026

A Shaky Voice Is Not Always a Dodge: Benchmarking Textual and Vocal Evasion Detection in Earnings Calls

arXiv:2608.28040v1 Announce Type: new Abstract: Existing approaches to evasion detection in earnings calls focus on textual transcripts, treating evasion as a single-dimensional phenomenon. We argue t

31 Aug 2026

GLM-5.3-Flash Benchmarks on TensorSharp and llama.cpp

Model: unsloth/GLM-5.3-Flash-GGUF UD-Q2_K_XL (4 shards, 101 GiB) + mmproj-BF16. Reference: llama.cpp PR #27754 (glm5next), CUDA 12.8, SM 120. Throughput Both engines back to back in one session, flash

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models

arXiv:2608.26295v1 Announce Type: cross Abstract: Tool-augmented LLMs must arbitrate between two fallible sources when a tool return conflicts with their parametric memory, yet existing evaluations me

Order Matters: A Chinese Multi-Panel Meme Benchmark for Vision-Language Reasoning

arXiv:2608.26866v1 Announce Type: new Abstract: Many multimodal tasks depend on how visual elements are ordered and composed, not only on recognizing them in isolation. Internet memes are a compact ca

StreamAV-Bench: A Comprehensive Benchmark for Streaming Audio-Video Generation

arXiv:2608.26336v1 Announce Type: cross Abstract: Recent advancements in generative models are pushing video generation toward unbounded streaming audio-video generation for real-time interactive worl

CropCop: An Auditable 120-Class Plant-Health Model from Benchmark Reconstruction to a Quantised Runtime Artifact

arXiv:2608.25539v1 Announce Type: new Abstract: A plant-health score can appear precise while resting on duplicated image families, a long-tailed label space, or a runtime file that was never evaluate

27 Aug 2026

Towards Fine-Grained Text-to-3D Quality Assessment: A Benchmark and A Two-Stage Rank-Learning Metric

arXiv:2509.23841v3 Announce Type: replace Abstract: Recent advances in Text-to-3D (T23D) generative models have enabled the synthesis of diverse, high-fidelity 3D assets from textual prompts. However,

27 Aug 2026

MatReplace: A Reference-Free, Conditioning-Aligned Benchmark for Material Replacement in Interior Scenes

arXiv:2608.24107v1 Announce Type: cross Abstract: Material replacement is a common interior-design operation: changing the material of a selected surface while preserving its geometry, surroundings, a

26 Aug 2026

Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges

arXiv:2608.23845v1 Announce Type: new Abstract: Object-counting methods have rapidly shifted from class-specific density regression to open-vocabulary, foundation-model-backed counters. These methods

26 Aug 2026

Applications

When Seeing Is Not Enough: Benchmarking Interactive Visual Grounding in LVLMs

arXiv:2608.23978v1 Announce Type: new Abstract: Visual grounding is typically evaluated as a one-shot mapping from an informative referring expression to a visual target. This formulation misses a cen

applicationsarxiv-cs-ai

26 Aug 2026

AI Surrogate Modeling for Real-Time Tokamak Equilibrium Prediction: Benchmarking Neural Architectures and Validation on EXL-50U

arXiv:2608.23217v1 Announce Type: cross Abstract: Fast and reliable plasma equilibrium prediction is essential for real-time tokamak operation and control, but conventional Grad-Shafranov (GS) solvers

CallScreenBench: Benchmarking Small Language Models as Phone Secretaries

arXiv:2608.01033v2 Announce Type: replace-cross Abstract: Language models small enough to run on a handset, quantized to a few bits, are increasingly capable of acting on their user's behalf -- which

Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asia

arXiv:2608.23065v1 Announce Type: cross Abstract: Cultural understanding in video means more than recognizing what is visible; it requires grasping the symbolic and temporal significance of cultural c

InstructMove: A Text-Indispensable Benchmark for Instruction-Following Manipulation

model-releasesarxiv-cs-ro

arXiv:2608.22990v1 Announce Type: new Abstract: Vision-language-action (VLA) models have made general-purpose robot manipulation increasingly plausible by conditioning robot actions on natural-languag

LetzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents

arXiv:2608.21714v1 Announce Type: new Abstract: Recent page-image retrievers such as ColPali have improved retrieval over visually rich documents, yet little is known about how they behave in cross-li

Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models

arXiv:2508.07173v3 Announce Type: replace Abstract: Omni-modal Large Language Models (OLLMs) that integrate visual, auditory, and textual processing face severe safety risks. They exhibit fragile defe

SchemaGUI: A Schema-Driven Benchmark for Controllable GUI Generation Evaluation

arXiv:2608.22390v1 Announce Type: new Abstract: Large language models (LLMs) have demonstrated strong potential in graphical user interface (GUI) generation, but reliable evaluation remains challengin

Signal or Noise? A Benchmark Study of Agent Skills in Web Development

arXiv:2608.23067v1 Announce Type: new Abstract: Agent Skills are reusable procedural modules that are increasingly injected into coding-agent sessions to encode framework conventions, anti-patterns, a

WADE: A Reasoning-Annotated Benchmark for Multi-Instance Floating-Waste Grounding with Compact Vision-Language Models

arXiv:2608.22950v1 Announce Type: new Abstract: Floating waste in inland waterways threatens aquatic ecosystems and requires timely monitoring under cluttered, multi-object conditions. Existing aquati

WildHandBench: A Benchmark for Handwritten Text Understanding that Challenges MLLMs and Humans

arXiv:2608.22959v1 Announce Type: cross Abstract: While the top model on OmniDocBench now reaches 96.34% overall on printed-document parsing, the ability of current models to handle challenging handwr

Benchmarking noisy label detection methods

model-releasesarxiv-cs-lg

arXiv:2510.16211v2 Announce Type: replace Abstract: Label noise is a common problem in real-world datasets, affecting both model training and validation. Clean data are essential for achieving strong

24 Aug 2026

I benchmark DFlash 2 (PR build) in llama.cpp on Qwen 3.8 27B against all speculative methods for 3 days. 2.26x on 100 real coding prompts, 4.68x with one n-gram drafter on top. Up to 8x on specific cases.

Hey guys, Inco AI shipped DFlash 2 a few days ago with a drafter for Qwen 3.8 27B and a llama.cpp PR. I built the PR and ran it against plain decoding, MTP, the n-gram lookup drafters, and my July DFl

22 Aug 2026

Tutorials

MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use

arXiv:2608.20202v1 Announce Type: new Abstract: Memory has become a key component of large language models, enabling them to retain information and learn from long-term interactions. However, existing

tutorialsarxiv-cs-ai

21 Aug 2026

NepOOC-M: Bilingual Nepali-English Benchmark and Comparative Analysis of Multimodal Architectures for OOC Detection

arXiv:2608.19212v1 Announce Type: new Abstract: Out-of-context (OOC) misinformation pairs authentic images with misleading captions to construct false narratives without image manipulation, making det

21 Aug 2026

FairGlucose: A CGM Fairness Benchmark Reveals Subgroup Disparities Hidden in Population-Level Validation

arXiv:2608.18296v1 Announce Type: cross Abstract: As CGM-based AI tools approach clinical deployment, whether their accuracy is equitable across patient demographics remains insufficiently tested. To

20 Aug 2026

New benchmark just dropped!

The pelican on a bicycle is sooo outdated, so I came up with a new, improved version. Qwen3.8-27b medium (UD-Q4_K_XL) vs. Sol 5.6 high vs. Qwen3.6-35B (UD-Q6_K_XL) Prompt (only real with typo!): 'Crea

20 Aug 2026

OmniHandwritingOCR: A Diagnostic Benchmark for Evaluating Multimodal LLMs in Handwritten OCR Scenarios

arXiv:2608.18586v1 Announce Type: cross Abstract: Multimodal large language models (MLLMs) are increasingly used as OCR systems in document and knowledge-processing pipelines, but their ability to fai

20 Aug 2026

OV3D-Bench: A Diagnostic Benchmark for Open-Vocabulary Monocular 3D Detection

arXiv:2608.17110v1 Announce Type: new Abstract: Open-vocabulary monocular 3D detectors report strong in-domain performance, but each evaluates under a different protocol, several rely on per-image cat

19 Aug 2026

A Large-Scale Chinese Knowledge Graph-Text Alignment Dataset for Benchmarking Knowledge-Grounded LLMs

arXiv:2510.06039v2 Announce Type: replace-cross Abstract: Reliable evaluation of knowledge-grounded Large Language Models (LLMs) in Chinese requires resources that explicitly align Chinese-language te

Automating Learner Assessment: Benchmarking Machine Learning and Deep Learning Models for EEG-Based Familiarity Prediction

model-releasesarxiv-cs-lg

arXiv:2608.16541v1 Announce Type: cross Abstract: Objective assessment of learning remains a fundamental challenge in education. Electroencephalography (EEG) provides a direct, non-invasive window int

Benchmarking Identity-Sensitive LLM Outputs for Surveillance and Security Robots

model-releasesarxiv-cs-ro

arXiv:2608.16030v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used to generate textual robot design specifications, interaction policies, and risk assessments during ea

CBX-Bench: A Human-Aligned MLLM Council for Benchmarking Concept Bottleneck Model Explanations

arXiv:2608.15404v1 Announce Type: new Abstract: Concept Bottleneck Models (CBMs) are designed to make visual classification interpretable by expressing predictions through human-understandable concept

Comprehensive Benchmarking of Deep Learning Architectures for Lung Cancer Histopathology

arXiv:2608.15915v1 Announce Type: cross Abstract: Lung cancer remains the leading cause of cancer-related mortality worldwide, while histopathological diagnosis is often affected by inter-observer var

FinFraudBench: A Heterogeneous Graph Benchmark for Financial Fraud Detection

arXiv:2608.15177v1 Announce Type: cross Abstract: The increasing complexity of digital financial systems has reshaped financial fraud detection from isolated transaction classification into relational

IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages

arXiv:2608.16344v1 Announce Type: new Abstract: Indic quality estimation (QE) and automatic post-editing (APE) data is spread across separate releases, so no single resource supports training and eval

Ling-3.0 (BailingMoE3) lands in llama.cpp mainline - Quick benchmarks on Intel Arc B580

Finally llama.cpp now officially supports Ling-3.0! (Starting from build b10472+) If you want to run them locally, bartowski has already released the GGUF imatrix quantizations for both models: - Ling

OODBench: Out-of-Distribution Benchmark for Large Vision-Language Models

arXiv:2602.18094v2 Announce Type: replace-cross Abstract: Existing Visual-Language Models (VLMs) have achieved significant progress by being trained on massive-scale datasets, typically under the assu

SLUM-i: Semi-supervised Learning for Urban Mapping of Informal Settlements and Data Quality Benchmarking

arXiv:2602.04525v3 Announce Type: replace-cross Abstract: Very-high-resolution remote-sensing imagery provides a scalable basis for delineating informal settlements, but sparse annotations, severe imb

VideoGAIA: A Benchmark for General AI Assistants on Agentic Video Understanding

arXiv:2608.14718v1 Announce Type: cross Abstract: Video understanding is a fundamental task for evaluating the capabilities of multimodal large language models (MLLMs). However, existing leading model

AnchorBench: A Multi-Pathway Benchmark for the Anchoring Effect in LLMs

arXiv:2608.14320v1 Announce Type: new Abstract: The anchoring effect is a cognitive bias in which an initial reference value shifts a later judgment toward itself. This effect is well established in h

Benchmarking data-driven material models on the classic Treloar dataset

arXiv:2608.14063v1 Announce Type: new Abstract: Machine learning is rapidly reshaping constitutive modeling, offers new ways to learn material behavior directly from experimental data, and challenges

CarbonBench: A Global Benchmark for Upscaling of Carbon Fluxes Using Zero-Shot Learning

model-releasesarxiv-cs-lg

arXiv:2603.09868v2 Announce Type: replace Abstract: Accurately quantifying terrestrial carbon exchange is essential for climate policy and carbon accounting, yet models must generalize to ecosystems u

Generating Benchmark Health Data Using a Tabular Diffusion Transformer

arXiv:2608.14496v1 Announce Type: cross Abstract: Cross-Tabular Data Generation (CTDG) seeks to learn a generative model from multiple heterogeneous tables and produce new synthetic tabular datasets.

Llama.cpp ROCm 7.2->7.14 upgrade, Radeon 780m iGPU benchmarks: ROCm vs Vulkan

With all the new models released recently one important upgrade went unnoticed: Llama.cpp bumped ROCm from 7.2 to 7.14. I was waiting for that because in 7.14 support for gfx1103 (Radeon 780m) was int

16 Aug 2026

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

arXiv:2603.14501v2 Announce Type: replace-cross Abstract: Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resou