Search: “benchmark”

What people call 'distillation' is a super common practice (you use other models to benchmark your model, to evaluate your inputs or to add …

model-releasesclem-delangue--x

What people call 'distillation' is a super common practice (you use other models to benchmark your model, to evaluate your inputs or to add a little bit to your datasets) that in my opinion should be

30 Apr 2026

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

FCMBench-Video: Benchmarking Document Video Intelligence

DGX agent

arXiv:2604.25186v1 Announce Type: new Abstract: Document understanding is a critical capability in financial credit review, onboarding, and remote verification, where both decision accuracy and eviden

model-releasesarxiv-cs-cv

29 Apr 2026

Forward and backward benchmark results across common configurations.

The document titled 'Forward and backward benchmark results across common configurations' is inaccessible because JavaScript is disabled in the browser. Users are prompted to enable JavaScript or swit

model-releasesqwen--x

M^3-VQA: A Benchmark for Multimodal, Multi-Entity, Multi-Hop Visual Question Answering

arXiv:2604.25122v1 Announce Type: new Abstract: We present M^3-VQA, a novel knowledge-based Visual Question Answering (VQA) benchmark, to enhance the evaluation of multimodal large language models (ML

MMLANDMARKS: a Cross-View Instance-Level Benchmark for Geo-Spatial Understanding

arXiv:2512.17492v2 Announce Type: replace Abstract: Geo-spatial analysis of our world benefits from a multimodal approach, as every single geographic location can be described in numerous ways (images

SARU: A Shadow-Aware and Removal Unified Framework for Remote Sensing Images with New Benchmarks

arXiv:2604.25432v1 Announce Type: new Abstract: Shadows are a prevalent problem in remote sensing imagery (RSI), degrading visual quality and severely limiting the performance of downstream tasks like

A Benchmark Suite of Reddit-Derived Datasets for Mental Health Detection

arXiv:2604.23458v1 Announce Type: new Abstract: The growing availability of online support groups has opened up new windows to study mental health through natural language processing (NLP). However, i

AutoGUI-v2: A Comprehensive Multi-Modal GUI Functionality Understanding Benchmark

arXiv:2604.24441v1 Announce Type: new Abstract: Autonomous agents capable of navigating Graphical User Interfaces (GUIs) hold the potential to revolutionize digital productivity. However, achieving tr

CUB: Benchmarking Context Utilisation Techniques for Language Models

arXiv:2505.16518v3 Announce Type: replace-cross Abstract: Incorporating external knowledge is crucial for knowledge-intensive tasks, such as question answering and fact checking. However, language mod

EmoBench-M: Benchmarking Emotional Intelligence for Multimodal Large Language Models

arXiv:2502.04424v4 Announce Type: replace-cross Abstract: With the integration of multimodal large language models (MLLMs) into robotic systems and AI applications, embedding emotional intelligence (E

Energy-Arena: A Dynamic Benchmark for Operational Energy Forecasting

model-releasesarxiv-cs-lg

arXiv:2604.24705v1 Announce Type: cross Abstract: Energy forecasting research faces a persistent comparability gap that makes it difficult to measure consistent progress over time. Reported accuracy g

GAMMAF: A Common Framework for Graph-Based Anomaly Monitoring Benchmarking in LLM Multi-Agent Systems

arXiv:2604.24477v1 Announce Type: cross Abstract: The rapid integration of Large Language Models (LLMs) into Multi-Agent Systems (MAS) has significantly enhanced their collaborative problem-solving ca

MetaGAI: A Large-Scale and High-Quality Benchmark for Generative AI Model and Data Card Generation

arXiv:2604.23539v1 Announce Type: new Abstract: The rapid proliferation of Generative AI necessitates rigorous documentation standards for transparency and governance. However, manual creation of Mode

Results on the remainder of our benchmarks are in for Kimi K2.6, the new #1 open-weight model on the Vals Index (#8 overall) Kimi K2.6 is co…

model-releaseskimi-moonshot--x

Results on the remainder of our benchmarks are in for Kimi K2.6, the new #1 open-weight model on the Vals Index (#8 overall) Kimi K2.6 is competitive with many closed-weight models, at a fraction of t

Knowledge Visualization: A Benchmark and Method for Knowledge-Intensive Text-to-Image Generation

arXiv:2604.22302v1 Announce Type: new Abstract: Recent text-to-image (T2I) models have demonstrated impressive capabilities in photorealistic synthesis and instruction following. However, their reliab

27 Apr 2026

OpenAI just released HealthBench Professional on Hugging Face A medical evaluation benchmark designed to improve AI assistants for clinician…

model-releasesclem-delangue--x

OpenAI just released HealthBench Professional on Hugging Face A medical evaluation benchmark designed to improve AI assistants for clinicians, featuring physician-curated conversations and rubric-base

25 Apr 2026

Applications

Organizational design for agents is hard, benchmarking agents working in concert is hard. Together, this is the next critical frontier for m…

applicationsethan-mollick--x

Organizational design for agents is hard, benchmarking agents working in concert is hard. Together, this is the next critical frontier for making AI matter in economically valuable tasks, and we reall

25 Apr 2026

GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark

arXiv:2601.13711v2 Announce Type: replace Abstract: Authorship verification (AV) is the task of determining whether two texts were written by the same author and has been studied extensively, predomin

Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores

arXiv:2511.20697v4 Announce Type: replace-cross Abstract: Understanding complete musical scores entails integrated reasoning over pitch, rhythm, harmony, and large-scale structure, yet the ability of

VidHal: Benchmarking Temporal Hallucinations in Vision LLMs

arXiv:2411.16771v3 Announce Type: replace Abstract: Vision Large Language Models (VLLMs) are widely acknowledged to be prone to hallucinations. Existing research addressing this problem has primarily

WorldMark: A Unified Benchmark Suite for Interactive Video World Models

arXiv:2604.21686v1 Announce Type: new Abstract: Interactive video generation models such as Genie, YUME, HY-World, and Matrix-Game are advancing rapidly, yet every model is evaluated on its own benchm

From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents

arXiv:2604.20006v1 Announce Type: new Abstract: Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. Ho

PipeMFL-240K: A Large-scale Dataset and Benchmark for Object Detection in Pipeline Magnetic Flux Leakage Imaging

arXiv:2602.07044v2 Announce Type: replace-cross Abstract: Pipeline integrity is critical to industrial safety and environmental protection, with Magnetic Flux Leakage (MFL) detection being a primary n

ThermoQA: A Three-Tier Benchmark for Evaluating Thermodynamic Reasoning in Large Language Models

arXiv:2604.19758v1 Announce Type: new Abstract: We present ThermoQA, a benchmark of 293 open-ended engineering thermodynamics problems in three tiers: property lookups (110 Q), component analysis (101

WildFireVQA: A Large-Scale Radiometric Thermal VQA Benchmark for Aerial Wildfire Monitoring

arXiv:2604.20190v1 Announce Type: new Abstract: Wildfire monitoring requires timely, actionable situational awareness from airborne platforms, yet existing aerial visual question answering (VQA) bench

A Controlled Benchmark of Visual State-Space Backbones with Domain-Shift and Boundary Analysis for Remote-Sensing Segmentation

arXiv:2604.18721v1 Announce Type: cross Abstract: Visual state-space models (SSMs) are increasingly promoted as efficient alternatives to Vision Transformers, yet their practical advantages remain unc

Alibaba launches Qwen3.6-27B, an open-weight dense model with 27B parameters, saying it surpasses Qwen3.5-397B-A17B on major coding benchmarks (Qwen)

Qwen: Alibaba launches Qwen3.6-27B, an open-weight dense model with 27B parameters, saying it surpasses Qwen3.5-397B-A17B on major coding benchmarks — · 4226 words · QwenTeam丨Translations:.体中文 — HUGGI

model-releasestechmeme

Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks

arXiv:2512.22673v3 Announce Type: replace Abstract: Travel planning is a natural real-world task to test large language models' (LLMs) planning and tool-use abilities. Although prior work has studied

Beyond Rating: A Comprehensive Evaluation and Benchmark for AI Reviews

arXiv:2604.19502v1 Announce Type: new Abstract: The rapid adoption of Large Language Models (LLMs) has spurred interest in automated peer review; however, progress is currently stifled by benchmarks t

HalluAudio: A Comprehensive Benchmark for Hallucination Detection in Large Audio-Language Models

arXiv:2604.19300v1 Announce Type: cross Abstract: Large Audio-Language Models (LALMs) have recently achieved strong performance across various audio-centric tasks. However, hallucination, where models

LegalBench-BR: A Benchmark for Evaluating Large Language Models on Brazilian Legal Decision Classification

arXiv:2604.18878v1 Announce Type: new Abstract: We introduce LegalBench-BR, the first public benchmark for evaluating language models on Brazilian legal text classification. The dataset comprises 3,10

RoboWM-Bench: A Benchmark for Evaluating World Models in Robotic Manipulation

arXiv:2604.19092v1 Announce Type: cross Abstract: Recent advances in large-scale video world models have enabled increasingly realistic future prediction, raising the prospect of leveraging imagined v

Sources: Tencent and Alibaba are in talks to invest in DeepSeek at a 20B+ valuation, partly benchmarked against Moonshot's pending round at an 18B valuation (The Information)

The Information: Sources: Tencent and Alibaba are in talks to invest in DeepSeek at a 20B+ valuation, partly benchmarked against Moonshot's pending round at an 18B valuation — Chinese tech giants Tenc

model-releasestechmeme

Towards Reliable Human Evaluations in Gesture Generation: Insights from a Community-Driven State-of-the-Art Benchmark

arXiv:2511.01233v3 Announce Type: replace Abstract: We review human evaluation practices in automatic, speech-driven 3D gesture generation and find a lack of standardisation and frequent use of flawed

A Real-World Grasping-in-Clutter Performance Evaluation Benchmark for Robotic Food Waste Sorting

model-releasesarxiv-cs-ro

arXiv:2602.18835v2 Announce Type: replace Abstract: Food waste management is critical for sustainability, yet inorganic contaminants hinder recycling potential. Robotic automation accelerates sorting

A Systematic Survey and Benchmark of Deep Learning for Molecular Property Prediction in the Foundation Model Era

model-releasesarxiv-cs-lg

arXiv:2604.16586v1 Announce Type: new Abstract: Molecular property prediction integrates quantum chemistry, cheminformatics, and deep learning to connect molecular structure with physicochemical and b

Benchmarking Real-Time Question Answering via Executable Code Workflows

arXiv:2604.16349v1 Announce Type: cross Abstract: Retrieving real-time information is a fundamental capability for search-integrated agents in real-world applications. However, existing benchmarks are

CFMS: Towards Explainable and Fine-Grained Chinese Multimodal Sarcasm Detection Benchmark

arXiv:2604.16372v1 Announce Type: new Abstract: Multimodal sarcasm detection has recently garnered significant attention. However, existing benchmarks suffer from coarse-grained annotations and limite

DSH-Bench: A Difficulty- and Scenario-Aware Benchmark with Hierarchical Subject Taxonomy for Subject-Driven Text-to-Image Generation

arXiv:2603.08090v2 Announce Type: replace Abstract: Significant progress has been achieved in subject-driven text-to-image (T2I) generation, which aims to synthesize new images depicting target subjec

ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection

arXiv:2410.04509v3 Announce Type: replace Abstract: As the field of Multimodal Large Language Models (MLLMs) continues to evolve, their potential to revolutionize artificial intelligence is particular

I came up with a somewhat foolish new benchmark for testing image generation models, to exercise the new ChatGPT Images 2.0: 'Do a where's W…

model-releasessimon-willison--x

I came up with a somewhat foolish new benchmark for testing image generation models, to exercise the new ChatGPT Images 2.0: 'Do a where's Waldo style image but it's where is the raccoon holding a ham

Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluation

arXiv:2510.09275v2 Announce Type: replace Abstract: Medical diagnostics is a high-stakes and complex domain that is critical to patient care. However, current evaluations of large language models (LLM

MedProbeBench: Systematic Benchmarking at Deep Evidence Integration for Expert-level Medical Guideline

arXiv:2604.18418v1 Announce Type: new Abstract: Recent advances in deep research systems enable large language models to retrieve, synthesize, and reason over large-scale external knowledge. In medici

Precise Debugging Benchmark: Is Your Model Debugging or Regenerating?

arXiv:2604.17338v1 Announce Type: cross Abstract: Unlike code completion, debugging requires localizing faults and applying targeted edits. We observe that frontier LLMs often regenerate correct but o

ResearchBench: Benchmarking LLMs in Scientific Discovery via Inspiration-Based Task Decomposition

arXiv:2503.21248v3 Announce Type: replace Abstract: Large language models (LLMs) have shown potential in assisting scientific research, yet their ability to discover high-quality research hypotheses r

Revisiting a Pain in the Neck: A Semantic Reasoning Benchmark for Language Models

arXiv:2604.16593v1 Announce Type: new Abstract: We present SemanticQA, an evaluation suite designed to assess language models (LMs) in semantic phrase processing tasks. The benchmark consolidates exis

Screen Before You Interpret: A Portable Validity Protocol for Benchmark-Based LLM Confidence Signals

arXiv:2604.17714v1 Announce Type: new Abstract: LLM confidence signals are used for abstention, routing, and safety-critical decisions. No standard practice exists for checking whether a confidence si

Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks

arXiv:2604.17159v1 Announce Type: cross Abstract: We present, to our knowledge, the most comprehensive cross-model evaluation of LLM agents on offensive cybersecurity tasks, benchmarking 10 frontier m