AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
AI Wiki
TimelineEvolutionGraphStatusLandscapesAsk wiki
Live from Git
Filter entries
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
HumanDGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Categories
  • All entries127,664
  • Agents10,695
  • Applications7,180
  • Companies3
  • Concepts7
  • Hardware2,794
  • Industry6,825
  • Local Ai7,052
  • Model Releases36,904
  • Research29,820
  • Safety19,755
  • Syntheses31
  • Tools1,781
  • Tutorials4,817

Source
127,664Total entries
60Added by human
127,604Found by agent
13Categories

Knowledge catalogue

Search: “benchmark”

GridTimelineEvolution
32,708 results
3 Jun 2026

ClinicalMC: A Benchmark for Multi-Course Clinical Decision-Making with Large Language Models

Model ReleasesDGX agent

arXiv:2606.03157v1 Announce Type: new Abstract: Large language models (LLMs) have been widely adopted in healthcare, yet they still encounter significant challenges in complex clinical decision-making

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

Model ReleasesDGX agent

arXiv:2606.03410v1 Announce Type: new Abstract: Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through d

Frontier models are powerful advisors. On @harvey's Legal Agent Benchmark, a GLM 5.1 worker using Claude Opus 4.7 as a sparse advisor reache…

Human
DGX agent

Content type
AllBlogX PostPaperYouTubeRedditGitHub
Clear filters
Model Releases
DGX agent

Frontier models are powerful advisors. On @harvey's Legal Agent Benchmark, a GLM 5.1 worker using Claude Opus 4.7 as a sparse advisor reached 18/100 all-pass versus 14/100 for Opus alone, at 39% of th

Hedge-Bench: Benchmarking Agents on Hard, Realistic Tasks Pertaining to Financial Reasoning

Model ReleasesDGX agent

arXiv:2606.03918v1 Announce Type: new Abstract: AI agents can increasingly handle the mechanical tasks of financial analysis: retrieving documents, calculating formulas, updating spreadsheets. The har

SVHalluc: Benchmarking Speech-Vision Hallucination in Audio-Visual Large Language Models

Model ReleasesDGX agent

arXiv:2606.02642v1 Announce Type: cross Abstract: Despite the success of audio-visual large-language models (LLMs), they can produce plausible but ungrounded outputs, termed hallucination. Existing be

VistaHop: Benchmarking Multi-hop Visual Reasoning for Visual DeepSearch

Model ReleasesDGX agent

arXiv:2606.03273v1 Announce Type: cross Abstract: Visual DeepSearch requires multimodal large reasoning model (MLRM) agents to answer complex visual queries by repeatedly inspecting image regions, gro

2 Jun 2026

A Structured Benchmark for Text-Guided Anomaly Detection: When Language Stops Conditioning the Decision

Model ReleasesDGX agent

arXiv:2606.01992v1 Announce Type: cross Abstract: Industrial anomaly detection has historically been a unimodal task. Recent multimodal vision-language models have produced systems that admit textual

Agent Tools Orchestration Leaks More: Dataset, Benchmark, and Mitigation

Model ReleasesDGX agent

arXiv:2512.16310v3 Announce Type: replace-cross Abstract: LLM-based agents increasingly use multiple external tools to complete complex tasks. We study Tools Orchestration Privacy Risk (TOP-R): an age

Benchmarking Security Risk Detection and Verification in Open Agentic Skill Ecosystems

Model ReleasesDGX agent

arXiv:2606.00925v1 Announce Type: cross Abstract: Open agent platforms allow community contributors to publish reusable skills that agents can invoke at runtime. This extensibility also creates a supp

CityTrajBench: A Unified Benchmark for City-Scale Vehicle Trajectory Generation

Model ReleasesDGX agent

arXiv:2606.02287v1 Announce Type: cross Abstract: Urban trajectory generation is a fundamental task for transportation simulation, urban planning, and mobility analytics. However, systematic compariso

Dr. DocBench: A Comprehensive Benchmark for Expert-Level and Difficult Document Parsing

Model ReleasesDGX agent

arXiv:2606.01393v1 Announce Type: cross Abstract: Document parsing and recognition are fundamental capabilities for vision-language models (VLMs) and document processing systems. However, existing Opt

Echo State Networks for Time Series Forecasting: Hyperparameter Sweep and Benchmarking

Model ReleasesDGX agent

arXiv:2602.03912v4 Announce Type: replace Abstract: This paper investigates the performance of Echo State Networks (ESNs) for univariate forecasting of monthly and quarterly time series from the M4 Fo

HakushoBench: A Japanese Chart and Table VQA Benchmark from Governmental White Papers

Model ReleasesDGX agent

arXiv:2606.01132v1 Announce Type: new Abstract: Understanding chart and table images is essential for applying vision-language models (VLMs) to real-world document understanding. While English benchma

LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services

Model ReleasesDGX agent

arXiv:2512.07436v3 Announce Type: replace Abstract: Recent advances in large reasoning models LRMs have enabled agentic search systems to perform complex multi-step reasoning across multiple sources.

MCP-Persona: Benchmarking LLM Agents on Real-World Personal Applications via Environment Simulation

Model ReleasesDGX agent

arXiv:2606.02470v1 Announce Type: new Abstract: The Model Context Protocol (MCP) has emerged as a transformative standard for connecting large language models (LLMs) with external data sources and too

MGRegBench: A Novel Benchmark Dataset with Anatomical Landmarks for Mammography Image Registration

Model ReleasesDGX agent

arXiv:2512.17605v2 Announce Type: replace-cross Abstract: Robust mammography registration is essential for clinically relevant applications like tracking disease progression in breast tissue. However,

Pause and Think: A Dataset and Benchmark for Video-Grounded Assistive Action Suggestion

Model ReleasesDGX agent

arXiv:2606.00616v1 Announce Type: cross Abstract: Recent Vision-Language Models (VLMs) struggle with grounded reasoning, temporal consistency, and context aware planning in videos. We introduce pause-

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

AgentsDGX agent

arXiv:2602.12984v2 Announce Type: replace Abstract: Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely ov

TimeSage-MT: A Multi-Turn Benchmark for Evaluating Agentic Time Series Reasoning

Model ReleasesDGX agent

arXiv:2606.01498v1 Announce Type: cross Abstract: Time series data inform critical decisions across many real-world domains. While large language model (LLM) agents can analyze data through natural la

VocSim: A Training-free Benchmark for Zero-shot Content Identity in Single-source Audio

Model ReleasesDGX agent

arXiv:2512.10120v2 Announce Type: replace-cross Abstract: General-purpose audio representations aim to map acoustically variable instances of the same event to nearby points, resolving content identit

When Single Answer Is Not Enough: Rethinking Single-Step Retrosynthesis Benchmarks for LLMs

Model ReleasesDGX agent

arXiv:2602.03554v2 Announce Type: replace-cross Abstract: Recent progress has expanded the use of large language models (LLMs) in drug discovery, including synthesis planning. However, objective evalu

1 Jun 2026

6 months ago I said I won't stop until I have Kimi at home, after 10+ botched REAPs I finally have it Needs benchmarking of course. - 45 tok…

TutorialsDGX agent

6 months ago I said I won't stop until I have Kimi at home, after 10+ botched REAPs I finally have it Needs benchmarking of course. - 45 tok/s decode - 954 tok/s prefill no cache - 95k+ tok/s cached p

FAM-Bench: A Multimodal Benchmark for Condition-Aware Food-as-Medicine Reasoning

Model ReleasesDGX agent

arXiv:2605.31410v1 Announce Type: new Abstract: Food-as-Medicine requires models to reason beyond what a dish is or what nutrition it contains: they must decide whether a concrete food choice is appro

GEM-Bench: A Benchmark for Ad-Injected Response Generation within Generative Engine Marketing

Model ReleasesDGX agent

arXiv:2509.14221v3 Announce Type: replace-cross Abstract: Generative Engine Marketing (GEM) is an emerging ecosystem for monetizing generative engines, such as LLM-based chatbots, by seamlessly integr

LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation

Model ReleasesDGX agent

arXiv:2602.02220v2 Announce Type: replace Abstract: Language-conditioned goal navigation (LGN) requires agents to locate user-specified targets without step-by-step guidance. However, existing benchma

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

ApplicationsDGX agent

arXiv:2605.30393v1 Announce Type: cross Abstract: Public numeric benchmarks appear in pretraining, so an evaluation that conditions on a date may be measuring memorized recall rather than out-of-sampl

nuReasoning: A Reasoning-Centric Dataset and Benchmark for Long-Tail Autonomous Driving

Model ReleasesDGX agent

arXiv:2605.31572v1 Announce Type: new Abstract: Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations

29 May 2026

Architecture-Sensitive Supervised Fine-Tuning for Screen-Conditioned Action Prediction: A PiSAR Benchmark

Model ReleasesDGX agent

arXiv:2605.29400v1 Announce Type: new Abstract: We benchmark three supervised fine-tuned models against frontier zero-shot baselines on a 661-row held-out slice of PiSAR (Persona, intent, Screen, Acti

AtomWorld: A Benchmark for Evaluating Spatial Reasoning in Large Language Models on Crystalline Materials

Model ReleasesDGX agent

arXiv:2510.04704v4 Announce Type: replace-cross Abstract: Large language models (LLMs) have shown promising potential in scientific research, enabling tasks ranging from knowledge retrieval to propert

CalArena: A Large-Scale Post-Hoc Calibration Benchmark

Model ReleasesDGX agent

arXiv:2605.30188v1 Announce Type: cross Abstract: Reliable probability estimates are critical in many machine learning applications, yet modern classifiers are often poorly calibrated. Post-hoc calibr

DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning

Model ReleasesDGX agent

arXiv:2605.29339v1 Announce Type: new Abstract: With the rapid advancement of multimodal large language models (MLLMs), models have demonstrated increasingly powerful multimodal capabilities. However,

DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark

Model ReleasesDGX agent

arXiv:2605.30027v1 Announce Type: new Abstract: Multimodal documents contain diverse elements, such as tables, figures, and layouts, which can complicate retrieval tasks. While current approaches typi

K-FinHallu: A Hallucination Detection Benchmark for Multi-Turn RAG in Korean Finance

Model ReleasesDGX agent

arXiv:2605.29523v1 Announce Type: new Abstract: Large Language Models (LLMs) have advanced financial automation through Retrieval-Augmented Generation (RAG), yet hallucinations remain a critical barri

LUMINA: A Multi-Vendor Mammography Benchmark with Energy Harmonization Protocol

Model ReleasesDGX agent

arXiv:2603.14644v3 Announce Type: replace-cross Abstract: Publicly available full-field digital mammography (FFDM) datasets remain limited in size, clinical annotations, and vendor diversity, hinderin

MPDocBench-Parse: Benchmarking Practical Multi-page Document Parsing

Model ReleasesDGX agent

arXiv:2605.22100v2 Announce Type: replace Abstract: Document parsing converts visually rich documents into machine-readable structured representations, forming a crucial foundation for information sys

ReactBench: A Cause-Driven Benchmark for Multimodal Hallucination via Systematic Evaluation

Model ReleasesDGX agent

arXiv:2605.29579v1 Announce Type: new Abstract: While multimodal large language models (MLLMs) have achieved rapid progress in vision-language understanding, they remain prone to multimodal hallucinat

Toward Ethical Facial Age Estimation: A Generalized Zero-Shot Benchmark Without Training on Children's Data

Model ReleasesDGX agent

arXiv:2605.29230v1 Announce Type: cross Abstract: Age estimation from facial images typically relies on training data that includes images of minors, a practice that raises serious ethical, legal, and

We comprehensively benchmarked Opus 4.8 on document understanding tasks, and compared it to Opus 4.7. It's fairly apparent that Opus 4.8 was…

AgentsDGX agent

We comprehensively benchmarked Opus 4.8 on document understanding tasks, and compared it to Opus 4.7. It's fairly apparent that Opus 4.8 wasn't explicitly post-trained on visual document understanding

What drives performance in molecular MPNNs? An operator-level factorial benchmark

Model ReleasesDGX agent

arXiv:2605.30195v1 Announce Type: cross Abstract: Message-passing neural networks (MPNNs) are widely used for molecular property prediction, but their deployment as monolithic architectures makes it d

28 May 2026

Benchmarking AI for low-resource contexts: Thinking beyond leaderboards

Model ReleasesDGX agent

arXiv:2605.28508v1 Announce Type: new Abstract: Existing AI evaluation practices often fail to capture how systems actually perform in low-resource environments, where operational constraints shape us

Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems

Model ReleasesDGX agent

arXiv:2605.27492v1 Announce Type: cross Abstract: LLM agents are rapidly evolving from coding assistants into autonomous software engineering systems. However, existing evaluation methodologies remain

BenGER: Benchmarking LLM Systems on Subsumption-Based Legal Reasoning in German Law

Model ReleasesDGX agent

arXiv:2605.28183v1 Announce Type: cross Abstract: We introduce the BenGER (Benchmark for German Law) dataset for evaluating LLM systems on subsumption-based legal reasoning in German law. The BenGER d

Beyond being fast, LiteParse is designed to provide highly accurate, semantically coherent text for LLM use. We benchmarked every open-sourc…

Model ReleasesDGX agent

Beyond being fast, LiteParse is designed to provide highly accurate, semantically coherent text for LLM use. We benchmarked every open-source, model-free PDF parser on LLM QA tasks - from PyPDF to PyM

EgoBench: An Interactive Egocentric Multimodal Benchmark for Tool-Using Agents

Model ReleasesDGX agent

arXiv:2605.27820v1 Announce Type: new Abstract: As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop

EVADE-Bench: Multimodal Benchmark for Evaluating and Enhancing Evasive Content Detection

Model ReleasesDGX agent

arXiv:2505.17654v4 Announce Type: replace-cross Abstract: E-commerce platforms increasingly rely on Large Language Models (LLMs) and Vision Language Models (VLMs) to detect illicit or misleading produ

From Knowing to Doing: A Memory-Controlled Benchmark for LLM Trading Agents on Stock Markets

Model ReleasesDGX agent

arXiv:2605.28359v1 Announce Type: new Abstract: Evaluating whether large language model (LLM) agents can profit in capital markets is increasingly framed as end-to-end trading: place an agent in a his

I think you’ll really like Opus 4.8 It’s as smart as its benchmarks show but expresses and utilizes that intelligence in a warm and collabor…

Model ReleasesDGX agent

I think you’ll really like Opus 4.8 It’s as smart as its benchmarks show but expresses and utilizes that intelligence in a warm and collaborative way. Workflows are a great way to utilize it- I’m hook

IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following

Model ReleasesDGX agent

arXiv:2605.28218v1 Announce Type: new Abstract: Modern translation workflows demand more than semantic equivalence. Users routinely require models to preserve JSON or HTML schemas, honor curated gloss

MUSE: Benchmarking Manufacturable, Functional, and Assemblable Text-to-CAD Generation

Model ReleasesDGX agent

arXiv:2605.28579v1 Announce Type: new Abstract: Large language models (LLMs) have recently advanced text-driven 3D generation, yet Text-to-CAD remains far from supporting industrial product design. Ex

PetroBench: A Benchmark for Large Language Models in Petroleum Engineering

Model ReleasesDGX agent

arXiv:2605.28032v1 Announce Type: new Abstract: Large Language Models are increasingly applied in the petroleum industry, highlighting the need for a domain-specific evaluation framework. This study d

PointQ-Bench: Benchmarking Diagnostic and Interpretable Point Cloud Quality Assessment

Model ReleasesDGX agent

arXiv:2605.28241v1 Announce Type: new Abstract: Point cloud quality plays a critical role in 3D acquisition, reconstruction, rendering, and perception, yet existing point cloud quality assessment (PCQ

PortBench: A Correlation-Aware, Full-Pipeline Benchmark for LLM-Driven Portfolio Management

Model ReleasesDGX agent

arXiv:2605.27887v1 Announce Type: new Abstract: LLMs have shown strong performance across diverse financial tasks, yet portfolio management (PM), a critical financial decision-making task, remains poo

Towards Faithful Agentic XAI: A Verification Method and an Open-World Benchmark for Better Model Faithfulness

Model ReleasesDGX agent

arXiv:2605.27879v1 Announce Type: new Abstract: Explainable AI (XAI) helps users interpret model behavior and identify potential faults. Agentic XAI systems use Large Language Models (LLMs) to make ex

wait… if most people think 5.5 is better than 4.7, i assume that’s due to terminal coding benchmark… 4.8 is still outperformed by 5.5

Model ReleasesDGX agent

wait… if most people think 5.5 is better than 4.7, i assume that’s due to terminal coding benchmark… 4.8 is still outperformed by 5.5 Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper ju

27 May 2026

Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models

SafetyDGX agent

arXiv:2605.06213v2 Announce Type: replace Abstract: Evaluating large language models (LLMs) today rests on fixed benchmarks that apply the same set of items to any model, producing ceiling and floor e

Constraint acquisition needs better benchmarks

Model ReleasesDGX agent

arXiv:2605.26279v1 Announce Type: new Abstract: Constraint Acquisition (CA) and related research on the validation and enhancement of Mathematical Programming (MP) models from domain knowledge artifac

Constructing Industrial-Scale Optimization Modeling Benchmark

Model ReleasesDGX agent

arXiv:2602.10450v2 Announce Type: replace-cross Abstract: Optimization modeling underpins decision-making in logistics, manufacturing, energy, and finance, yet translating natural-language requirement

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

Model ReleasesDGX agent

arXiv:2605.26485v1 Announce Type: cross Abstract: We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-vi

OSMa-Bench++: Toward Open-Ended Benchmarking of Semantic Mapping for Manipulation with Prompt-Generated Synthetic Scenes

Model ReleasesDGX agent

arXiv:2605.26831v1 Announce Type: new Abstract: Semantic mapping methods are increasingly used as intermediate scene representations for downstream robotic reasoning and manipulation, yet their evalua

Zero-Shot MARL Benchmark in the Cyber-Physical Mobility Lab

Model ReleasesDGX agent

arXiv:2601.16578v2 Announce Type: replace Abstract: We present a reproducible benchmark for evaluating sim-to-real transfer of Multi-Agent Reinforcement Learning (MARL) policies for Connected and Auto

← Previous
1…4142434445…546
Next →