Search: “benchmark”

AI World Cup 2026: Benchmarking Large Language Models for End-to-End Football Tournament Prediction

arXiv:2608.03416v1 Announce Type: new Abstract: Large language models (LLMs) are now regularly asked to forecast real-world events, but comparisons are often difficult because models receive different

Human

Content type

All Blog X Post Paper YouTube Reddit GitHub

Clear filters

Model Releases

DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

DGX agent

arXiv:2608.03591v1 Announce Type: cross Abstract: Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to

model-releasesarxiv-cs-ai

5 Aug 2026

Distractor-Aware Truncation: Disentangling Context-Length Effects from Signal Loss in Long-Context LLM Benchmarks

arXiv:2608.03297v1 Announce Type: new Abstract: A standard claim in the literature on retrieval-augmented and memory-augmented language models is that shorter context is better when the relevant infor

Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation

arXiv:2608.03791v1 Announce Type: new Abstract: Vision-Language Models (VLMs), like Large Language Models (LLMs), may memorize sensitive, copyrighted, or harmful knowledge from their pretraining corpo

EduClaw-Bench: A Long-Horizon Benchmark for Pedagogical LLM Agents with Simulated Learners

arXiv:2608.03206v1 Announce Type: cross Abstract: Large language models (LLMs) power educational applications from tutoring to essay scoring, but each is a point solution to a single task, and only re

Evaluating LLMs in Database Scenarios: A Lifecycle Benchmark for Assessing Their Potential in Core Database Tasks

arXiv:2608.03794v1 Announce Type: cross Abstract: Large Language Models (LLMs) are transforming database interaction paradigms, evolving from simple query translators to autonomous database administra

HomeSafeBench: A Benchmark for Embodied Vision-Language Models in Free-Exploration Home Safety Inspection

arXiv:2509.23690v2 Announce Type: replace-cross Abstract: Safety hazards in the home are a leading cause of preventable domestic injuries, motivating an automated inspector that actively explores a ho

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

arXiv:2608.02639v1 Announce Type: cross Abstract: Production prompts rarely carry a single instruction. One system message may require valid JSON, a word limit, three citations, and a fixed tone at th

MultiGlobeQA: A Multilingual and Globally Diverse Benchmark for Geospatial Reasoning

arXiv:2608.03882v1 Announce Type: cross Abstract: Geospatial reasoning, i.e., computing distances, containment, and other spatial relations over real-world entities, is central to navigation and logis

VIBE: A VAD-Informed Benchmark for Entity-Centered Affective Profiling of Large Language Model Outputs

arXiv:2608.03810v1 Announce Type: cross Abstract: Large language models routinely describe socially salient targets, including political figures, countries, religions, organizations, historical events

When Agents Learn to Be You: Benchmarking Privacy Leakage, Impersonation Risk, and Defenses in Persona Skills

arXiv:2608.03700v1 Announce Type: cross Abstract: Persona skills distill personal interaction histories into portable and executable artifacts for downstream agents. While enabling flexible personaliz

Research

A reproducible and extensible framework for benchmarking competing risks survival models

arXiv:2608.00271v1 Announce Type: cross Abstract: A wide range of statistical and machine learning methods have been proposed for survival analysis with competing risks, where the occurrence of one ev

researcharxiv-cs-lg

Research

Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarking

arXiv:2608.01021v1 Announce Type: cross Abstract: In an age of rapid model turnover, how do we make hallucination evaluation more perennial? We explore whether human-written hallucination samples coul

researcharxiv-cs-cl

Does Explainability Transfer? A Controlled Benchmark of Attribution Methods on Vision Transformers and CNNs

arXiv:2608.02396v1 Announce Type: new Abstract: Most evidence on the effectiveness of explainable artificial intelligence (XAI) attribution methods has been established on convolutional neural network

EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next

arXiv:2603.12147v2 Announce Type: replace Abstract: Egocentric video provides a natural modality for studying human behavior, but conventional visual understanding captures mainly observable scenes, o

First Deployable Dynamic-CoM: A Unified Policy and Method-Agnostic Benchmark for Humanoid Single-Leg Balance

model-releasesarxiv-cs-ro

arXiv:2608.00500v1 Announce Type: new Abstract: Unified humanoid policies handle agile whole-body motion, yet stumble on a simple demand: staying balanced on one leg. On our single-leg-balance benchma

How Benchmarks and Evaluation Protocols Shape Conclusions in Provenance-Based Intrusion Detection

arXiv:2608.01454v1 Announce Type: cross Abstract: Provenance-based intrusion detection systems (PIDS) frequently report strong performance, but the conclusions drawn from these results can be highly s

Learning Compositional Meta-Routing for Agentic Workflows: An Executable Benchmark

arXiv:2608.00106v1 Announce Type: new Abstract: Agentic systems must decide not only what answer to produce, but which reasoning and execution operations should precede it. A controller may answer dir

SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials

arXiv:2509.21079v2 Announce Type: replace Abstract: Foundation models have shown remarkable capabilities in various domains, but their performance on complex, multimodal engineering problems remains l

SpatialQuery: Benchmarking Geometry-Grounded Multi-Instance Spatial Reasoning in Vision-Language Models

arXiv:2608.01709v1 Announce Type: new Abstract: Vision-language models (VLMs) achieve strong semantic understanding but remain unreliable in metric spatial reasoning, particularly when queries require

Agents

SWE-Touch: Benchmarking Coding Agents When Users Touch the Code

arXiv:2608.02499v1 Announce Type: cross Abstract: Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task,

agentsarxiv-cs-cl

What Carries the Signal in Pathology Foundation-Model Atlases? A Patient-Level Controlled Benchmark in Breast Cancer

arXiv:2608.00105v1 Announce Type: new Abstract: Pathology foundation models are reported to encode molecular programmes in tissue morphology, but the evidence is usually a cohort-wide ranked gene list

WorldExam: Benchmarking World Models from Apparent Appearance to Inherent Reactivity

arXiv:2608.02603v1 Announce Type: new Abstract: Controllable video generation models are increasingly being developed as world models. Accordingly, evaluating them in this role extends beyond the appa

CBCT-IQ: A Publicly Available Annotated Cone-Beam CT Dataset for Image Quality Assessment and Benchmarking

arXiv:2607.29253v1 Announce Type: cross Abstract: Medical image quality plays a critical role in diagnostic accuracy, especially in X-ray-based imaging modalities such as cone-beam computed tomography

EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environment

arXiv:2604.08342v2 Announce Type: replace Abstract: Long context egocentric video understanding has recently attracted significant research attention, with augmented reality (AR) highlighted as one of

LayoutBench: Performance Benchmarking of Cloud Storage Layouts for Multimedia Data

arXiv:2607.28880v1 Announce Type: cross Abstract: Modern multimedia machine learning workloads increasingly store large-scale datasets in cloud object storage services such as AWS S3. How these sample

So-Fake: Benchmarking and Explaining Social Media Image Forgery Detection

arXiv:2505.18660v5 Announce Type: replace Abstract: Recent advances in AI-powered generative models have enabled the creation of increasingly realistic synthetic images, posing significant risks to in

AHA-Memes: A Fine-Grained Multimodal Benchmark for Understanding Hate in Arabic Memes

arXiv:2607.27393v1 Announce Type: new Abstract: Hateful memes are a growing form of multimodal online harm, where hostile intent is often conveyed through the joint interpretation of images, text, cul

Is it just me, or are current LLM benchmarks failing to capture actual usability? (Gemma 4 vs. Gemini/Claude Opus)

model-releasesr-localllama

Disclaimer, this was kinda written with AI (Gemma 4 again) but it also did really well here, it outputted what I wanted, when I asked it to refine stuff or improve on certain areas it did that without

MMHBench: A Multi-Perspective Benchmark for Mental Health Understanding in Long-Form Videos

arXiv:2607.27895v1 Announce Type: cross Abstract: Mental health understanding in long-form videos requires nuanced reasoning over observable behavior, interpersonal context, and latent psychological s

PlatformBid: An Auto-Bidding Benchmark from a Unified Advertising Platform's Perspective

arXiv:2607.27265v1 Announce Type: new Abstract: Real-time bidding is central to computational advertising, comprising three elements: Supply Side Platform (SSP) selling ad impressions, Demand Side Pla

When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses

arXiv:2607.26348v1 Announce Type: new Abstract: Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and

30 Jul 2026

HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following

arXiv:2607.25398v1 Announce Type: new Abstract: Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context,

29 Jul 2026

RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

arXiv:2512.06276v3 Announce Type: replace-cross Abstract: Referring Expression Comprehension (REC) is a vision-language task that localizes a specific image region based on a textual description. Exis

29 Jul 2026

WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing

arXiv:2607.25765v1 Announce Type: new Abstract: Enterprise agents often need to integrate heterogeneous knowledge sources: documents for narrative facts, tables for computation, and dependency graphs

29 Jul 2026

A Controlled Visual-Backbone Benchmark for Multimodal Short-Term Solar Irradiance Forecasting

arXiv:2607.23633v1 Announce Type: cross Abstract: Sky-image irradiance studies often compare forecasting systems in which the image encoder, temporal model, fusion block, target definition, and traini

DriveDNA: A Large-Scale Multimodal Naturalistic Driving Dataset and Benchmark for Driving Style Identification

arXiv:2607.23822v1 Announce Type: new Abstract: Driving style captures stable, driver-specific patterns in how a vehicle is driven. In naturalistic data, however, this signal is hard to isolate becaus

Subject-Level Heterogeneity in EEG Motor Imagery Decoding: A Large-Scale Benchmark and Portfolio-Based Reduction of the Search Space

arXiv:2607.22778v1 Announce Type: cross Abstract: Robust EEG motor imagery decoding remains limited by strong inter-individual variability, making it difficult to identify pipelines that generalize ac

When Less Is More: A Controlled Benchmark of Lightweight CNNs for Satellite Land-Cover Segmentation on DeepGlobe

arXiv:2607.23024v1 Announce Type: new Abstract: High-resolution satellite imagery is the backbone of good land-cover classification, and without that, environmental monitoring, urban planning, and sus

DBA-Bench: A Production-Fidelity Benchmark for LLM-Based Database Operations Agents

arXiv:2607.22165v1 Announce Type: cross Abstract: LLM-based database agents show promise, but differing task scopes, testbeds, and metrics hinder comparison. We identify four gaps between evaluation a

27 Jul 2026

ReCowGnition: A Realistic Biometric Benchmark for Cow Face Recognition

arXiv:2607.22071v1 Announce Type: new Abstract: With the development of precision livestock farming and the advances in computer vision, visual animal biometrics has gained attention. Using biometric

27 Jul 2026

AISE-Bench: A Full-Cycle Curated Benchmark for Information Seeking on Academic Knowledge Graphs

arXiv:2607.20498v1 Announce Type: new Abstract: Large language models (LLMs) augmented with tools are emerging as autonomous agents capable of using Web engine, APIs, and code to solve complex, long-h

CANN Bench: Benchmarking Agent Generated Kernels against Real NPU and Algorithmic Limits

arXiv:2607.20518v1 Announce Type: new Abstract: AI agents are now capable of writing, compiling, and iteratively optimizing low-level operator kernels on different hardware platforms. Existing benchma

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

arXiv:2607.20482v1 Announce Type: new Abstract: Recent advances in large language models have enabled web agents to autonomously execute complex tasks. In practice, users frequently provide underspeci

QuantiBias: Benchmarking Quantization-Induced Bias in LLMs

arXiv:2607.21063v1 Announce Type: new Abstract: Almost every large language model that reaches a broad audience is quantized: trained in full precision, then compressed for efficiency. This step is as

The RealDefocus Benchmark for Defocus Deblurring

arXiv:2607.21078v1 Announce Type: new Abstract: Single-Image Defocus Deblurring (SIDD) aims to recover an all-in-focus image from a single defocused observation, but rigorous and reproducible evaluati

VoLN: Vision-Only Long-Horizon Navigation---Paradigm, Benchmark, and Method

arXiv:2607.21400v1 Announce Type: cross Abstract: Vision-and-Language Navigation (VLN) enables embodied agents to follow natural-language instructions. However, route-level instructions commonly encod

WildTrace: Benchmarking Natural Evidence Trails in Long-Context Reasoning

arXiv:2607.09328v2 Announce Type: replace-cross Abstract: Answering complex questions over long documents frequently requires integrating evidence that the source itself disperses naturally across dis